Table 1 Outline of key proposed principles for analysing flawed, uncertain, proximate or sparse (FUPS) data and how they were employed in CYP IAPT

1

Treat data as a partial remnant

Present data in such a way as to convey any limitations to interpretation stemming from its FUPS characteristics

• Introduced notion of FUPS data at start of report

• Kept reiterating limitations of data at relevant points in report

• Presented flow diagram of data loss

• Chapter on implications of missing data and hypotheses as to potential impact

• Undertook and invited blogs and responses on potential interpretations and limitations

2

Transparency of analyses: avoid ‘black box’ statistics

Ensure that all use of data follows principles of transparency and clarity

• Included detail of questionnaires

• Ensure all axis labels on graphs are factual (what was questionnaire) rather than interpretive (performance or quality of care)

• Did not use terms ‘significance’ or ‘performance data’

• Detailed descriptions given on all metrics

• Clarified where different measures had different thresholds or other key metrics

• Kept different groups separate, i.e. parent and child reports considered separately

• Included raw numbers in the report and reiterated denominators regularly. All statistical techniques used in the report were clearly explained and not complicated

• No ‘black box’ techniques used

• Very clear that data not collected as part of a trial so could not be taken as an evaluation of the programme itself

3

Triangulation

Consider the data in the context of other information to see what supports or undermines the findings from these particular FUPS data

• Reviewed other relevant information from the literature

• Contextualised against information from other areas of healthcare

• Made clear that status quo may not be safer or more effective than alternative