Avoidable waste of research related to outcome planning and reporting in clinical trials

Background Inadequate planning, selective reporting, and incomplete reporting of outcomes in randomized controlled trials (RCTs) contribute to the problem of waste of research. We aimed to describe such a waste and to examine to what extent this waste could be avoided. Methods This research-on-research study was based on RCTs included in Cochrane reviews with a summary of findings (SoF) table. We considered the outcomes reported in the SoF tables as surrogates for important outcomes for patients and other decision makers. We used a three-step approach. (1) First, in each review, we identified, for each important outcome, RCTs that were excluded from the corresponding meta-analysis. (2) Then, for these RCTs, we systematically searched for registrations and protocols to distinguish between inadequate planning (an important outcome was not reported in registries or protocols), selective reporting (an important outcome was reported in registries or protocols but not in publications), and incomplete reporting (an important outcome was incompletely reported in publications). (3) Finally, we assessed, with the consensus of five experts, the feasibility and cost of measuring the important outcomes that were not planned. We considered inadequately planned or selectively or incompletely reported important outcomes as avoidable waste if the outcome could have been easily measured at no additional cost based on expert evaluation. Results Of the 2711 RCTs included in the main comparison of 290 reviews, 2115 (78%) were excluded from at least one meta-analysis of important outcomes. Every trial contributed to 55%, on average, of the meta-analyses of important outcomes. Of the 310 RCTs published in 2010 or later, 156 were registered. Inadequate planning affected 79% of these RCTs, whereas incomplete and selective reporting affected 41% and 15%, respectively. For 63% of RCTs, we found at least one missing important outcome for which the waste was avoidable and for 30%, the waste was avoidable for all important outcomes. Conclusions Most of the RCTs included in our sample did not contribute to all the important outcomes in meta-analyses, mostly because of inadequate planning or incomplete reporting. A large part of this waste of research seemed to be avoidable. Electronic supplementary material The online version of this article (10.1186/s12916-018-1083-x) contains supplementary material, which is available to authorized users.

Therefore, when planning an RCT, researchers are expected to measure all important outcomes [9]. However, previous studies found that less than one-fifth of diabetes RCTs and less than one-quarter of cardiovascular trials considered patient-important outcomes as their primary outcomes [19,20]. Instead, researchers frequently rely on surrogates as a proxy for final patient-important outcomes because these outcomes allow for smaller, faster, and thus, cheaper clinical trials [21][22][23][24]. However, surrogates can be misleading, because they may show exaggerated treatment effect sizes or even an apparent benefit of harmful treatments, as was the case for the use of antiarrhythmic drugs after myocardial infarction, which led to the deaths of several thousand patients decades ago [25][26][27][28].
Selective reporting has been repeatedly described as another important issue affecting RCT outcomes [29][30][31][32]. Outcome reporting bias arises when outcomes are selectively reported based on the nature and direction of the results [33]. In a recent study, the median proportion of RCTs with discrepancies between registered and published primary outcomes was 31% [34]. Statistically significant outcomes were 2 to 4 times more likely to be reported in publications than non-significant ones, which biases the available body of evidence toward more positive results [30].
Similarly, a median of 31% to 50% of efficacy outcomes were found to be incompletely reported in RCT articles [29,35,36]. With incomplete reporting of outcomes, outcomes cannot be included in meta-analyses, which poses a serious threat to the usability of trial results: not including all available results in meta-analyses can lead to a truncated vision of the overall body of evidence.
The purpose of this study was to describe the waste of research related to inadequate planning, selective reporting, or incomplete reporting of outcomes in RCTs and to examine to what extent this waste could be avoided. For this, we addressed the following specific questions: (1) What proportion of RCTs are excluded from meta-analyses due to outcome reasons? (2) Were the exclusions related to inadequate planning, selective reporting, or incomplete reporting? (3) Was it feasible to measure the missing outcomes at the planning stage, and at what cost?

Methods
We performed a research-on-research study based on RCTs included in Cochrane systematic reviews. We used the outcomes reported in the summary of findings (SoF) tables of reviews as surrogates for important outcomes, because Cochrane systematic review SoF tables should include the most important outcomes for patients and other decision makers, whether they are available in RCTs or not [7]. This study used a three-step approach. (1) First, for all important outcomes, we identified the RCTs that were included in the Cochrane reviews but excluded from the corresponding meta-analyses because the important outcome was missing. (2) Then, we systematically searched for trial registrations and protocols to distinguish between the outcomes that were not planned (not included in registries or protocols, i.e., inadequate planning) and outcomes that were planned. For the planned outcomes, we distinguished between those that were adequately reported and those that were incompletely reported (poor reporting) or not reported (selective reporting). (3) Finally, we assessed, via expert consensus, the feasibility and cost of measuring the outcomes that were not planned.
The eligibility criteria at each phase of the study are summarized in Additional file 1.

Identification of RCTs excluded from meta-analyses Data sources
We obtained data from all Cochrane systematic reviews published between March 2011 and September 2014. They were provided by the Cochrane Collaboration editorial unit as XML files and contained all information reported by the review authors in RevMan, the software developed by the Cochrane Collaboration for preparing and maintaining systematic reviews [37]. Cochrane systematic reviews of interventions are organized by comparisons of two treatment groups. Meta-analyses are then performed for one or more outcomes within each comparison. Cochrane reviewers are encouraged to present an SoF table summarizing information on the quality of evidence and treatment effect magnitude (from the meta-analysis result) for the most important outcomes (see example in the review by Mocellin and colleagues [38]) [7]. According to the Cochrane collaboration, these outcomes should be important to all research end users, that is, patients and other decision makers [7]. These outcomes include a "wide variety of events such as mortality and major morbidity (such as stroke and myocardial infarction); however, they may also represent frequent minor and rare major side effects, symptoms and quality of life, burdens associated with treatment, and resource issues (costs)" [7].

Selection of relevant Cochrane systematic reviews
Using R 3.1.1 and the XML package, one of us (IA) removed from the set of reviews provided by the Cochrane Collaboration withdrawn Cochrane reviews and then identified all reviews of RCTs with an SoF table. We excluded reviews that included observational studies and those including only RCTs published before 2007 because we focused on recent RCTs. From all eligible Cochrane reviews, using a random number generator, we drew a random sample of 300 for an in-depth evaluation but excluded a further 10 reviews because their SoF tables mixed various interventions, which resulted in a final sample of 290 Cochrane reviews.

Identification of important outcomes
We considered outcomes reported in the SoF table as a surrogate for important outcomes. Therefore, we manually identified and extracted all outcomes reported in the SoF tables. Most reviews had a single SoF table, but some had several tables, corresponding to different comparisons. In this case, we focused on the main comparison as acknowledged by the authors. If the main comparison was not reported by the review authors or if various comparisons were reported in the same SoF table, we selected the comparison with the most outcomes and included the largest number of RCTs. If the SoF table reported various interventions (e.g., presented three meta-analyses of different outcomes for three different interventions), the review was excluded.

Identification of RCTs excluded from meta-analyses
For each review and for each important outcome, we identified all RCTs excluded from the outcome in the corresponding meta-analysis. To do so, by using the reference list of all studies included in the Cochrane review, we first identified all RCTs available for the selected comparison by extracting all RCTs included in any meta-analysis reported for this comparison. Then, for each of these RCTs, we manually evaluated whether they contributed to the meta-analysis of each important outcome by screening RCTs included in the corresponding meta-analysis.

Evaluation of the reason for a missing outcome
We a priori hypothesized that the exclusion of an RCT from a meta-analysis for outcome reasons could be related to inadequate planning (i.e., the outcome was not planned to be measured), selective reporting (i.e., the outcome was planned but not reported in the publication), or incomplete reporting (i.e., the outcome was reported in the publication but not in a way that allowed for pooling of data).
To distinguish among inadequate planning, selective reporting, or incomplete reporting, for each RCT excluded from at least one meta-analysis, we screened the data available in Cochrane reviews and systematically searched for trial registration and/or protocols.
We focused on RCTs published in 2010 or later to maximize the chance of identifying trial registration or protocols.

Search for RCT registration and protocols
For each RCT, we individually assessed all available reports and information. To do that, we extracted the references identified from the "References to studies included in this review" section of the Cochrane reviews. One of us (YY) retrieved all the articles, conference abstracts, reports etc. related to the identified RCT. We screened all articles for any information regarding registration (name of trial registry and/or registration number) and/or protocol availability. When no reference to a trial registration was found, we used the following approach: 1. We searched the full text of the Cochrane review for any information regarding trial registration. 2. If no information was found, we searched the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP). Also, according to the author's affiliation, we searched the local registry if not part of the WHO ICTRP. We used the article title, the first and last author's name as author or as investigator, or the author's affiliation as search keywords. 3. If no information was found, we searched Google with the publication title and keywords regarding registration (e.g., registry, registration, NCT etc.). 4. If no information was found, we contacted the corresponding author to ask whether the trial was registered and whether a protocol was available.

Evaluation of the reason for the missing outcome
For each identified RCT, one of us (YY) classified each missing outcome into one of the following five categories: 1. Inadequate planning The outcome was not planned according to the protocol or the registry entry. It was not reported in the available trial reports.

Selective reporting
The outcome was planned according to the protocol or the registry entry but was not reported in the available trial reports.

Incomplete reporting
The outcome was planned according to the protocol or the registry entry and was reported in the available trial reports but incompletely or not in a way that allowed for meta-analysis (e.g., mixed model analyses reported, no results per group, missing control group results, difference in means without the standard error, etc.). No protocol or registry entry was found. The outcome was reported in the available trial reports but not in a way that allowed for meta-analysis. 4. Unable to distinguish between selective reporting and inadequate planning No protocol or registry entry was found and the outcome was not reported in the available trial reports.

Other situations
The outcome was listed in the trial reports, but there was no event (e.g., the outcome was death but no death occurred). The outcome concerned adverse events, but there was no event (e.g., the outcome was "Major Complications-Visceral injury, " but no complications were reported).
The outcome was reported in the available trial reports in a way that could allow for inclusion in a meta-analysis but was not included in the meta-analysis.
As a quality assessment measure, 10% of the RCTs were classified independently by two reviewers (YY and AD), with no disagreements.

Evaluation of research waste
Feasibility and cost of measuring the missing outcomes that were not planned To evaluate whether the missing outcomes that were not planned could have been easily measured, we used an expert consensus approach. One of us (YY) presented to an expert panel of five methodologists and trialists (AD, IB, PC, PR, and VT) each RCT for which at least one missing outcome was classified as not planned. After evaluating standardized information on the population, interventions in the experimental and control group, and other outcomes evaluated, the panel of experts were asked to answer the following questions regarding the missing outcomes: "According to you, given the other outcomes measured in the trial, and based on your experience, would you consider that measuring the presented outcome would be easy, moderately easy, difficult or impossible in most cases from a trialist's perspective?" "According to you, given the other outcomes measured in the trial, and based on your experience, would you consider that measuring the presented outcome would be easy, moderately easy, difficult or impossible in most cases from a patient's perspective?" "According to you, given the other outcomes measured in the trial, and based on your experience, what would be the approximate cost of measuring this outcome: no cost defined as ≤ 1% of the total cost of the trial; minor cost, defined as ≤ 5%; moderate cost, 5% to 15%; or major cost, 15% or more". These percentages were indicative. "According to you, how important is the missing outcome: major importance; non-major importance?" One of us (YY) ensured that all experts first gave their opinion and then discussed together, to avoid the opinion of one leading person influencing the others. The final feasibility and cost evaluation of every proposed adjustment was based on the group consensus.
The qualifications and areas of expertise of the experts involved are summarized in Additional file 2.

Avoidable waste of research related to missing outcomes
We defined avoidable waste related to missing outcomes as missing important outcomes related to: Selective reporting Incomplete reporting Inadequate planning, if the outcome was judged by the expert panel as easy to collect from both the trialist and patient perspective, not costly (i.e., no or minor additional cost), and of critical importance At the trial level, such waste could have been partially avoided if the trial could have been included in the meta-analysis of at least one missing outcome and totally if it could have been included in all meta-analyses of missing outcomes.

Identification of RCTs excluded from meta-analyses Selection and characteristics of the Cochrane systematic reviews
The complete selection process is described in Fig. 1. Briefly, from the 2796 Cochrane systematic reviews published between March 2011 and September 2014, 820 reviews corresponded to our eligibility criteria. Our selection process resulted in a sample of 290 reviews (5047 RCTs), with a median of 11 RCTs per review (Q1-Q3: 5-21) (Fig. 1). The subset of included reviews appeared comparable to the eligible Cochrane Reviews (Additional file 3). The reviews investigated 47 different health research topics, the most common being airways (7%, n = 21/290), menstrual disorders and subfertility (7%, n = 20/290), and anesthesia (6%, n = 18/290) ( Table 1). Experimental interventions were pharmacological in 60% of reviews (n = 175/290).
Proportion of RCTs excluded from the meta-analyses due to outcome reasons The 290 reviews included a total of 2711 RCTs in the main comparison; 596 (22%; 95% CI: 20-24) were included in all meta-analyses of important outcomes, and Fig. 1 Flow chart of the selection process. This figure summarizes the selection process for Cochrane reviews, RCTs, and outcomes. The analyzed sample involved 290 Cochrane reviews, which included 2711 RCTs in the SoF table of the main comparison. The SoF tables reported 1414 important outcomes. RCT randomized controlled trial, SoF summary of findings 2115 (78%, 95% CI: 76-80) were excluded from at least one meta-analysis. Every RCT contributed to 55%, on average, of the meta-analyses for important outcomes.

Evaluation of the reason for missing outcomes
Among the 2115 RCTs excluded from at least one meta-analysis of important outcomes, 310 were published in 2010 or later. We further excluded 19 RCTs, because their reports were not accessible (e.g., retracted paper) or were not in English or French (e.g., Chinese), which left 291 RCTs for further evaluation, with a total of 971 missing outcomes. A registration number or a protocol was retrieved for 54% (n = 156/291) of these RCTs (corresponding to 461 missing outcomes).

Evaluation of research waste
Feasibility of measuring the missing outcomes that were not planned We submitted the 282 outcomes missing due to inadequate planning from 123 RCTs to our panel of experts. For 78% of outcomes (n = 221/282), the experts judged the outcome of critical importance given the context. For these 221 critically important outcomes, they considered that 82% (n = 182) could have been easily measured from both the trialist and patient perspective at no cost (Additional file 4).

Discussion
We evaluated the avoidable waste of research due to outcome-related reasons across a large number of RCTs included in recent Cochrane systematic reviews in a variety of medical fields. Our analysis revealed that 78% of the RCTs were not included in all meta-analyses of important outcomes and that every RCT contributed to 55%, on average, of the meta-analyses of these important outcomes. Among registered RCTs, inadequate planning was the most common reason affecting 79% of RCTs with missing outcomes. Such waste could have been partially avoided for 63% of the RCTs and totally for 30%. Our findings suggest that many RCTs were not included in all meta-analyses of important outcomes and that every RCT contributed to only half of the meta-analyses, on average. These results seem consistent with the literature. In a recent study of patient-important outcomes, the median proportion of trials included in a meta-analysis for such outcomes was about 60% [39]. Approximately half of the RCTs included in another sample of Cochrane reviews were included in the pooled effect size estimates in the meta-analyses of patient-important outcomes [40], whereas more than one-third of the outcomes prespecified in the review were not reported in the results sections [18,41].
Although possible reasons for excluding RCTS from meta-analyses have been hypothesized, the respective contribution of inadequate planning and selective reporting or incomplete reporting was unknown. Our results provide major insights into this question, showing that many RCTs do not plan the most important outcomes to evaluate. Inadequate planning represents a missed opportunity within a trial because equipoise remains unchanged after this trial. However, a missed opportunity at the trial level will result in waste of research at the meta-analysis level. Actually, clinical research should be considered a sequential process, with each new trial contributing to the existing body of evidence. If a trial fails to do so because of a missed opportunity, it becomes a source of waste of research and resources at the meta-analysis level because this trial will not contribute to the overall evidence. Cochrane reviewers may select outcomes that differ in importance from those selected by trialists (typically those defined as primary outcomes and used for sample size calculation). Trialists rarely consider adverse events (particularly severe adverse events) as primary outcomes and for sample size calculation. However, these outcomes are crucial to assess and report because they could be included in meta-analyses to increase power and inform decisionmaking. Identifying important outcomes is challenging, and without a consensus on outcomes, heterogeneity in outcomes evaluated will remain [41].
To overcome these issues, several initiatives have emerged to improve the relevance and consistency of outcomes used in clinical research [4,[42][43][44][45]; one is the Core Outcome Measures in Effectiveness Trials (COMET) initiative promoting the development and use of standardized core outcome sets (COSs). COMET offers extremely helpful tools for researchers aiming to develop COSs in their fields. However, although the number of COSs is rapidly increasing, it remains limited [46][47][48], and searching for specific existing COSs for a given condition is difficult. The COMET website may evolve to present available COSs in a tabular fashion to help trialists identify the outcomes to use and researchers the conditions for which COSs are needed. Another perspective would be to use the outcomes reported in the SoF table of Cochrane reviews to develop COSs, because most of these outcomes seem important to patients.
Incomplete reporting was another common reason for outcomes excluded from meta-analyses. Two main situations can be distinguished. First, the failure to report results adequately (e.g., reporting means without standard deviations) and second, the reporting of an analysis that cannot be included as such in a meta-analysis (e.g., reporting of repeated data analysis). The first situation is clearly related to poor reporting. We previously showed that results were more completely reported at *The total exceeds 100% because some outcomes were included in more than one category ClinicalTrials.gov than in publications, probably because of the standardized template used to report results at ClinicalTrials.gov [49]. Such standardized templates may help trialists determine which data should be reported and should be included in reporting guidelines and required by journals to improve the presentation of results. The other situation is more complex because it does not reflect poor reporting, just reporting in a different way. Studies with repeated data are meant to record measurements at numerous time points to inform researchers of changes over time. However, several approaches for meta-analyzing these types of data exist, and they can differ in terms of the data needed for analysis [50]. Therefore, reporting trial results not just for an immediate use but also considering a later inclusion in   Finally, selective reporting of outcomes seems to affect a few trials. It remains an important issue in clinical trials, with, on average, one-third of discrepancies in primary outcomes between protocols and publications [29] or between registry information and publications [31]. In our situation, this concerns only outcomes that were planned and not reported at all in publications. Actually, reviewers consider a given outcome whether this outcome is reported as a primary or a secondary outcome.

Strengths and limitations of the study
In this study, we used an original approach to evaluate the main reasons why a trial was not included in a meta-analysis and the part of this waste that could be avoided. We analyzed a large set of trials included in a vast, unselected, sample of recent Cochrane reviews exploring various health-care research topics. Assembling numerous experts in such a variety of fields to identify the most important outcomes to evaluate would have been challenging. As a proxy, we used the outcomes reported in the SoF tables that were considered by the review authors as the most important to measure for a given comparison in a particular health condition [7]. We used the consensus of an expert panel of recognized methodologists and trialists to assess the feasibility and costs of measuring the missing outcomes. To avoid overestimating waste of research, we also asked the panel to evaluate the importance of outcomes and considered that research was wasted for only those outcomes the panel confirmed to be of critical importance.
Our study has several limitations. First, we focused only on trials included in Cochrane reviews published between 2010 and 2014. However, it has been reported that Cochrane and non-Cochrane reviews have systemic differences, likely reflective of different methodology [51]. Then, we focused on trials included in the reviews but excluded from at least one meta-analysis. We did not consider the number of trials excluded from the reviews for outcome-related reasons. Therefore, we probably underestimated the proportion of trials excluded from meta-analyses for outcome-related reasons. Despite the recommendation of the Cochrane Handbook to determine trial eligibility independently of outcomes measured (studies should not be excluded just because they provide no usable data), some trials may be excluded for this reason [7]. Thus, although, our sample of RCTs was from a large variety of medical fields, the database used (Cochrane reviews published between 2010 and 2014   and with an SoF table), the limited sample size, the language restrictions, or the expertise of our panel of experts may limit the generalizability of our results. Finally, as a quality measure, 10% of the RCTs were classified independently by two reviewers, since this assessment is challenging and implies some subjectivity.

Conclusions
Our study shows that most RCTs included in our sample of Cochrane reviews did not contribute to all meta-analyses of the most important outcomes mainly because of inadequate planning or incomplete reporting. Such waste could have been partially avoided for 63% of the trials and totally for 30%. We need to accelerate the development and dissemination of COSs and reduce poor outcome reporting to avoid this waste.