Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Empirical evidence of study design biases in nutrition randomised controlled trials: a meta-epidemiological study

BMC Medicine volume 20, Article number: 330 (2022) Cite this article

Abstract

Background

Instruments to critically appraise randomised controlled trials (RCTs) are based on evidence from meta-epidemiological studies. We aim to conduct a meta-epidemiological study on the average bias associated with reported methodological trial characteristics such as random sequence generation, allocation concealment, blinding, incomplete outcome data, selective reporting, and compliance of RCTs in nutrition research.

Methods

We searched the Cochrane Database of Systematic Reviews, for systematic reviews of RCTs, published between 01 January 2010 and 31 December 2019. We combined the estimates of the average bias (e.g. ratio of risk ratios [RRR] or differences in standardised mean differences) in meta-analyses using the random-effects model. Subgroup analyses were conducted to investigate the potential differences among the RCTs with low versus high/unclear risk of bias with respect to the different types of interventions (e.g. micronutrients, fatty acids, dietary approach), outcomes (e.g. mortality, pregnancy outcomes), and type of outcome (objective, subjective). Heterogeneity was assessed through I2 and Ï„2, and prediction intervals were calculated.

Results

We included 27 Cochrane nutrition reviews with 77 meta-analyses (n = 927 RCTs). The available evidence suggests that intervention effect estimates may not be exaggerated in RCTs with high/unclear risk of bias (versus low) judgement for sequence generation (RRR 0.97, 95% CI 0.93 to 1.02; I2 = 28%; τ2 = 0.002), allocation concealment (RRR 1.00, 95% CI 0.96 to 1.04; I2 = 27%; τ2 = 0.001), blinding of participants and personnel (RRR 0.95, 95% CI 0.91 to 1.00; I2 = 23%; τ2 = 0), selective reporting (RRR 0.97, 95% CI 0.92 to 1.02; I2 = 24%; τ2 = 0), and compliance (RRR 0.95, 95% CI 0.89 to 1.02; I2 = 0%; τ2 = 0). Intervention effect estimates seemed to be exaggerated in RCTs with a high/unclear risk of bias judgement for blinding of outcome assessment (RRR 0.81, 95% CI 0.70 to 0.94; I2 = 26%; τ2 = 0.03), which was predominately driven by subjective outcomes, and incomplete outcome data (RRR 0.92, 95% CI 0.88 to 0.97; I2 = 22%; τ2 = 0.001). For continuous outcomes, no differences were observed, except for selective reporting.

Conclusions

On average, most characteristics of nutrition RCTs may not exaggerate intervention effect estimates, but the average bias appears to be greatest in trials of subjective outcomes. Replication of this study is suggested in this field to keep this conclusion updated.

Peer Review reports

Background

Dietary factors are the main risk factor for chronic diseases, especially for type 2 diabetes and cardiovascular diseases, according to the Global Burden of Disease Study [1]. Decision-making in the context of diet-related diseases should be informed by the most reliable available research evidence. Thus, dietary guidance is frequently derived from systematic reviews (SRs) of randomised controlled trials (RCTs) [2]. RCTs are considered to provide the most trustworthy effect estimates of dietary interventions and are the gold standard for establishing causal relations between dietary exposures (e.g. nutrients, foods, or dietary patterns) and health outcomes (e.g. event rates or intermediate disease markers) [3].

Limitations in a RCT, such as in the design, conduct, and analysis of the study, can bias the effect estimates resulting in an overestimation or underestimation of the true intervention effect [4]. This can potentially lead to suboptimal decision-making in nutrition research, e.g. implementing or not a specific dietary intervention for a certain disease. Thus, it is important for the authors of SRs to assess the internal validity of the included primary studies by evaluating the risk of bias (RoB) and incorporating the findings into their statistical analysis, GRADE (Grading of Recommendations, Assessment, Development and Evaluation) assessment and conclusion [5].

The Cochrane tool [6] is the most widely used tool to assess RoB in evidence synthesis in medical and nutrition research [7]. It guides review authors through six pre-defined domains (and methodological characteristics) that need to be considered for the RoB judgement: selection bias (random sequence generation and allocation concealment), performance bias (blinding of participants and personnel), detection bias (blinding of outcome assessment), attrition bias (incomplete outcome data), reporting bias (selective reporting), and other bias [6].

Instruments to critically appraise RCTs are based on evidence from meta-epidemiological studies. To date, several meta-epidemiological studies have investigated the influence of reported study design characteristics on intervention effect estimates in the medical field. They found that trials with inadequate sequence generation and allocation concealment and lack of blinding of participants and outcome assessors yielded more beneficial estimates of intervention effects [8,9,10]. However, to the best of our knowledge, no such systematic evaluation has been conducted for RCTs in the nutritional field.

To close this important research gap, this meta-epidemiological study aims to synthesise evidence on the average bias associated with seven commonly reported methodological trial characteristics in nutrition research. Moreover, we aim to determine whether average bias estimates are relatively similar or not across meta-analyses addressing different interventions, and outcomes. The findings from this study will allow us to better understand and explore the impact of trial characteristics on effects estimates from nutrition RCTs.

Methods

This meta-epidemiological study adheres to the guidelines for reporting meta-epidemiological methodology research [11].

Search strategy and data selection

The sample consists of 33 SRs of RCTs which were identified through a previous meta-epidemiological study [12]. For this project, we searched for SRs of RCTs in the Cochrane Database of Systematic Reviews published between 01 January 2010 and 31 December 2019, which investigated the effects of nutritional interventions (e.g. dietary pattern, micronutrients) on patient-relevant outcomes (e.g. mortality, cardiovascular disease [CVD]). Details of the inclusion and exclusion criteria, as well as the search strategy, are displayed in Additional file 1: Appendix S1-S2.

For each identified Cochrane review, we chose a maximum of six outcomes (maximum three binary and three continuous outcomes) for each given intervention, based on the ranking in the Summary of Findings tables (from top to bottom). Furthermore, we selected only the meta-analyses that contain at least one trial judged as ‘low risk of bias’ and another for ‘unclear or high risk of bias’ for a specific methodological characteristic according to the RoB assessment provided by the original SR authors.

Data extraction

We extracted the following data for each included Cochrane review: name of the first author, year of publication, type of intervention, description of outcome, effect estimates (risk ratio [RR] or mean difference [MD] with the corresponding 95% confidence interval [CI]), and the number of included RCTs. For continuous outcomes, we calculated the standard deviations [SD], and the effect measures were converted to standardised mean differences [SMD] with the transformation formula SMD = MD/SD.

For each RCT included in the eligible meta-analyses of each considered Cochrane review, we additionally extracted all effect estimates with their corresponding 95% CI and the judgements (low, unclear, high) for each methodological trial characteristic of the Cochrane tool [6] (random sequence, generation and allocation concealment, blinding, incomplete outcome data, and selective reporting) as provided by the original SR authors. Although dietary compliance was not suggested in the RoB domains of the Cochrane tool, we extracted judgements if available, due to its importance for nutrition RCTs [13]. Judgements for each methodological trial characteristic were extracted from the risk of bias summary figure and/or the ‘characteristics of studies’ tables in the appendix.

Two authors extracted data using a piloted standardised data collection form. Data extraction was performed by one reviewer (IR) and checked by a second reviewer (JS). Divergences were resolved by consensus, or by consulting a third reviewer (LS).

Statistical analysis

We standardised the direction of effect of the outcomes so that binary effect estimates < 1 (or continuous outcomes < 0) are always expressing a beneficial effect.

Relative effect estimates (quantified as the ratio of risk ratios [RRR] or difference in standardised mean differences [DSMD]) between RCTs with high/unclear and those with low RoB were measured for each methodological trial characteristic. To do so, we re-calculated the effect estimate [RR or MD] for each group of trials included in the same meta-analysis, by pooling first only those RCTs which were judged to be at low RoB according to the methodological trial characteristic under investigation and second only those RCTs with an unclear or high RoB judgement (see Additional file 2: Fig. S1). By using RCTs with low RoB judgements as the reference group, we examined the pooled estimate to see whether there was a relatively larger or smaller estimate coming from RCTs with high/unclear RoB judgements. For example, a RR from high/unclear RoB RCTs of 0.95 and a RR from low RoB RCTs of 0.90 result in a RRR of 1.06, whereas a RR of 1.00 in low RoB RCTs compared to a RR of 1.06 in high/unclear RoB RCTs is also a RRR of 1.06. Therefore, the RRRs should not be interpreted as larger or smaller treatment effects in one RoB category, but only as differences between the two RoB categories, and the direction of difference depends on the direction of effect of the underlying RCTs.

RCTs with high and unclear RoB judgements were grouped together, since the number of available judgements was small (Table 1). This approach was in line with landmark meta-epidemiological studies [8, 10, 14]. We performed meta-analyses to synthesise the relative effect estimates (average bias). Statistical analyses were conducted across all outcomes and for each methodological characteristic independently. When methodological trial characteristics were defined differently across the Cochrane reviews (e.g. ‘blinding’ in general), we excluded these from the analyses. We performed subgroup analyses with respect to the different types of interventions (e.g. dietary approach, fatty acids, micronutrients), and types of outcomes (e.g. mortality, cardiovascular disease, cancer). Moreover, we performed subgroup analyses comparing the objective (e.g. all-cause mortality) versus pregnancy outcomes versus mostly subjective outcomes according to Savovic and colleagues [9, 10].

Table 1 Number of randomised controlled trials included in the data set, by risk of bias judgement
Full size table

We also performed a sensitivity analysis excluding the highly correlated outcomes (Additional file 3: Tables S1-S7). The determination of highly correlated outcomes was based on experts’ opinions and is in line with our previous study [12]. For each intervention, we only chose the outcome that was mentioned first from the Summary of Findings tables of the identified Cochrane reviews. For example, for the intervention of α-linolenic acid, the outcomes CVD, CVD mortality, and coronary heart disease are probably highly correlated. CVD mortality outcome was chosen as it was mentioned first in the Summary of Findings table, while the other two outcomes were excluded. In a post hoc analysis, we conducted a sensitivity analysis including only one outcome per comparison from each included SR. We chose the outcome with the largest number of RCTs included. In case there were two outcomes with the same number of RCTs, we chose the one that was ranked higher in the Summary of Findings table.

We used the random-effects model in meta-analysis to take into account the between-study variability (heterogeneity). We assessed heterogeneity with I2 and Ï„2 measures [15, 16]. The Ï„2 statistic was estimated by the restricted maximum-likelihood (REML) method [17], which performs well for both binary and continuous outcomes [18,19,20]. We also calculated 95% prediction intervals (PI) to provide the range of possible values for the difference between the characteristics of RCTs that might be observed in future comparisons.

We performed post hoc multivariable meta-regressions to account for confounding variables such as sample size and conflicts of interest. For each methodological trial characteristic, we investigated whether ‘sample size’ or reported ‘conflicts of interest’ are associated with the differences in the effect estimates across the groups of low and high/unclear RoB RCTs.

All statistical analyses were conducted using the R package meta (version 5.2–0) [21].

Results

Of the 33 SRs of RCTs identified in our previous study [22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54] including 97 meta-analyses, five SRs [25, 32, 35, 41, 49], including 11 meta-analyses, were excluded since they provide only one RCT for any eligible outcome. Moreover, we excluded nine meta-analyses in five eligible SRs [29, 44, 48, 51, 53] since comparisons were not possible for any methodological trial characteristic. Overall, 27 SRs of RCTs (Cochrane reviews) [22,23,24, 26,27,28,29,30,31, 33, 34, 36,37,38,39,40, 42,43,44,45,46,47,48, 50, 52,53,54] with 77 meta-analyses (n = 927 RCTs) were included. The details of the selection process are displayed in Fig. 1 and Additional file 3: Tables S8-S9 [55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86].

Fig. 1
figure 1

Flow diagram showing study selection process for eligible Cochrane reviews. MA, meta-analyses; RCT, randomised controlled trials; SR, systematic reviews. Reasons for exclusion are displayed in Additional file 3: Tables S8-S9

Full size image

Characteristics of meta-analyses of randomised controlled trials

The meta-analyses included a median of ten RCTs (range: 2 to 64). The interventions evaluated in the identified meta-analyses can be categorised into micronutrients (n = 36), fatty acids (n = 16), dietary approach (n = 14), food groups (n = 5), fibre (n = 4), and food (n = 2).

The intervention was administered in the form of dietary supplements (n = 31), dietary intake (n = 30), or both (n = 16). The outcome clusters included intermediate disease markers (n = 21), CVD (n = 18), pregnancy outcomes (n = 17), all-cause mortality (n = 12), cancer (n = 6), bone health (n = 2), and eye disease (n = 1). Additional file 3: Tables S10-S11 presents the characteristics of all meta-analyses in our sample.

All included SRs assessed the RoB of the primary studies using the criteria of the original version of the Cochrane tool [6]. Among the 77 meta-analyses, 60 were informative for ‘random sequence generation’, 69 for ‘allocation concealment’, 39 for ‘blinding of participants and personnel’, 49 for ‘blinding of outcome assessment’, 60 for ‘incomplete outcome data’, 36 for ‘selective reporting’, and 16 for ‘dietary compliance’. Overall, we included 52 meta-analyses on binary outcomes and 25 on continuous outcomes (Additional file 3: Table S12).

For ‘random sequence generation’, 618 trials (74.4%) included in the meta-analyses were judged as having low RoB by the original review authors (see Table 1). For ‘allocation concealment’, there were 541 trials (58.5%) with a low RoB judgement, for ‘blinding of participants and personnel’ 264 trials (61.4%), ‘blinding of outcomes’ 348 trials (69.6%), ‘incomplete outcome data’ 583 trials (66.9%), and ‘selective reporting’ 369 trials (65.1%). The proportion of trials judged as being at low RoB was lowest for the ‘dietary compliance’ domain (50.9%, 119 trials). In the comparison group enclosing unclear and high RoB trials, most trials (51.6–96.4%) were judged as having an unclear RoB by the original review authors. Most of the trials with a high RoB were found for ‘blinding of participants and personnel’ (78 trials, 18.1%) and ‘incomplete outcome data’ (140 trials, 16.0%).

Average bias associated with the methodological trial characteristics

Tables 2 and 3 present the results of the association of the reported characteristics on intervention effect estimates, from the main analyses, subgroup analyses, and sensitivity analyses. All meta-analyses conducted are illustrated with forest plots in Additional file 2: Figs. S2-S47.

Table 2 Overview of the main results for binary outcomes
Full size table
Table 3 Overview of the main results for continuous outcomes
Full size table

Random sequence

Based on 39 meta-analyses with binary outcomes [22,23,24, 26,27,28,29, 31, 36, 37, 40, 42, 46, 53, 54], high/unclear RoB judgement (versus low) for sequence generation may not exaggerate intervention effect estimates on average (RRR 0.97, 95% CI 0.93 to 1.02; I2 = 28%; τ2 = 0.002; 95% PI 0.88 to 1.08; see Additional file 2: Fig. S2). The sensitivity analyses excluding highly correlated outcomes (Additional file 2: Fig. S3) or including only one outcome per comparison from each included SR (Additional file 2: Fig. S4), and subgroup analyses for the types of intervention and outcomes (Additional file 2: Figs. S5-S7) also yielded no exaggeration of intervention effect estimates on average, comparing RCTs with high/unclear RoB judgement (versus low) for sequence generation.

Based on 21 meta-analyses with continuous outcomes [22, 24, 30, 33, 34, 39, 43, 45, 47, 48, 52], no exaggeration of intervention effect estimates on average was detected (Additional file 2: Fig. S8), also when excluding highly correlated outcomes (Additional file 2: Fig. S9) or including only one outcome per comparison from each SR (Additional file 2: Fig. S10).

Allocation concealment

Based on 46 meta-analyses with binary outcomes [22,23,24, 26,27,28,29, 31, 36,37,38, 40, 42, 45, 46, 50, 53, 54], high/unclear RoB (versus low) judgement for allocation concealment may not exaggerate the intervention effect estimates on average (RRR 1.00, 95% CI 0.96 to 1.04; I2 = 27%; τ2 = 0.001; 95% PI 0.92 to 1.08; see Additional file 2: Fig. S11). This finding was confirmed in a sensitivity analysis by excluding highly correlated outcomes (Additional file 2: Fig. S12) or including only one outcome per comparison from each SR (Additional file 2: Fig. S13), and the additional subgroup analyses for the different types of interventions and outcomes (Additional file 2: Figs. S14-S16).

For continuous outcomes [22, 24, 30, 34, 37, 39, 43, 45, 47, 48, 50, 52], the average effect of high/unclear RoB (versus low) judgement for allocation concealment was close to the null (DSMD 0.03, 95% CI − 0.07 to 0.12; I2 = 51%; τ2 = 0.03; 95% PI − 0.32 to 0.37; see Additional file 2: Fig. S17), also in the sensitivity analysis with excluding highly correlated outcomes (Additional file 2: Fig. S18) or including only one outcome per comparison from each SR (Additional file 2: Fig. S19).

Blinding of participants and personnel

A total of 29 meta-analyses with binary outcomes [22,23,24, 28, 36, 40, 42, 45, 46, 54] contributed to the analysis of RCTs with high/unclear RoB (versus low) judgement for blinding of participants and personnel. No exaggeration of intervention effect estimates on the average was observed in the main analysis (RRR 0.95, 95% CI 0.91 to 1.00; I2 = 23%; τ2 = 0; 95% PI 0.90 to 1.01; see Additional file 2: Fig. S20) and by excluding highly correlated outcomes (Additional file 2: Fig. S21) or including only one outcome per comparison from each SR (Additional file 2: Fig. S22). When analyses were stratified according to the type of intervention or subjective and objective outcomes, the average effect of bias associated with high/unclear RoB (versus low) judgement for blinding of participants and personnel was not exaggerated (Additional file 2: Figs. S23-S24). However, in subgroup analysis stratified by the cluster of outcomes, the outcome ‘cancer’ showed an average of 32% exaggeration (RRR 0.68, 95% CI 0.51 to 0.91; I2 = 0%; τ2 = 0; 95% PI 0.36 to 1.30; see Additional file 2: Fig. S25). It is worth notable that this finding is limited by the low number of included meta-analyses (n = 4).

Based on ten meta-analyses with continuous outcomes [22, 24, 30, 34, 45], no exaggeration of intervention effect estimates on average was detected, in both the main and the sensitivity analyses (Additional file 2: Figs. S26-S28).

Blinding of outcome assessment

In the analysis of 28 meta-analyses with binary outcomes, the influence of high/unclear RoB (versus low) judgement for blinding of outcome assessors on the effect estimates was substantial (RRR 0.81, 95% CI 0.70 to 0.94; I2 = 26%; τ2 = 0.03; 95% PI 0.54 to 1.23; see Additional file 2: Fig. S29). This finding was confirmed in the sensitivity analyses excluding highly correlated outcomes (Additional file 2: Fig. S30), but not when including only one outcome per comparison from each SR (RRR 0.88, 95% CI 0.73 to 1.07; I2 = 17%; τ2 = 0.03; 95% PI 0.58 to 1.35; see Additional file 2: Fig. S31).

Subgroup analysis stratified by the type of interventions (Additional file 2: Fig. S32) indicated that the exaggeration of intervention effect estimates on average was driven by ‘fatty acids’ interventions (RRR 0.62, 95% CI 0.47 to 0.82; I2 = 0%; τ2 = 0; 95% PI 0.45 to 0.86) and differed substantially from ‘micronutrients’ interventions (RRR 0.90, 95% CI 0.77 to 1.05; I2 = 37%; τ2 = 0.02; 95% PI 0.62 to 1.29). Moreover, when analyses were stratified according to the type of outcomes (Additional file 2: Figs. S33-S34), exaggeration of intervention effect estimates on average was driven by ‘mostly subjectively assessed’ outcomes (RRR 0.74, 95% CI 0.59 to 0.93; I2 = 0%; τ2 = 0.02; 95% PI 0.48 to 1.13). It was not statistically significant for all-cause mortality (RRR 0.83, 95% CI 0.64 to 1.08; I2 = 0%; τ2 = 0.02; 95% PI 0.46 to 1.51) or pregnancy outcomes (RRR 0.86, 95% CI 0.62 to 1.18; I2 = 49%; τ2 = 0.11; 95% PI 0.37 to 1.98).

For continuous outcomes [22, 24, 30, 33, 34, 43, 45, 47, 48, 50], the average effect of high/unclear RoB judgements for blinding of outcome assessment was close to the null in the main analysis (DSMD 0.07, 95% CI − 0.02 to 0.16; I2 = 33%; τ2 = 0.01; 95% PI − 0.19 to 0.33; see Additional file 2: Fig. S35) and when excluding highly correlated outcomes (Additional file 2: Fig. S36) or including only one outcome per comparison from each SR (Additional file 2: Fig. S37).

Incomplete outcome data

Based on our analysis of 37 meta-analyses [22,23,24, 27,28,29, 36,37,38, 40, 42, 44, 45, 50], high/unclear RoB (versus low) judgement for incomplete outcome data may exaggerate the intervention effect estimates, on average by 8% (RRR 0.92, 95% CI 0.88 to 0.97; I2 = 22%; τ2 = 0.001; 95% PI 0.85 to 1.00; see Additional file 2: Fig. S38). However, this was also the case when excluding highly correlated outcomes in the sensitivity analysis (RRR 0.93, 95% CI 0.88 to 0.99; I2 = 19%; τ2 = 0.001; 95% PI 0.84 to 1.03; see Additional file 2: Fig. S39) or including only one outcome per comparison from each SR (RRR 0.92, 95% CI 0.86 to 0.98; I2 = 38%; τ2 = 0.002; 95% PI 0.82 to 1.03; see Additional file 2: Fig. S40).

When analyses were stratified according to type of interventions or type of outcomes, the average effect of bias associated with high/unclear RoB judgement for incomplete outcome data was largest for ‘fatty acids’ (RRR 0.82, 95% CI 0.72 to 0.94; I2 = 0%; τ2 = 0; 95% PI 0.71 to 0.96; see Additional file 2: Fig. S41) and ‘all-cause mortality’ (RRR 0.91, 95% CI 0.85 to 0.98; I2 = 0%; τ2 = 0; 95% PI 0.84 to 0.99; see Additional file 2: Fig. S42-S43), respectively. The subgroup analysis focusing on cancer as an outcome showed an average of 43% exaggeration (RRR 0.57, 95% CI 0.34 to 0.95; I2 = 0%; τ2 = 0; 95% PI 0.02 to 16.15; see Additional file 2: Fig. S42), but this finding is limited by the low number of included meta-analyses (n = 3).

Based on 23 meta-analyses with continuous outcomes [22, 24, 30, 34, 37, 39, 43, 45, 47, 48, 50, 52], no exaggeration of intervention effect estimates on average was observed (Additional file 2: Fig. S44), also in the sensitivity analyses when excluding highly correlated outcomes or including only one outcome per comparison from each SR (Additional file 2: Figs. S45-S46).

Selective reporting

Based on the meta-analysis of 23 meta-analyses [22,23,24, 27,28,29, 36, 40, 45, 50], there was no evidence that trials rated at high/unclear (versus low) RoB have different effect estimates due to selective reporting (RRR 0.97, 95% CI 0.92 to 1.02; I2 = 24%; τ2 = 0; 95% PI 0.92 to 1.02; see Additional file 2: Fig. S47). This finding was confirmed in sensitivity analyses and also across all subgroup analyses (Additional file 2: Figs. S48-S52).

For the continuous outcomes [22, 24, 39, 43, 45, 48, 50, 52], an exaggeration of intervention effect estimates on average was observed (DSMD − 0.10, 95% CI − 0.18 to − 0.03; I2 = 0%; τ2 = 0; 95% PI − 0.19 to − 0.02; see Additional file 2: Fig. S53). This was also the case when only outcome per intervention was included in the sensitivity analysis (Additional file 2: Fig. S54). However, the sensitivity analysis excluding highly correlated outcomes does not confirm this finding (DSMD − 0.08, 95% CI − 0.18 to 0.03; I2 = 16%; τ2 = 0.002; 95% PI − 0.25 to 0.10; see Additional file 2: Fig. S55).

Dietary compliance

Based on 15 meta-analyses with binary outcomes [22,23,24, 38, 40], there was no evidence that trials rated at high/unclear (versus low) RoB for dietary compliance have different effect estimates (RRR 0.95, 95% CI 0.89 to 1.02; I2 = 0%; τ2 = 0.00; 95% PI 0.88 to 1.03; see Additional file 2: Fig. S56). This finding was confirmed in the sensitivity and subgroup analyses (Additional file 2: Figs. S57-S61).

Due to the low number of meta-analyses available (n = 1) [22], we did not perform any analysis for continuous outcomes for dietary compliance.

Meta-regression analyses did not show any statistically significant effects of the confounding variables ‘sample size’ and ‘conflicts of interest’ on the pooled estimates, in any of the methodological trial characteristics (Additional file 3: Tables S13-S25).

Discussion

Summary of findings

This meta-epidemiological study of 77 meta-analyses synthesised evidence on the average bias associated with reported methodological trial characteristics such as random sequence generation, allocation concealment, blinding, incomplete outcome data, selective reporting, and dietary compliance of RCTs in the nutritional field. The main findings suggest that most characteristics of nutrition RCTs considered in our sample may not exaggerate the average intervention effect estimates. However, we observed that intervention effect estimates may exaggerate on average by 18% in trials rated at high/unclear (versus low) RoB for blinding of outcome assessment. For this methodological trial characteristic, the average bias appears to be larger for subjective outcomes, whereas for all-cause mortality and pregnancy outcomes, no association was observed. Moreover, high/unclear (versus low) RoB judgements for incomplete outcome data may exaggerate intervention effect estimates on average by 9%, which was confirmed in various subgroup and sensitivity analyses. Overall, the statistical heterogeneity was low, which was confirmed by a narrow 95% PI. For a pooled estimate of continuous outcomes, no differences were observed between high/unclear (versus low) RoB methodological trial characteristics, except for selective reporting in the main analysis.

Comparison with other studies

We did not identify any similar empirical study investigating the impact of study design biases on intervention effects across meta-analyses of nutrition RCTs. However, several meta-epidemiological studies have been conducted in the medical research field so far. Savovic and colleagues [14], for instance, showed that intervention effect estimates seemed to be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (by 11%), allocation concealment (by 7%), or double-blinding (by 13%). This exaggeration was, however, not statistically significant for mortality and other objective outcomes, which is in line with our findings. Similarly, Page and colleagues [8] synthesised evidence from 24 meta-epidemiological studies and observed that certain characteristics of RCTs such as sequence generation, allocation concealment, and double blinding may exaggerate the intervention effect estimates for subjective compared with objective outcomes. Moreover, inadequate/unclear blinding of the outcome assessor (versus adequate blinding) may exaggerate the intervention for subjective outcomes, which is in line with our findings. Regarding the detected exaggeration of intervention effects observed in RCTs with high or unclear RoB judgements for incomplete outcome data, previous studies reported inconsistent findings: Savovic and colleagues [10] found little evidence that intervention effects were exaggerated in trials which were judged to be at high or unclear RoB for incomplete outcome data, whereas Abraha and colleagues [87] observed an exaggeration.

In congruence with our findings, two small meta-epidemiological studies concluded that there was no convincing evidence that trials rated at high/unclear (versus low) risk of bias due to selective reporting have different effect estimates [88, 89].

A recent meta-epidemiological study investigated the impact of blinding and found no evidence for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors [90]. This study result is in agreement with our findings on double blinding of participants and personnel but in disagreement with our findings on blinding of outcome assessment for subjective outcomes.

Implications for the research nutrition field

Systematic evidence syntheses based on high-quality evidence are highly needed to generate trustworthy dietary guidance. Ideally, these should base on direct evidence from RCTs with a low risk of bias. In the field of nutrition research, however, trialists face a number of methodological challenges that are difficult to resolve [13]. Whereas certain trial characteristics such as random sequence generation, allocation concealment, and blinding of outcome assessment can be performed for most RCTs, irrespective of the research field, others are context-specific and are more difficult to incorporate such as blinding of participants and personnel, incomplete outcome data, and dietary compliance. For instance, people are in general aware of what they consume. Therefore, it is often impossible to blind study participants, except for RCTs with dietary supplements. Moreover, longer-term nutrition research trials often face attrition of participants (40–50% dropout rate is fairly common in free-living populations [91]) and low compliance to a specific dietary regimen is often observed [92].

Nonetheless, in our meta-epidemiological study, we observed that on average, most characteristics of nutrition RCTs may often not exaggerate intervention effect estimates. Of note, our sample is mainly based on interventions with dietary supplements where blinding is possible and more than 60% of the included RCTs were judged as low RoB for blinding of participants and personnel. Moreover, two-third of the RCTs in our sample were judged with low RoB for incomplete outcome data. Attrition seems to be lower in interventions with supplements, as the lifestyle behaviour change introduced by the intervention is small [93].

Regarding dietary compliance, about half of the RCTs (n = 119) included in our sample were judged with a low RoB, and all interventions based on the supplementation (and partly on dietary intake) of fatty acids or micronutrients. Surprisingly, RCTs with high/unclear RoB for dietary compliance showed no differences compared to low RoB trials. This could possibly be explained by the fact that most of the included RCTs were rated as ‘unclear’ (n = 100) and only few (n = 15) at ‘high’ RoB. Furthermore, the number of included studies which were informative about dietary compliance was small. Dietary compliance is not a specific domain in the original version of the Cochrane tool [6]. The authors of the Cochrane reviews included in this study assessed dietary compliance by introducing a new RoB domain [23, 40], within ‘other sources of bias’ [22, 38] or did not provide any details on the assessment in the methods Sect. [24]. Moreover, definitions of ‘dietary compliance’ differ between the included reviews: some authors, for instance, based their judgement on thresholds, whereas others on reported differences in intake or biomarkers between the control and treatment groups (see Additional file 3: Table S26). In the most recent version of the Cochrane tool (RoB 2.0), however, compliance to a specific dietary intervention will be assessed within the domain ‘assessing deviations from intended interventions’ [94].

In order to highlight the methodological weaknesses of included RCTs and address the study design bias, it is crucial for the authors of SRs to take RoB assessments into account in the statistical analysis, in the GRADE assessment, and conclusions of the SRs.

Strengths and limitations

This study has several strengths. First, to the best of our knowledge, no similar study has been conducted in the nutritional field so far. Second, we analysed a large sample of diet-disease effects (n = 77 meta-analyses). Third, we selected meta-analyses of RCTs published as Cochrane reviews, which are internationally recognised as the highest standard in evidence-based healthcare. The high methodological quality of Cochrane nutrition reviews was confirmed recently [95]. Fourth, our study was based on meta-analyses of both binary and continuous outcomes.

However, our study has also several limitations: First, although we pooled a large sample of diet-disease associations, our sample may not be representative of all meta-analyses, and the totality of the evidence of available diet-disease associations might provide different results. Second, some trials may be included multiple times in the main analysis due to the different outcomes. However, in our post hoc sensitivity analyses, we also took a conservative approach by including only one outcome per comparison from each included SR. These sensitivity analyses confirmed all the findings of the main analysis. Third, we did not use the most recent version of the RoB tool [94] since we extracted the RoB judgements as provided by the original review authors. The usage of this tool, which was published in 2019 [94], would have made it impossible to generate an adequate sample. Additional considerations added to the revised tool were thus not considered for our analysis. For example, differences in baseline characteristics or the effect of adhering to intervention are not addressed by the original tool [6]. Fourth, we did not verify the RoB assessment of the included Cochrane reviews including the judgements made for ‘dietary compliance’. Finally, the RoB assessment is based on the reported methodological characteristics of the included primary studies. It remains however unclear, whether trials that were classified as ‘unclear’ might have been in fact adequately conducted. Since we merged both, RCTs with high and unclear risk of bias, we might have misclassified some of the trials which might be at low risk of bias. Due to these limitations, our findings need to be interpreted with caution.

Conclusions

On average, most characteristics of nutrition RCTs may not exaggerate intervention effect estimates, but the average bias appears to be greatest in trials of subjective outcomes. These results could reflect that certain trial characteristics are less important than often believed for RCTs in the nutritional field. We suggest the replication of this meta-epidemiological study in the nearly future to keep this evidence up to date, and to perform it with a larger sample to gain more insight into the impact of methodological quality on the exaggeration of effect estimates. In particular, the impact of dietary compliance with regard to subjectively assessed outcomes needs to be investigated. Moreover, future meta-epidemiological analyses should focus on the methodological trial characteristics as defined by the most recent version of the RoB tool [94]. We conclude that the RoB assessment should remain a methodological safeguard in nutrition trials.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its additional files.

Abbreviations

CI:

Confidence interval

CVD:

Cardiovascular disease

DSMD:

Difference in standardised mean differences

GRADE:

Grading of Recommendations, Assessment, Development and Evaluation

I 2 :

Heterogeneity measure

MD:

Mean difference

n/a:

Not applicable

PI:

Prediction interval

PICO:

Patient/Population, Intervention, Comparator, Outcome

RCT:

Randomised controlled trial

REML:

Restricted maximum-likelihood method

RoB:

Risk of bias

RR:

Risk ratio

RRR:

Ratio of risk ratios

SMD:

Standardised mean difference

SR:

Systematic review

Ï„2:

Heterogeneity value with restricted maximum-likelihood estimation method

References

  1. Afshin A, Sur PJ, Fay KA, Cornaby L, Ferrara G, Salama JS, et al. Health effects of dietary risks in 195 countries, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. The Lancet. 2019;393(10184):1958–72.

    Article  Google Scholar 

  2. Johnston BC, Seivenpiper JL, Vernooij RWM, de Souza RJ, Jenkins DJA, Zeraatkar D, et al. The philosophy of evidence-based principles and practice in nutrition. Mayo Clin Proc Innov Qual Outcomes. 2019;3(2):189–99.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Lichtenstein AH, Petersen K, Barger K, Hansen KE, Anderson CAM, Baer DJ, et al. Perspective: design and conduct of human nutrition randomized controlled trials. Adv Nutr. 2021;12(1):4–20.

    Article  PubMed  Google Scholar 

  4. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336(7644):601–5.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane. 2022. Available from www.training.cochrane.org/handbook.

  6. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Schwingshackl L, Knüppel S, Schwedhelm C, Hoffmann G, Missbach B, Stelmach-Mardas M, et al. Perspective: NutriGrade: a scoring system to assess and judge the meta-evidence of randomized controlled trials and cohort studies in nutrition research. Adv Nutr. 2016;7(6):994–1004.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Page MJ, Higgins JP, Clayton G, Sterne JA, Hrobjartsson A, Savovic J. Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies. PLoS One. 2016;11(7):e0159267.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  9. Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012;16(35):1–82.

    Article  PubMed  Google Scholar 

  10. Savovic J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, et al. Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES Meta-Epidemiologic Study. Am J Epidemiol. 2018;187(5):1113–22.

    Article  PubMed  Google Scholar 

  11. Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Schwingshackl L, Balduzzi S, Beyerbach J, Bröckelmann N, Werner SS, Zähringer J, et al. Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study. BMJ. 2021;374:n1864.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Schwingshackl L, Schünemann HJ, Meerpohl JJ. Improving the trustworthiness of findings from nutrition evidence syntheses: assessing risk of bias and rating the certainty of evidence. Eur J Nutr. 2021;60(6):2893–903.

    Article  PubMed  Google Scholar 

  14. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157(6):429–38.

    Article  PubMed  Google Scholar 

  15. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342: d549.

    Article  PubMed  Google Scholar 

  17. Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat. 2005;30(3):261–93.

    Article  Google Scholar 

  18. Langan D, Higgins JPT, Jackson D, Bowden J, Veroniki AA, Kontopantelis E, et al. A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res Synth Methods. 2019;10(1):83–98.

    Article  PubMed  Google Scholar 

  19. Petropoulou M, Mavridis D. A comparison of 20 heterogeneity variance estimators in statistical synthesis of results from studies: a simulation study. Stat Med. 2017;36(27):4266–80.

    Article  PubMed  Google Scholar 

  20. Veroniki AA, Jackson D, Viechtbauer W, Bender R, Bowden J, Knapp G, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7(1):55–79.

    Article  PubMed  Google Scholar 

  21. Balduzzi S, Rucker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health. 2019;22(4):153–60.

    Article  PubMed  Google Scholar 

  22. Abdelhamid AS, Brown TJ, Brainard JS, Biswas P, Thorpe GC, Moore HJ, et al. Omega-3 fatty acids for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2018;11:CD003177.

    PubMed  Google Scholar 

  23. Abdelhamid AS, Martin N, Bridges C, Brainard JS, Wang X, Brown TJ, et al. Polyunsaturated fatty acids for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2018;11:CD012345.

    PubMed  Google Scholar 

  24. Adler AJ, Taylor F, Martin N, Gottlieb S, Taylor RS, Ebrahim S. Reduced dietary salt for the prevention of cardiovascular disease. Cochrane Database Syst Rev. 2014;12:CD009217.

    Google Scholar 

  25. Al-Khudairy L, Flowers N, Wheelhouse R, Ghannam O, Hartley L, Stranges S, et al. Vitamin C supplementation for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2017;3:CD011114.

    PubMed  Google Scholar 

  26. Avenell A, Mak JC, O’Connell D. Vitamin D and vitamin D analogues for preventing fractures in post-menopausal women and older men. Cochrane Database Syst Rev. 2014;4:CD000227.

    Google Scholar 

  27. Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev. 2012;3:CD007176.

    Google Scholar 

  28. Bjelakovic G, Gluud LL, Nikolova D, Whitfield K, Krstic G, Wetterslev J, et al. Vitamin D supplementation for prevention of cancer in adults. Cochrane Database Syst Rev. 2014;6:CD007469.

    Google Scholar 

  29. Bjelakovic G, Gluud LL, Nikolova D, Whitfield K, Wetterslev J, Simonetti RG, et al. Vitamin D supplementation for prevention of mortality in adults. Cochrane Database Syst Rev. 2014;1:CD007470.

    Google Scholar 

  30. Cormick G, Ciapponi A, Cafferata ML, Belizan JM. Calcium supplementation for prevention of primary hypertension. Cochrane Database Syst Rev. 2015;6:CD010037.

    Google Scholar 

  31. De-Regil LM, Pena-Rosas JP, Fernandez-Gaxiola AC, Rayco-Solon P. Effects and safety of periconceptional oral folate supplementation for preventing birth defects. Cochrane Database Syst Rev. 2015;12:CD007950.

    Google Scholar 

  32. El Dib R, Gameiro OL, Ogata MS, Modolo NS, Braz LG, Jorge EC, et al. Zinc supplementation for the prevention of type 2 diabetes mellitus in adults with insulin resistance. Cochrane Database Syst Rev. 2015;5:CD005525.

    Google Scholar 

  33. Hartley L, Igbinedion E, Holmes J, Flowers N, Thorogood M, Clarke A, et al. Increased consumption of fruit and vegetables for the primary prevention of cardiovascular diseases. Cochrane Database Syst Rev. 2013;6:CD009874.

    Google Scholar 

  34. Hartley L, May MD, Loveman E, Colquitt JL, Rees K. Dietary fibre for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2016;1:CD011472.

    Google Scholar 

  35. Hemmingsen B, Gimenez-Perez G, Mauricio D, Roque IFM, Metzendorf MI, Richter B. Diet, physical activity or both for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk of developing type 2 diabetes mellitus. Cochrane Database Syst Rev. 2017;12:CD003054.

    PubMed  Google Scholar 

  36. Hofmeyr GJ, Lawrie TA, Atallah AN, Torloni MR. Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems. Cochrane Database Syst Rev. 2018;10:CD001059.

    PubMed  Google Scholar 

  37. Hooper L, Summerbell CD, Thompson R, Sills D, Roberts FG, Moore HJ, et al. Reduced or modified dietary fat for preventing cardiovascular disease. Cochrane Database Syst Rev. 2012;5:CD002137.

    Google Scholar 

  38. Hooper L, Martin N, Abdelhamid A, Davey Smith G. Reduction in saturated fat intake for cardiovascular disease. Cochrane Database Syst Rev. 2015;6:CD011737.

    Google Scholar 

  39. Hooper L, Abdelhamid A, Bunn D, Brown T, Summerbell CD, Skeaff CM. Effects of total fat intake on body weight. Cochrane Database Syst Rev. 2015;8:CD011834.

    Google Scholar 

  40. Hooper L, Al-Khudairy L, Abdelhamid AS, Rees K, Brainard JS, Brown TJ, et al. Omega-6 fats for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2018;11:CD011094.

    PubMed  Google Scholar 

  41. Jin H, Leng Q, Li C. Dietary flavonoid for preventing colorectal neoplasms. Cochrane Database Syst Rev. 2012;8:CD009350.

    Google Scholar 

  42. Keats EC, Haider BA, Tam E, Bhutta ZA. Multiple-micronutrient supplementation for women during pregnancy. Cochrane Database Syst Rev. 2019;3:CD004905.

    PubMed  Google Scholar 

  43. Kelly SA, Hartley L, Loveman E, Colquitt JL, Jones HM, Al-Khudairy L, et al. Whole grain cereals for the primary or secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2017;8:CD005051.

    PubMed  Google Scholar 

  44. Mathew MC, Ervin AM, Tao J, Davis RM. Antioxidant vitamin supplementation for preventing and slowing the progression of age-related cataract. Cochrane Database Syst Rev. 2012;6:CD004567.

    Google Scholar 

  45. Palacios C, Trak-Fellermeier MA, Martinez RX, Lopez-Perez L, Lips P, Salisi JA, et al. Regimens of vitamin D supplementation for women during pregnancy. Cochrane Database Syst Rev. 2019;10:CD013446.

    PubMed  Google Scholar 

  46. Rees K, Hartley L, Day C, Flowers N, Clarke A, Stranges S. Selenium supplementation for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2013;1:CD009671.

    Google Scholar 

  47. Rees K, Dyakova M, Wilson N, Ward K, Thorogood M, Brunner E. Dietary advice for reducing cardiovascular risk. Cochrane Database Syst Rev. 2013;12:CD002128.

    Google Scholar 

  48. Rees K, Takeda A, Martin N, Ellis L, Wijesekara D, Vepa A, et al. Mediterranean-style diet for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2019;3:CD009825.

    PubMed  Google Scholar 

  49. Rutjes AW, Denton DA, Di Nisio M, Chong LY, Abraham RP, Al-Assaf AS, et al. Vitamin and mineral supplementation for maintaining cognitive function in cognitively healthy people in mid and late life. Cochrane Database Syst Rev. 2018;12:CD011906.

    PubMed  Google Scholar 

  50. Tieu J, Shepherd E, Middleton P, Crowther CA. Dietary advice interventions in pregnancy for preventing gestational diabetes mellitus. Cochrane Database Syst Rev. 2017;1:CD006674.

    PubMed  Google Scholar 

  51. Sydenham E, Dangour AD, Lim WS. Omega 3 fatty acid for the prevention of cognitive decline and dementia. Cochrane Database Syst Rev. 2012;6:CD005379.

    Google Scholar 

  52. Usinger L, Reimer C, Ibsen H. Fermented milk for hypertension. Cochrane Database Syst Rev. 2012;4:CD008118.

    Google Scholar 

  53. Vinceti M, Filippini T, Del Giovane C, Dennert G, Zwahlen M, Brinkman M, et al. Selenium for preventing cancer. Cochrane Database Syst Rev. 2018;1:CD005195.

    PubMed  Google Scholar 

  54. Yao Y, Suo T, Andersson R, Cao Y, Wang C, Lu J, et al. Dietary fibre for the prevention of recurrent colorectal adenomas and carcinomas. Cochrane Database Syst Rev. 2017;1:CD003430.

    PubMed  Google Scholar 

  55. Balogun OO, da Silva Lopes K, Ota E, Takemoto Y, Rumbold A, Takegata M, et al. Vitamin supplementation for preventing miscarriage. Cochrane Database Syst Rev. 2016;5:CD004073.

    Google Scholar 

  56. Buppasiri P, Lumbiganon P, Thinkhamrop J, Ngamjarus C, Laopaiboon M, Medley N. Calcium supplementation (other than for preventing or treating hypertension) for improving pregnancy and infant outcomes. Cochrane Database Syst Rev. 2015;2:CD007079.

    Google Scholar 

  57. Evans JR, Lawrenson JG. Antioxidant vitamin and mineral supplements for preventing age-related macular degeneration. Cochrane Database Syst Rev. 2017;7:CD000253.

    PubMed  Google Scholar 

  58. Harding KB, Peña-Rosas JP, Webster AC, Yap CM, Payne BA, Ota E, et al. Iodine supplementation for women during the preconception, pregnancy and postpartum period. Cochrane Database Syst Rev. 2017;3:CD011761.

    PubMed  Google Scholar 

  59. Hemilä H, Louhiala P. Vitamin C for preventing and treating pneumonia. Cochrane Database Syst Rev. 2013;8:CD005532.

    Google Scholar 

  60. Imdad A, Ahmed Z, Bhutta ZA. Vitamin A supplementation for the prevention of morbidity and mortality in infants one to six months of age. Cochrane Database Syst Rev. 2016;9:CD007480.

    PubMed  Google Scholar 

  61. Imdad A, Mayo-Wilson E, Herzer K, Bhutta ZA. Vitamin A supplementation for preventing morbidity and mortality in children from six months to five years of age. Cochrane Database Syst Rev. 2017;3:CD008524.

    PubMed  Google Scholar 

  62. Lassi ZS, Moin A, Bhutta ZA. Zinc supplementation for the prevention of pneumonia in children aged 2 months to 59 months. Cochrane Database Syst Rev. 2016;12:CD005978.

    PubMed  Google Scholar 

  63. Martí-Carvajal AJ, Solà I, Lathyris D, Dayer M. Homocysteine-lowering interventions for preventing cardiovascular events. Cochrane Database Syst Rev. 2017;8:CD006612.

    PubMed  Google Scholar 

  64. Mayo-Wilson E, Junior JA, Imdad A, Dean S, Chan XH, Chan ES, et al. Zinc supplementation for preventing mortality, morbidity, and growth failure in children aged 6 months to 12 years of age. Cochrane Database Syst Rev. 2014;5:CD009384.

    Google Scholar 

  65. Oliveira JM, Allert R, East CE. Vitamin A supplementation for postpartum women. Cochrane Database Syst Rev. 2016;3:CD005944.

    PubMed  Google Scholar 

  66. Salam RA, Zuberi NF, Bhutta ZA. Pyridoxine (vitamin B6) supplementation during pregnancy or labour for maternal and neonatal outcomes. Cochrane Database Syst Rev. 2015;6:CD000179.

    Google Scholar 

  67. Schwenger EM, Tejani AM, Loewen PS. Probiotics for preventing urinary tract infections in adults and children. Cochrane Database Syst Rev. 2015;12:CD008772.

    Google Scholar 

  68. Suchdev PS, Peña-Rosas JP, De-Regil LM. Multiple micronutrient powders for home (point-of-use) fortification of foods in pregnant women. Cochrane Database Syst Rev. 2015;6:CD011158.

    Google Scholar 

  69. Barrett HL, Dekker Nitert M, Conwell LS, Callaway LK. Probiotics for preventing gestational diabetes. Cochrane Database Syst Rev. 2014;2:CD00995.

    Google Scholar 

  70. Chen N, Yang M, Zhou M, Xiao J, Guo J, He L. L-carnitine for cognitive enhancement in people without cognitive impairment. Cochrane Database Syst Rev. 2017;3:CD009374.

    PubMed  Google Scholar 

  71. Crawford TJ, Crowther CA, Alsweiler J, Brown J. Antenatal dietary supplementation with myo-inositol in women during pregnancy for preventing gestational diabetes. Cochrane Database Syst Rev. 2015;12:CD011507.

    Google Scholar 

  72. Evans JR, Lawrenson JG. Antioxidant vitamin and mineral supplements for slowing the progression of age-related macular degeneration. Cochrane Database Syst Rev. 2017;7:CD000254.

    PubMed  Google Scholar 

  73. Graudal NA, Hubeck-Graudal T, Jurgens G. Effects of low sodium diet versus high sodium diet on blood pressure, renin, aldosterone, catecholamines, cholesterol, and triglyceride. Cochrane Database Syst Rev. 2017;4:CD004022.

    PubMed  Google Scholar 

  74. Hartley L, Clar C, Ghannam O, Flowers N, Stranges S, Rees K. Vitamin K for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2015;9:CD011148.

    Google Scholar 

  75. He FJ, Li J, MacGregor GA. Effect of longer-term modest salt reduction on blood pressure. Cochrane Database Syst Rev. 2013;4:CD004937.

    Google Scholar 

  76. Makrides M, Crosby DD, Bain E, Crowther CA. Magnesium supplementation in pregnancy. Cochrane Database Syst Rev. 2014;4:CD000937.

    Google Scholar 

  77. Martin N, Germanò R, Hartley L, Adler AJ, Rees K. Nut consumption for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2015;9:CD011583.

    Google Scholar 

  78. Middleton P, Gomersall JC, Gould JF, Shepherd E, Olsen SF, Makrides M. Omega-3 fatty acid addition during pregnancy. Cochrane Database Syst Rev. 2018;11:CD003402.

    PubMed  Google Scholar 

  79. Muktabhant B, Lawrie TA, Lumbiganon P, Laopaiboon M. Diet or exercise, or both, for preventing excessive weight gain in pregnancy. Cochrane Database Syst Rev. 2015;6:CD007145.

    Google Scholar 

  80. Lawrenson JG, Evans JR. Omega 3 fatty acids for preventing or slowing the progression of age-related macular degeneration. Cochrane Database Syst Rev. 2015;4:CD010015.

    Google Scholar 

  81. Low MS, Speedy J, Styles CE, De-Regil LM, Pasricha SR. Daily iron supplementation for improving anaemia, iron status and health in menstruating women. Cochrane Database Syst Rev. 2016;4:CD009747.

    PubMed  Google Scholar 

  82. Ota E, Mori R, Middleton P, Tobe-Gai R, Mahomed K, Miyazaki C, et al. Zinc supplementation for improving pregnancy and infant outcome. Cochrane Database Syst Rev. 2015;2:CD000230.

    Google Scholar 

  83. Peña-Rosas JP, De-Regil LM, Garcia-Casal MN, Dowswell T. Daily oral iron supplementation during pregnancy. Cochrane Database Syst Rev. 2015;7:CD004736.

    Google Scholar 

  84. Ried K, Sullivan TR, Fakler P, Frank OR, Stocks NP. Effect of cocoa on blood pressure. Cochrane Database Syst Rev. 2012;8:CD008893.

    Google Scholar 

  85. Rumbold A, Ota E, Nagata C, Shahrook S, Crowther CA. Vitamin C supplementation in pregnancy. Cochrane Database Syst Rev. 2015;9:CD004072.

    Google Scholar 

  86. Shepherd E, Gomersall JC, Tieu J, Han S, Crowther CA, Middleton P. Combined diet and exercise interventions for preventing gestational diabetes mellitus. Cochrane Database Syst Rev. 2017;11:CD010443.

    PubMed  Google Scholar 

  87. Abraha I, Cherubini A, Cozzolino F, De Florio R, Luchetta ML, Rimland JM, et al. Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study. BMJ. 2015;350:h2445.

    Article  PubMed  PubMed Central  Google Scholar 

  88. Bialy L, Vandermeer B, Lacaze-Masmonteil T, Dryden DM, Hartling L. A meta-epidemiological study to examine the association between bias and treatment effects in neonatal trials. Evid Based Child Health. 2014;9(4):1052–9.

    Article  PubMed  Google Scholar 

  89. Hartling L, Hamm MP, Fernandes RM, Dryden DM, Vandermeer B. Quantifying bias in randomized controlled trials in child health: a meta-epidemiological study. PLoS ONE. 2014;9(2):e88008.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  90. Moustgaard H, Clayton GL, Jones HE, Boutron I, Jørgensen L, Laursen DRT, et al. Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ. 2020;368:l6802.

    Article  PubMed  PubMed Central  Google Scholar 

  91. Dansinger ML, Gleason JA, Griffith JL, Selker HP, Schaefer EJ. Comparison of the Atkins, Ornish, weight watchers, and zone diets for weight loss and heart disease risk reduction: a randomized trial. JAMA. 2005;293(1):43–53.

    Article  CAS  PubMed  Google Scholar 

  92. Crichton GE, Howe PRC, Buckley JD, Coates AM, Murphy KJ, Bryan J. Long-term dietary intervention trials: critical issues and challenges. Trials. 2012;13(1):111.

    Article  PubMed  PubMed Central  Google Scholar 

  93. Hubbard GP, Elia M, Holdoway A, Stratton RJ. A systematic review of compliance to oral nutritional supplements. Clin Nutr. 2012;31(3):293–312.

    Article  PubMed  Google Scholar 

  94. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.

    Article  PubMed  Google Scholar 

  95. Naude CE, Durao S, Harper A, Volmink J. Scope and quality of Cochrane reviews of nutrition interventions: a cross-sectional study. Nutr J. 2017;16(1):22.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

Open Access funding enabled and organized by Projekt DEAL. Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—project number 459430615. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Institute for Evidence in Medicine, Medical Centre - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany

    Julia Stadelmaier, Isabelle Roux & Lukas Schwingshackl

  2. Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Centre, University of Freiburg, Freiburg, Germany

    Maria Petropoulou

Authors
  1. Julia Stadelmaier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Isabelle Roux
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Petropoulou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lukas Schwingshackl
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LS conceived and designed this study. JS, IR, and LS conducted the study selection and data extraction. JS, MP, and LS conducted the statistical analyses and participated in the interpretation of the data. JS and LS drafted the manuscript. All authors critically reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Julia Stadelmaier.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Appendix 1. Inclusion and exclusion criteria. Appendix 2. Search strategy.

Additional file 2:

Fig. S1. Example of calculating effect estimates. Fig. S2. Forest plot, random sequence (binary outcomes), main analysis. Fig. S3. Forest plot, random sequence (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S4. Forest plot, random sequence (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S5. Forest plot, random sequence (binary outcomes), subgroup analysis: clusters of interventions. Fig. S6. Forest plot, random sequence (binary outcomes), subgroup analysis: clusters of outcomes. Fig. S7. Forest plot, random sequence (binary outcomes), subgroup analysis: subjective versus objective outcomes. Fig. S8. Forest plot, random sequence (continuous outcomes), main analysis. Fig. S9. Forest plot, random sequence (continuous outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S10. Forest plot, random sequence (continuous outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S11. Forest plot, allocation concealment (binary outcomes), main analysis. Fig. S12. Forest plot, allocation concealment (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S13. Forest plot, allocation concealment (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S14. Forest plot, allocation concealment (binary outcomes), subgroup analysis: clusters of interventions. Fig. S15. Forest plot, allocation concealment (binary outcomes), subgroup analysis: Clusters of outcomes. Fig. S16. Forest plot, allocation concealment (binary outcomes), subgroup analysis: subjective versus objective outcomes. Fig. S17. Forest plot, allocation concealment (continuous outcomes), main analysis. Fig. S18. Forest plot, allocation concealment (continuous outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S19. Forest plot, allocation concealment (continuous outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S20. Forest plot, blinding of participants/personnel (binary outcomes), main analysis. Fig. S21. Forest plot, blinding of participants/personnel (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S22. Forest plot, blinding of participants/personnel (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig S23. Forest plot, blinding of participants/personnel (binary outcomes), subgroup analysis: clusters of interventions. Fig. S24. Forest plot, blinding of participants/personnel (binary outcomes), subgroup analysis: subjective versus objective outcomes. Fig. S25. Forest plot, blinding of participants/personnel (binary outcomes), subgroup analysis: clusters of outcomes. Fig. S26. Forest plot, blinding of participants/personnel (continuous outcomes), main analysis. Fig. S27. Forest plot, blinding of participants/personnel (continuous outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S28. Forest plot, blinding of participants/personnel (continuous outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S29. Forest plot, blinding of outcome assessment (binary outcomes), main analysis. Fig. S30. Forest plot, blinding of outcome assessment (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S31. Forest plot, blinding of outcome assessment (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S32. Forest plot, blinding of outcome assessment (binary outcomes), subgroup analysis: clusters of interventions. Fig. S33. Forest plot, blinding of outcome assessment (binary outcomes), subgroup analysis: clusters of outcomes. Fig. S34. Forest plot, blinding of outcome assessment (binary outcomes), subgroup analysis: subjective versus objective outcomes. Fig. S35. Forest plot, blinding of outcome assessment (continuous outcomes), main analysis. Fig. S36. Forest plot, blinding of outcome assessment (continuous outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S37. Forest plot, blinding of outcome assessment (continuous outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S38. Forest plot, incomplete outcome data (binary outcomes), main analysis. Fig. S39. Forest plot, incomplete outcome data (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S40. Forest plot, incomplete outcome data (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S41. Forest plot, incomplete outcome data (binary outcomes), subgroup analysis: clusters of interventions. Fig. S42. Forest plot, incomplete outcome data (binary outcomes), subgroup analysis: clusters of outcomes. Fig. S43. Forest plot, incomplete outcome data (binary outcomes), subgroup analysis: subjective versus objective outcomes. Fig. S44. Forest plot, incomplete outcome data (continuous outcomes), main analysis. Fig. S45. Forest plot, incomplete outcome data (continuous outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S46. Forest plot, incomplete outcome data (continuous outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S47. Forest plot, selective reporting (binary outcomes), main analysis. Fig. S48. Forest plot, selective reporting (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S49. Forest plot, selective reporting (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S50. Forest plot, selective reporting (binary outcomes), subgroup analysis: clusters of interventions. Fig. S51. Forest plot, selective reporting (binary outcomes), subgroup analysis: clusters of outcomes. Fig. S52. Forest plot, selective reporting (binary outcomes), subgroup analysis: subjective versus objective outcomes. Fig. S53. Forest plot, selective reporting (continuous outcomes), main analysis. Fig. S54. Forest plot, selective reporting (continuous outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S55. Forest plot, selective reporting (continuous outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S56. Forest plot, dietary compliance (binary outcomes), main analysis. Fig. S57. Forest plot, dietary compliance (binary outcomes), sensitivity analysis: excluding highly correlated outcomes. Fig. S58. Forest plot, dietary compliance (binary outcomes), sensitivity analysis: including only one outcome per comparison. Fig. S59. Forest plot, dietary compliance (binary outcomes), subgroup analysis: clusters of interventions. Fig. S60. Forest plot, dietary compliance (binary outcomes), subgroup analysis: clusters of outcomes. Fig. S61. Forest plot, dietary compliance (binary outcomes), subgroup analysis: subjective versus objective outcomes.

Additional file 3:

Table S1. Exclusion reasons for highly correlated outcomes (Random sequence). Table S2. Exclusion reasons for highly correlated outcomes (Allocation concealment). Table S3. Exclusion reasons for highly correlated outcomes (Blinding of participants/personnel). Table S4. Exclusion reasons for highly correlated outcomes (Blinding of outcome assessment). Table S5. Exclusion reasons for highly correlated outcomes (Incomplete outcome data). Table S6. Exclusion reasons for highly correlated outcomes (Selective Reporting). Table S7. Exclusion reasons for highly correlated outcomes (Dietary Compliance). Table S8. References excluded in the full-text screening process. Table S9. Meta-analyses excluded from meta-epidemiologic study. Table S10. Characteristics of the included meta-analyses with binary outcomes. Table S11. Characteristics of the included meta-analyses with continuous outcomes. Table S12. Meta-Regression for sample size and conflicts of interest (Random sequence, binary outcomes). Table S13. Meta-Regression for sample size and conflicts of interest (Allocation concealment, binary outcomes). Table S14. Meta-Regression for sample size and conflicts of interest (Blinding of participants/personnel, binary outcomes). Table S15. Meta-Regression for sample size and conflicts of interest (Blinding of outcome assessment, binary outcomes). Table S16. Meta-Regression for sample size and conflicts of interest (Incomplete outcome data, binary outcomes). Table S17. Meta-Regression for sample size and conflicts of interest (Selective reporting, binary outcomes). Table S18. Meta-Regression for sample size and conflicts of interest (Dietary compliance, binary outcomes). Table S19. Meta-Regression for sample size and conflicts of interest (Random sequence, continuous outcomes). Table S20. Meta-Regression for sample size and conflicts of interest (Allocation concealment, continuous outcomes). Table S21. Meta-Regression for sample size and conflicts of interest (Blinding of participants/personnel, continuous outcomes). Table S22. Meta-Regression for sample size and conflicts of interest (Blinding of outcome assessment, continuous outcomes). Table S23. Meta-Regression for sample size and conflicts of interest (Incomplete outcome data, binary outcomes). Table S24. Meta-Regression for sample size and conflicts of interest (Selective reporting, continuous outcomes). Table S25. Meta-Regression for sample size and conflicts of interest (Dietary compliance, continuous outcomes). Table S26. Definition of dietary compliance.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stadelmaier, J., Roux, I., Petropoulou, M. et al. Empirical evidence of study design biases in nutrition randomised controlled trials: a meta-epidemiological study. BMC Med 20, 330 (2022). https://doi.org/10.1186/s12916-022-02540-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12916-022-02540-9

Keywords

BMC Medicine

ISSN: 1741-7015

Contact us