Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research

BMC Medicine volume 20, Article number: 355 (2022) Cite this article

Abstract

Background

Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same systematic review.

Methods

BoE derived from systematic reviews of RCTs and cohort studies published in the 13 medical journals with the highest impact factor were considered. We re-analyzed effect estimates of the included systematic reviews by pooling BoE from RCTs with BoE from cohort studies using random and common effects models. We evaluated statistical heterogeneity, 95% prediction intervals, weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from cohort studies modified the conclusion from BoE of RCTs.

Results

Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were pooled. By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% confidence interval (CI) excludes no effect. By pooling BoE from RCTs and cohort studies, the median I2 was 48%, and the median contributed percentage weight of RCTs to the pooled estimates was 40%. The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant (79.7%). The integration of BoE from cohort studies modified the conclusion (by examining the 95% CI) from BoE of RCTs in 32 (27%) of the 118 BoE-pairs, but the direction of effect was mainly concordant (88%).

Conclusions

Our findings provide insights for the potential impact of pooling both BoE in systematic reviews. In medical research, it is often important to rely on both evidence of RCTs and cohort studies to get a whole picture of an investigated intervention-disease association. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling.

Peer Review reports

Background

Randomized controlled trials (RCTs) and cohort studies are the most common study designs used to assess treatment effects of medical interventions [1, 2]. RCTs, if well-designed and well-conducted, are considered as the gold standard and are widely accepted as the ideal methodology for causal inference [1,2,3].

However, RCTs may not be available for certain medical treatments due to ethical reasons or may suffer from inherent methodological limitations such as low external validity [4]. On the other hand, cohort studies may often have higher external validity, but also a higher risk of confounding. It is generally considered that systematic reviews should be based on RCTs because these studies are more likely to provide unbiased information than other study designs [5].

According to recent GRADE guidance, cohort studies can be highly valuable and provide complementary, sequential, or replacement evidence for RCTs in a systematic review or other evidence syntheses [6]. However, the potential impact of integrating evidence from cohort studies in meta-analyses of RCTs in the medical field has not been investigated yet.

To close this important research gap, this empirical study aims to conduct a pooling scenario of bodies of evidence (BoE) from RCTs with matched BoE from cohort studies. We investigate the extent of how the integration of BoE from cohort studies modifies the conclusion of BoE of RCTs, the direction of effect estimates derived from BoE of RCTs, and its impact on statistical heterogeneity. Moreover, we will evaluate the contributed aggregated weights of RCTs to the pooled estimates, use random effects and common effects models for pooling, calculate 95% prediction intervals (PIs), and test for subgroup differences between BoE from RCTs and cohort studies.

Methods

The sample of this empirical study was based on a large meta-epidemiological study [7], which was planned, written, and reported in adherence to current guidance for meta-epidemiological methodology research [8]. Eligibility criteria (PI/ECO: patient/population, intervention/exposure, comparator, and outcome) are reported in Table 1. Briefly, we included systematic reviews on medical interventions (or exposures) that included both RCTs and cohort studies for the same patient-relevant outcome and that performed meta-analyses for at least one BoE [7].

Table 1 Detailed description of inclusion and exclusion criteria
Full size table

Identification of systematic reviews of RCTs and cohort studies

The original search for the meta-epidemiological study was conducted in MEDLINE on 04.05.2020 for the period between 01.01.2010 and 31.12.2019 in the 13 medical journals with the highest impact factor (according to the Journal Citation Report [JCR] 2018 category general and internal medicine). Initially, we planned to include the ten highest impact factor journals, but three journals did not publish any systematic review with an eligible BoE-pair. We therefore included the subsequent three journals according to the JCR 2018. The search strategy including the list of considered journals is given in Additional file 1: Appendix S1. Title and abstract screening was conducted by one reviewer (NB), and potentially relevant full texts were screened for eligibility by two reviewers independently (NB, LS). Any discrepancies were resolved by discussion.

For each included BoE from a systematic review, we included a maximum of three patient-relevant outcomes (e.g., mortality) and a maximum of three intermediate disease markers (e.g., blood lipids). If more than three outcomes were available for a given systematic review, we included the primary outcomes and thereafter we used a top-down approach (mentioned first). We evaluated the similarity of the PI/ECO criteria between BoE-pair from RCTs and cohort studies within each systematic review. For each BoE-pair, the similarity of each PI/ECO domain was rated as “more or less identical,” “similar but not identical,” or “broadly similar” (Additional file 1: Table S1). A detailed description of identification and evaluating similarity BoE-pairs can be found elsewhere [7].

Data extraction

Two reviewers (NB, LH) extracted the following data for each included BoE-pair into a piloted data extraction sheet: name of the first author, year of publication, type of intervention/exposure (e.g., antiretroviral therapy), description of the comparator, effect estimates (risk ratio [RR], hazard ratio [HR], odds ratio [OR], mean difference [MD], including 95% confidence interval [CI]), and number of studies. A detailed description of data extraction can be found elsewhere [7]. For the current analysis, we additionally extracted all effect estimates and corresponding 95% CI of the primary studies included for a relevant BoE (NB, LH).

Statistical analysis

For our pooling scenario, we re-analyzed the effect estimates of all eligible systematic reviews in a two-step approach: For each identified BoE-pair, we first pooled the effect estimates obtained from RCTs and cohort studies separately using a random effects model. Primary studies based on inappropriate study designs (i.e., case-control, cross-sectional, and quasi-RCTs) were excluded.

Second, we pooled the BoE from RCTs with the BoE from cohort studies with a random effects model for each BoE-pair. Binary outcomes (pooled as RRs, HRs, or ORs) and continuous outcomes (pooled as MDs on the same scale) were considered for analysis. Random effects models were used to account for potential between-study heterogeneity. For the sensitivity analysis, we used a common effects model to evaluate whether this hypothetical scenario is more conservative for pooling BoE from RCTs and cohort studies.

To explore the impact of including cohort studies on pooled effect estimates by combining BoE from RCTs and cohort studies (with or without subgroups), we compared the results and conclusions (95% CI including vs. excluding the null effect) between the BoE of RCTs only and that including both RCTs and cohort studies. Then, we evaluated the contributed weight of RCTs to the pooled estimates and conducted a statistical test for subgroup differences between the two types of BoE. A p-value < 0.05 was considered as statistical significant.

In an additional analysis, we used effect estimates of cohort studies as a reference and compared the results and conclusion between the BoE of cohort studies only and that including both, RCTs and cohort studies.

Heterogeneity in meta-analyses was tested with a standard χ2 test. We quantified any inconsistency by using the I2 parameter: I2=((Q−df))/Q × 100%, where Q is the χ2 statistic and df is its degrees of freedom [9]. An I2-value of greater than 50% was considered to represent considerable heterogeneity [10]. For binary outcomes, we additionally calculated τ2, which is independent of study size and describes variability between studies in relation to the risk estimates [11]. For continuous outcomes, we did not calculate τ2 due to the use of different scales between meta-analyses (blood pressure [mmHg] or body weight [kg]. Meta-analyses were conducted using Review Manager (RevMan) version 5.3 [12].

Whereas in a random effects meta-analysis, the focus is usually on the average treatment effect and its 95% CI, the calculation of a prediction interval (95% PI) also considers the potential treatment effect within an individual study setting, as this may differ from the average effect [11]. 95% PIs were calculated for the summary random effects for each meta-analysis since they further account for the degree of between-study heterogeneity and give a range for which we are 95% confident that the effect in a new study examining the same association lies within [11]. Calculations of 95% PI were conducted with Stata 15.

Results

Overall, 64 systematic reviews of RCTs and cohort studies were included [13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]. Of the identified 129 outcome pairs, 118 from 59 systematic reviews were included in the present pooling scenario and re-analyzed (Additional file 1: Table S2-S3) [7, 13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76] (109 dichotomous and nine continuous outcomes) (Additional file 1: Figs. S1-118). Eleven outcome pairs from five systematic reviews [13, 26, 73,74,75] were excluded from the current analysis. Reasons for exclusion are provided in Additional file 1: Table S4.

Our sample of 118 BoE-pairs was based on 653 RCTs and 804 cohort studies. Detailed study characteristics including a description of the population, intervention/comparator, outcomes, range of study length, and risk of bias/study quality of primary studies included for each outcome pair have been described elsewhere [7].

Two of the outcome pairs were classified (PI/ECO similarity degree) as “more or less identical” and 82 as “similar but not identical,” whereas 34 were classified as “broadly similar.” Out of the 118 BoE from RCTs, for 39 (33.1%), the 95% CI excludes no effect, whereas for the BoE for cohort studies, 58 (49.2%) indicated a 95% CI excluding no effect. Twenty-four (20.3%) out of 118 BoE-pairs showed simultaneously for BoE from RCTs and BoE from cohort studies a 95% CI excluding no effect and a concordant direction of effect. The median I2 was 5% (τ2=0) across BoE from RCTs and 41% (τ2=0.03) across BoE from cohort studies, whereas the mean I2 was 23% (τ2=0.14) and 42% (τ2=0.18), respectively. Table 2 shows the summary effects of the BoE from RCTs, cohort studies, and the pooling scenario.

Table 2 Pooling results of bodies of evidence from RCTs with cohort studies based on random effects and common effects models, 95% prediction intervals, heterogeneity, test for subgroup difference, and population (P), intervention (I)/exposure (E), comparator (C), and outcome similarity degree
Full size table

Pooling scenarios

By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% CI excludes no effect. For the common effects model, for 77 (62.3%) out of 118 BoE-pairs, the 95% CI excludes no effect. Approximately half of the binary effect sizes were in the range of 0.75 to 1.25, and 64.2% reported an effect estimate <1. The test for subgroup difference comparing BoE from RCTs and BoE of cohort studies was statistically significant (p<0.05) for 25 BoE-pairs (21.2%). By pooling BoE from RCTs and cohort studies, the median I2 was 51% (τ2=0.05), whereas the mean I2 was 45% (τ2=0.11). The contributed weight of RCTs to the pooled estimates was 40% (median) and 42% (mean). As for the 95% PIs, 21.2% (n=25) of the pooled BoE from RCTs and cohort studies excluded no effect.

The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant in 94 of 118 BoE-pairs (79.7%). The difference between effect estimates was >0.25 for 4.2% (n=5) and >0.50 for 3.4% (n=4) for the dichotomous effect measures. The integration of BoE from cohort studies modified the conclusion from BoE of RCTs in 32 (27.1%) of the 118 BoE-pairs (i.e., 95% CI excludes no effect changed to 95% CI overlaps no effect or vice versa); in 28 (87.5%) of these 32 BoE, the direction of effect was concordant. In nine (28.1%) of these 32 BoE-pairs, the test of subgroup difference was statistically significant (p<0.05) comparing BoE from RCTs and BoE from cohort studies (in three of these nine associations, the direction of effect was opposite). In 12 (37.5 %) of these 32 BoE-pairs, the overall degree of PI/ECO similarity was judged as “broadly similar.” Populations (n=7, 21.9%), interventions (n=5, 15.6%), and comparators (n=4, 12.5%) rated as “broadly similar” accounted for PI/ECO dissimilarities overall. In 20 (62.5%) of the 32 BoE-pairs, the degree of PI/ECO similarity was judged as “similar but not identical.” Populations (n=18; 56.3%), interventions (n=11; 34.4%), comparators (n=7; 21.9%), and outcomes (n=1; 3.1%) rated as “similar but not identical” accounted for PI/ECO dissimilarities.

In the additional analysis with cohort studies as reference (Additional file 1: Table S5), the direction of effect between BoE from cohort studies and pooled estimates was concordant in 106 (89.8%) of the 118 BoE-pairs. The integration of BoE from RCTs modified the conclusion from BoE of cohort studies in 24 (20.3%) of the 118 BoE-pairs.

Discussion

Summary of findings

This meta-epidemiological study is the first empirical study in medical research that evaluates the impact scenario of pooling bodies of evidence from RCTs and cohort studies. Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were included. By pooling BoE from RCTs and cohort studies in about 50% of the BoE-pairs, the 95% CI excludes no effect, whereas in about one-third of the included BoE from RCTs, the 95% CI excludes no effect. For 21% of pooled estimates, the test for subgroup difference comparing BoE from RCTs and BoE of cohort studies was statistically significant. The median weights of BoE from RCTs to the pooled estimates were 40%, suggesting that on average the contribution weight was not dissimilar between both BoE. Overall, the degree of statistical heterogeneity was moderate (I2=51%, τ2=0.05) and higher across meta-analyses of cohort studies (I2=41%, τ2=0.03) compared to meta-analyses of RCTs (I2=5%, τ2=0.00). The integration of BoE from cohort studies modified the conclusion derived from BoE of RCTs in nearly 30% of the BoE-pairs. The direction of effect between BoE of RCTs and pooled estimates, however, was mainly concordant. This suggests that by adding evidence from cohort studies statistical precision increased substantially.

Comparison with other studies

We did not identify any similar empirical study using a pooling scenario of different study designs in the field of medical research. However, a recent methodological study investigated a similar pooling scenario in nutrition research [77]. This large pooling scenario study showed that the integration of BoE from cohort studies modified the conclusion from BoE of RCTs in nearly 50% of included diet-disease associations, although the direction of effect was mainly concordant between BoE of RCTs and pooled estimates. The median weight of RCTs to the pooled estimates was 34%, and the statistical heterogeneity was substantially higher across meta-analyses of cohort studies (I2=55%, τ2=0.01) compared to RCTs (I2=0%, τ2=0). This finding is in line with our study. However, in our study, the integration of BoE from cohort studies modified the conclusion from BoE of RCTs less often (27% vs. 44%) [77]. Two main reasons may explain this difference. First, it has been suggested that effect estimates between RCTs and cohort studies differ quite often in nutrition research [78]. A recent meta-epidemiological study, however, has shown that on average the effect-difference between both study designs was even smaller than expected [79]. Second, the median weight of RCTs to the pooled estimated was larger in our study (40% vs. 34%) [79].

A recent meta-research study investigated how RCTs and observational studies were combined in meta-analyses [80]. In nearly 40% of meta-analyses, both observational studies and RCTs were combined in a single meta-analysis, without considering the two designs as subgroups. When comparing the results of those meta-analyses with meta-analyses restricted to RCTs only, the conclusion was modified by the integration of observational studies for nearly 71%, whereas in our study this was the case for 27%. In line with our findings, the authors found that including observational studies frequently increased statistical heterogeneity.

Implications for the broader research field

In a survey investigating the rationale, perceptions, and preferences for the integration of RCTs and observational studies in evidence syntheses by Cuello-Garcia and colleagues [81], it was shown that conducting separate meta-analyses for both study designs was the most frequent approach used. However, nearly half of the experts interviewed reported that they have already, on at least one occasion, pooled RCTs and observational studies in a meta-analysis [81].

According to the recent GRADE guidance on optimizing the integration of RCT and observational studies in evidence syntheses, observational studies can provide valuable information as complementary, sequential, or replacement evidence for RCTs [6]. In our empirical scenario, evidence from cohort studies was always considered as complementary evidence for RCTs. The GRADE guidance suggests, when RCTs provide already high certainty of evidence, looking for observational evidence will be unnecessary because the high certainty will not be improved [6]. However, in our sample of 118 BoE-pairs, only six BoE of RCTs were rated as high certainty, 18 as moderate, 11 as low, and two as very low. Thus, evidence from cohort studies seems valuable in the field of medical research [7].

In line with our findings, the Cochrane Handbook indicated that authors should expect greater statistical heterogeneity in a systematic review of observational studies compared to a systematic review of RCTs. Reasons include diverse ways in which observational studies may be designed to investigate the effects of interventions/exposures, and partly due to the increased potential for methodological variation between primary studies and the resulting variation in their risk of bias. Therefore, the Cochrane Handbook recommends that RCTs and observational studies should not be combined in a meta-analysis (although the power to detect an effect may increase [82]). In contrast to the recommendations of Cochrane, a recent framework for the synthesis of observational studies and RCTs does not reject the pooling of both study designs in principle. It presents recommendations on when and how to combine evidence from different study designs, but also highlights challenges in this process [83]. Moreover, a recent scoping review summarized the methods to systematically review and meta-analyze observational studies and highlighted that existing guidance is highly conflicting for pooling if results are similar over different study designs [84]. Finally, in several high-impact factor journal meta-analyses, both study designs were pooled [21, 32, 36, 56].

Overall, it looks like further methodological research is needed to shed light into this gray area. On the one hand, further research should address the application of existing guidance in terms of utility, acceptability, and reproducibility and elaborate ways to deal with occurring challenges [83]. On the other hand, factors such as risk of bias/study quality that may contribute to the differences in effect estimates between BoE of RCTs and cohort studies and conflicting results in pooling scenarios should be further explored. Our previously conducted study analyzed disagreement of effect estimates with regard to differences by each PI/ECO domain [7]. In the meta-regression, we showed that differences of interventions were the main drivers towards disagreement. The average effect on the other pooled effect estimators, however, was not statistically significant [7].

We assume that methodological trial characteristics are other possible drivers towards disagreement, since observational studies are prone to risk of bias by confounding [5], and appropriate adjustment for confounding is thus crucial to integrate both RCTs and cohort studies (or other non-randomized studies) in a pooling scenario. In the sample provided in this study, the tools used to assess the quality/risk of bias of primary studies included across the BoE were heterogeneous, which makes the comparison of results challenging. Future studies should focus on the impact of quality characteristics on pooling scenarios by using similar appraisal tools to increase comparability between RCTs and cohort studies (e.g., ROBINS-I [85] and the Cochrane Risk of Bias Tool [86]). Moreover, attention in future studies should also be paid to the integration of other non-randomized study designs a part from cohort studies. However, overall, we assume generalizability of our findings since concordance may not be linked to study design per se, but rather on the quality/risk of bias of the studies included [1].

This paper did not aim to provide insights on how pooling results from different study designs impacts the certainty rating of results and whether it reduces or increases the amount of low or very low certainty of evidence ratings. In a recently published hypothetical scenario analysis, we could show that pooling BoE from RCTs and cohort studies for nutrition-related research questions would reduce the amount of very low and low certainty of evidence ratings [87]. We recommend that future research should examine also the impact scenario of pooling BoE of RCTs and cohort studies for medical research questions on the overall GRADE rating and on individual GRADE domains in order to inform future guidance development.

Strengths and limitations

This study has several strengths. First, we analyzed a large sample of BoE-pairs (n=118), which was based on 653 RCTs and 804 cohort studies. Second, we selected BoE-pairs from systematic reviews published in high-impact medical journals, which have shown to be of higher methodological quality [88]. Third, our study was based on a broad methodological repertoire, i.e., by including meta-analyses of binary outcomes, and also continuous outcomes, investigating different statistical measures of heterogeneity, conducting random and common effects models, and calculating 95% PI.

Limitations of this study are as follows. First, although we pooled a large sample of BoE-pairs, our sample may not be representative of all meta-analyses, and the totality of evidence of available associations might provide different results. Second, we did not consider and weighted risk of bias of primary studies in our pooling scenario. Third, only two BoE-pairs were judged as “more or less identical,” indicating that BoE of RCTs and cohort studies differ at least slightly in terms of PI/ECO criteria and caution is therefore required when pooling both BoE. Fourth, the potential for confounding in the individual cohort studies and subgroup analyses in the meta-analysis cannot be ruled out. Moreover, several subgroups also included only a small number of studies. Fifth, the methodological quality of the systematic reviews included in this study was not assessed. Although we assume that systematic reviews published in high-impact factor journals adhere to high methodological standards, this is nevertheless an important limitation. Due to these limitations, our findings need to be interpreted with caution.

Conclusions

This large pooling scenario study showed that the integration of BoE from cohort studies modified the conclusion from BoE of RCTs in 27% of included BoE, although the direction of effect was mainly concordant between BoE of RCTs and pooled estimates. The median weight of RCTs to the pooled estimates was 40%, and the statistical heterogeneity was substantially driven by integrating BoE of cohort studies. Our findings provide a first insight regarding the potential impact of pooling of both BoE in evidence syntheses. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling.

Availability of data and materials

Data are based on published meta-analyses.

Abbreviations

BoE:

Bodies of evidence

CI:

Confidence interval

GRADE:

Grading of Recommendations, Assessment, Development and Evaluation

PI:

Prediction interval

PI/ECO:

Patient/population, intervention/exposure, comparator, outcome

RCT:

Randomized controlled trial

τ 2 :

Heterogeneity value with the restricted maximum-likelihood estimation method

References

  1. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med. 2016;21(4):125.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Oxford Centre. EBM Levels of Evidence Working Group. In: Jeremy Howick ICJLL, Glasziou P, Greenhalgh T, Heneghan C, Liberati A, Moschetti I, Phillips B, Thornton H, Goddard O, Hodgkinson M, editors. The Oxford 2011 levels of evidence. Oxford Centre for Evidence-Based Medicine; 2011.

    Google Scholar 

  3. Kabisch M, Ruckes C, Seibert-Grafe M, Blettner M. Randomized controlled trials: part 17 of a series on evaluation of scientific publications. Dtsch Arztebl Int. 2011;108(39):663–8.

    PubMed  PubMed Central  Google Scholar 

  4. Kostis JB, Dobrzynski JM. Limitations of randomized clinical trials. Am J Cardiol. 2020;129:109–15.

    Article  PubMed  Google Scholar 

  5. JPT H, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021). Cochrane. 2021; Available from www.training.cochrane.org/handbook.

  6. Cuello-Garcia CA, Santesso N, Morgan RL, Verbeek J, Thayer K, Ansari MT, et al. GRADE guidance 24 optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelines. J Clin Epidemiol. 2022;142:200–8.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Bröckelmann N, Balduzzi S, Harms L, Beyerbach J, Petropoulou M, Kubiak C, et al. Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study. BMC Med. 2022;20(1):174.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. Bmj. 2003;327(7414):557–60.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

    Article  PubMed  Google Scholar 

  11. Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549.

    Article  PubMed  Google Scholar 

  12. Review Manager (RevMan). Version 5.3. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration; 2014.

    Google Scholar 

  13. Abou-Setta AM, Beaupre LA, Rashiq S, Dryden DM, Hamm MP, Sadowski CA, et al. Comparative effectiveness of pain management interventions for hip fracture: a systematic review. Ann Intern Med. 2011;155(4):234–45.

    Article  PubMed  Google Scholar 

  14. Aburto NJ, Ziolkovska A, Hooper L, Elliott P, Cappuccio FP, Meerpohl JJ. Effect of lower sodium intake on health: systematic review and meta-analyses. BMJ. 2013;346:f1326.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Ahmad Y, Sen S, Shun-Shin MJ, Ouyang J, Finegold JA, Al-Lamee RK, et al. Intra-aortic balloon pump therapy for acute myocardial infarction: a meta-analysis. JAMA Intern Med. 2015;175(6):931–9.

    Article  PubMed  Google Scholar 

  16. Alipanah N, Jarlsberg L, Miller C, Linh NN, Falzon D, Jaramillo E, et al. Adherence interventions and outcomes of tuberculosis treatment: a systematic review and meta-analysis of trials and observational studies. PLoS Med. 2018;15(7):e1002595.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Anglemyer A, Rutherford GW, Horvath T, Baggaley RC, Egger M, Siegfried N. Antiretroviral therapy for prevention of HIV transmission in HIV-discordant couples. Cochrane Database Syst Rev. 2013;4:Cd009153.

    Google Scholar 

  18. Azad MB, Abou-Setta AM, Chauhan BF, Rabbani R, Lys J, Copstein L, et al. Nonnutritive sweeteners and cardiometabolic health: a systematic review and meta-analysis of randomized controlled trials and prospective cohort studies. Can Med Assoc J. 2017;189(28):E929.

    Article  Google Scholar 

  19. Barnard S, Kim C, Park MH, Ngo TD. Doctors or mid-level providers for abortion. Cochrane Database Syst Rev. 2015;2015(7):Cd011242.

    PubMed Central  Google Scholar 

  20. Bellemain-Appaix A, O'Connor SA, Silvain J, Cucherat M, Beygui F, Barthelemy O, et al. Association of clopidogrel pretreatment with mortality, cardiovascular events, and major bleeding among patients undergoing percutaneous coronary intervention: a systematic review and meta-analysis. JAMA. 2012;308(23):2507–16.

    Article  CAS  PubMed  Google Scholar 

  21. Bellemain-Appaix A, Kerneis M, O'Connor SA, Silvain J, Cucherat M, Beygui F, et al. Reappraisal of thienopyridine pretreatment in patients with non-ST elevation acute coronary syndrome: a systematic review and meta-analysis. BMJ. 2014;349:g6269.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  22. Bloomfield HE, Koeller E, Greer N, MacDonald R, Kane R, Wilt TJ. Effects on health outcomes of a Mediterranean diet with no restriction on fat intake: a systematic review and meta-analysis. Ann Intern Med. 2016;165(7):491–500.

    Article  PubMed  Google Scholar 

  23. Bolland MJ, Leung W, Tai V, Bastin S, Gamble GD, Grey A, et al. Calcium intake and risk of fracture: systematic review. BMJ. 2015;351:h4580.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  24. Brenner H, Stock C, Hoffmeister M. Effect of screening sigmoidoscopy and screening colonoscopy on colorectal cancer incidence and mortality: systematic review and meta-analysis of randomised controlled trials and observational studies. BMJ. 2014;348:g2467.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Chowdhury R, Stevens S, Gorman D, Pan A, Warnakula S, Chowdhury S, et al. Association between fish consumption, long chain omega 3 fatty acids, and risk of cerebrovascular disease: systematic review and meta-analysis. BMJ. 2012;345:e6698.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  26. Chowdhury R, Kunutsor S, Vitezova A, Oliver-Williams C, Chowdhury S, Kiefte-de-Jong JC, et al. Vitamin D and risk of cause specific death: systematic review and meta-analysis of observational cohort and randomised intervention studies. BMJ. 2014;348:g1903.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Chung M, Tang AM, Fu Z, Wang DD, Newberry SJ. Calcium intake and cardiovascular disease risk: an updated systematic review and meta-analysis. Ann Intern Med. 2016;165(12):856–66.

    Article  PubMed  Google Scholar 

  28. Ding M, Huang T, Bergholdt HK, Nordestgaard BG, Ellervik C, Qi L. Dairy consumption, systolic blood pressure, and risk of hypertension: Mendelian randomization study. BMJ. 2017;356:j1000.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Fenton JJ, Weyrich MS, Durbin S, Liu Y, Bang H, Melnikow J. Prostate-specific antigen-based screening for prostate cancer: evidence report and systematic review for the US Preventive Services Task Force. JAMA. 2018;319(18):1914–31.

    Article  PubMed  Google Scholar 

  30. Filippini G, Del Giovane C, Clerico M, Beiki O, Mattoscio M, Piazza F, et al. Treatment with disease-modifying drugs for people with a first clinical attack suggestive of multiple sclerosis. Cochrane Database Syst Rev. 2017;4:Cd012200.

    PubMed  Google Scholar 

  31. Fluri F, Engelter S, Lyrer P. Extracranial-intracranial arterial bypass surgery for occlusive carotid artery disease. Cochrane Database Syst Rev. 2010;2010(2):Cd005953.

    PubMed Central  Google Scholar 

  32. Gargiulo G, Sannino A, Capodanno D, Barbanti M, Buccheri S, Perrino C, et al. Transcatheter aortic valve implantation versus surgical aortic valve replacement: a systematic review and meta-analysis. Ann Intern Med. 2016;165(5):334–44.

    Article  PubMed  Google Scholar 

  33. Hartling L, Dryden DM, Guthrie A, Muise M, Vandermeer B, Donovan L. Benefits and harms of treating gestational diabetes mellitus: a systematic review and meta-analysis for the U.S. Preventive Services Task Force and the National Institutes of Health Office of Medical Applications of Research. Ann Intern Med. 2013;159(2):123–9.

    Article  PubMed  Google Scholar 

  34. Henderson JT, Webber EM, Bean SI. Screening for asymptomatic bacteriuria in adults: updated evidence report and systematic review for the US Preventive Services Task Force. JAMA. 2019;322(12):1195–205.

    Article  PubMed  Google Scholar 

  35. Higgins JP, Soares-Weiser K, Lopez-Lopez JA, Kakourou A, Chaplin K, Christensen H, et al. Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. Bmj. 2016;355:i5170.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Hopley C, Stengel D, Ekkernkamp A, Wich M. Primary total hip arthroplasty versus hemiarthroplasty for displaced intracapsular hip fractures in older patients: systematic review. Bmj. 2010;340:c2332.

    Article  PubMed  Google Scholar 

  37. Hüpfl M, Selig HF, Nagele P. Chest-compression-only versus standard cardiopulmonary resuscitation: a meta-analysis. Lancet. 2010;376(9752):1552–7.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Jamal SA, Vandermeer B, Raggi P, Mendelssohn DC, Chatterley T, Dorgan M, et al. Effect of calcium-based versus non-calcium-based phosphate binders on mortality in patients with chronic kidney disease: an updated systematic review and meta-analysis. Lancet. 2013;382(9900):1268–77.

    Article  CAS  PubMed  Google Scholar 

  39. Jefferson T, Di Pietrantonj C, Al-Ansary LA, Ferroni E, Thorning S, Thomas RE. Vaccines for preventing influenza in the elderly. Cochrane Database Syst Rev. 2010;17(2):Cd004876.

    Google Scholar 

  40. Jefferson T, Rivetti A, Di Pietrantonj C, Demicheli V, Ferroni E. Vaccines for preventing influenza in healthy children. Cochrane Database Syst Rev. 2012;2012(8):Cd004879.

    PubMed Central  Google Scholar 

  41. Jin H, Leng Q, Li C. Dietary flavonoid for preventing colorectal neoplasms. Cochrane Database Syst Rev. 2012;15(8):Cd009350.

    Google Scholar 

  42. Kansagara D, Dyer E, Englander H, Fu R, Freeman M, Kagen D. Treatment of anemia in patients with heart disease: a systematic review. Ann Intern Med. 2013;159(11):746–57.

    Article  PubMed  Google Scholar 

  43. Keag OE, Norman JE, Stock SJ. Long-term risks and benefits associated with cesarean delivery for mother, baby, and subsequent pregnancies: systematic review and meta-analysis. PLoS Med. 2018;15(1):e1002494.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Kredo T, Adeniyi FB, Bateganya M, Pienaar ED. Task shifting from doctors to non-doctors for initiation and maintenance of antiretroviral therapy. Cochrane Database Syst Rev. 2014;1(7):Cd007331.

    Google Scholar 

  45. Li L, Shen J, Bala MM, Busse JW, Ebrahim S, Vandvik PO, et al. Incretin treatment and risk of pancreatitis in patients with type 2 diabetes mellitus: systematic review and meta-analysis of randomised and non-randomised studies. Bmj. 2014;348:g2366.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  46. Li L, Li S, Deng K, Liu J, Vandvik PO, Zhao P, et al. Dipeptidyl peptidase-4 inhibitors and risk of heart failure in type 2 diabetes: systematic review and meta-analysis of randomised and observational studies. Bmj. 2016;352:i610.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  47. Matthews A, Stanway S, Farmer RE, Strongman H, Thomas S, Lyon AR, et al. Long term adjuvant endocrine therapy and risk of cardiovascular disease in female breast cancer survivors: systematic review. Bmj. 2018;363:k3845.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Menne J, Dumann E, Haller H, Schmidt BMW. Acute kidney injury and adverse renal events in patients receiving SGLT2-inhibitors: a systematic review and meta-analysis. PLoS Med. 2019;16(12):e1002983.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Mesgarpour B, Heidinger BH, Roth D, Schmitz S, Walsh CD, Herkner H. Harms of off-label erythropoiesis-stimulating agents for critically ill people. Cochrane Database Syst Rev. 2017;8:Cd010969.

    PubMed  Google Scholar 

  50. Moberley S, Holden J, Tatham DP, Andrews RM. Vaccines for preventing pneumococcal infection in adults. Cochrane Database Syst Rev. 2013;23(1):Cd000422.

    Google Scholar 

  51. Molnar AO, Fergusson D, Tsampalieros AK, Bennett A, Fergusson N, Ramsay T, et al. Generic immunosuppression in solid organ transplantation: systematic review and meta-analysis. Bmj. 2015;350:h3163.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Navarese EP, Gurbel PA, Andreotti F, Tantry U, Jeong YH, Kozinski M, et al. Optimal timing of coronary invasive strategy in non-ST-segment elevation acute coronary syndromes: a systematic review and meta-analysis. Ann Intern Med. 2013;158(4):261–70.

    Article  PubMed  Google Scholar 

  53. Nelson RL, Furner SE, Westercamp M, Farquhar C. Cesarean delivery for the prevention of anal incontinence. Cochrane Database Syst Rev. 2010;2010(2):Cd006756.

    PubMed Central  Google Scholar 

  54. Nieuwenhuijse MJ, Nelissen RG, Schoones JW, Sedrakyan A. Appraisal of evidence base for introduction of new implants in hip and knee replacement: a systematic review of five widely used device technologies. Bmj. 2014;349:g5133.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  55. Nikooie R, Neufeld KJ, Oh ES, Wilson LM, Zhang A, Robinson KA, et al. Antipsychotics for treating delirium in hospitalized adults: a systematic review. Ann Intern Med. 2019;171:485–95.

    Article  PubMed  Google Scholar 

  56. Ochen Y, Beks RB, van Heijl M, Hietbrink F, Leenen LPH, van der Velde D, et al. Operative treatment versus nonoperative treatment of Achilles tendon ruptures: systematic review and meta-analysis. Bmj. 2019;364:k5120.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Pittas AG, Chung M, Trikalinos T, Mitri J, Brendel M, Patel K, et al. Systematic review: vitamin D and cardiometabolic outcomes. Ann Intern Med. 2010;152(5):307–14.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Raman G, Moorthy D, Hadar N, Dahabreh IJ, O'Donnell TF, Thaler DE, et al. Management strategies for asymptomatic carotid stenosis: a systematic review and meta-analysis. Ann Intern Med. 2013;158(9):676–85.

    Article  PubMed  Google Scholar 

  59. Schweizer M, Perencevich E, McDanel J, Carson J, Formanek M, Hafner J, et al. Effectiveness of a bundled intervention of decolonization and prophylaxis to decrease Gram positive surgical site infections after cardiac or orthopedic surgery: systematic review and meta-analysis. Bmj. 2013;346:f2743.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Silvain J, Beygui F, Barthelemy O, Pollack C, Cohen M, Zeymer U, et al. Efficacy and safety of enoxaparin versus unfractionated heparin during percutaneous coronary intervention: systematic review and meta-analysis. Bmj. 2012;344:e553.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  61. Suthar AB, Lawn SD, del Amo J, Getahun H, Dye C, Sculier D, et al. Antiretroviral therapy for prevention of tuberculosis in adults with HIV: a systematic review and meta-analysis. PLoS Med. 2012;9(7):e1001270.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  62. Te Morenga L, Mallard S, Mann J. Dietary sugars and body weight: systematic review and meta-analyses of randomised controlled trials and cohort studies. Bmj. 2013;346:e7492.

    Article  Google Scholar 

  63. Thomas RE, Jefferson T, Lasserson TJ. Influenza vaccination for healthcare workers who work with the elderly. Cochrane Database Syst Rev. 2010;17(2):Cd005187.

    Google Scholar 

  64. Tickell-Painter M, Maayan N, Saunders R, Pace C, Sinclair D. Mefloquine for preventing malaria during travel to endemic areas. Cochrane Database Syst Rev. 2017;10:Cd006491.

    PubMed  Google Scholar 

  65. Tricco AC, Zarin W, Cardoso R, Veroniki AA, Khan PA, Nincic V, et al. Efficacy, effectiveness, and safety of herpes zoster vaccines in adults aged 50 and older: systematic review and network meta-analysis. Bmj. 2018;363:k4029.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Vinceti M, Filippini T, Del Giovane C, Dennert G, Zwahlen M, Brinkman M, et al. Selenium for preventing cancer. Cochrane Database Syst Rev. 2018;1:Cd005195.

    PubMed  Google Scholar 

  67. Wilson A, Gallos ID, Plana N, Lissauer D, Khan KS, Zamora J, et al. Effectiveness of strategies incorporating training and support of traditional birth attendants on perinatal and maternal mortality: meta-analysis. Bmj. 2011;343:d7102.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Wilson HA, Middleton R, Abram SGF, Smith S, Alvand A, Jackson WF, et al. Patient relevant outcomes of unicompartmental versus total knee replacement: systematic review and meta-analysis. Bmj. 2019;364:l352.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Yank V, Tuohy CV, Logan AC, Bravata DM, Staudenmayer K, Eisenhut R, et al. Systematic review: benefits and harms of in-hospital use of recombinant factor VIIa for off-label indications. Ann Intern Med. 2011;154(8):529–40.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Zhang XL, Zhu L, Wei ZH, Zhu QQ, Qiao JZ, Dai Q, et al. Comparative efficacy and safety of everolimus-eluting bioresorbable scaffold versus everolimus-eluting metallic stents: a systematic review and meta-analysis. Ann Intern Med. 2016;164(11):752–63.

    Article  PubMed  Google Scholar 

  71. Zhang XL, Zhu QQ, Yang JJ, Chen YH, Li Y, Zhu SH, et al. Percutaneous intervention versus coronary artery bypass graft surgery in left main coronary artery stenosis: a systematic review and meta-analysis. BMC Med. 2017;15(1):84.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Ziff OJ, Lane DA, Samra M, Griffith M, Kirchhof P, Lip GY, et al. Safety and efficacy of digoxin: systematic review and meta-analysis of observational and controlled trial data. Bmj. 2015;351:h4451.

    Article  PubMed  PubMed Central  Google Scholar 

  73. Alexander DD, Miller PE, Van Elswyk ME, Kuratko CN, Bylsma LC. A meta-analysis of randomized controlled trials and prospective cohort studies of eicosapentaenoic and docosahexaenoic long-chain omega-3 fatty acids and coronary heart disease risk. Mayo Clin Proc. 2017;92(1):15–29.

    Article  CAS  PubMed  Google Scholar 

  74. Chung M, Lee J, Terasawa T, Lau J, Trikalinos TA. Vitamin D with or without calcium supplementation for prevention of cancer and fractures: an updated meta-analysis for the U.S. Preventive Services Task Force. Ann Intern Med. 2011;155(12):827–38.

    Article  PubMed  Google Scholar 

  75. Johnston BC, Zeraatkar D, Han MA, Vernooij RWM, Valli C, El Dib R, et al. Unprocessed red meat and processed meat consumption: dietary guideline recommendations from the Nutritional Recommendations (NutriRECS) Consortium. Ann Intern Med. 2019;171(10):756–64.

    Article  PubMed  Google Scholar 

  76. Chowdhury R, Warnakula S, Kunutsor S, Crowe F, Ward HA, Johnson L, et al. Association of dietary, circulating, and supplement fatty acids with coronary risk: a systematic review and meta-analysis. Ann Intern Med. 2014;160(6):398–406.

    Article  PubMed  Google Scholar 

  77. Schwingshackl L, Bröckelmann N, Beyerbach J, Werner SS, Zähringer J, Schwarzer G, et al. An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in nutrition research. Adv Nutr. 2022;nmac042. https://doi.org/10.1093/advances/nmac042.

  78. Trepanowski JF, Ioannidis JPA. Perspective: limiting dependence on nonrandomized studies and improving randomized trials in human nutrition research: why and how. Adv Nutr. 2018;9(4):367–77.

    Article  PubMed  PubMed Central  Google Scholar 

  79. Schwingshackl L, Balduzzi S, Beyerbach J, Bröckelmann N, Werner SS, Zähringer J, et al. Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study. Bmj. 2021;374:n1864.

    Article  PubMed  PubMed Central  Google Scholar 

  80. Bun RS, Scheer J, Guillo S, Tubach F, Dechartres A. Meta-analyses frequently pooled different study types together: a meta-epidemiological study. J Clin Epidemiol. 2020;118:18–28.

    Article  PubMed  Google Scholar 

  81. Cuello-Garcia CA, Morgan RL, Brozek J, Santesso N, Verbeek J, Thayer K, et al. A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments. J Clin Epidemiol. 2018;98:33–40.

    Article  PubMed  Google Scholar 

  82. Verde PE, Ohmann C. Combining randomized and non-randomized evidence in clinical research: a review of methods and applications. Res Synth Methods. 2015;6(1):45–62.

    Article  PubMed  Google Scholar 

  83. Sarri G, Patorno E, Yuan H, Guo JJ, Bennett D, Wen X, et al. Framework for the synthesis of non-randomised studies and randomised controlled trials: a guidance on conducting a systematic review and meta-analysis for healthcare decision making. Bmj. 2022;27(2):109–19.

    Google Scholar 

  84. Mueller M, D'Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, et al. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

    Article  PubMed  PubMed Central  Google Scholar 

  85. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Bmj. 2016;355:i4919.

    Article  PubMed  PubMed Central  Google Scholar 

  86. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. Bmj. 2019;366:l4898.

    Article  PubMed  Google Scholar 

  87. Schwingshackl L, Nagavci B, Stadelmaier J, Werner SS, Cuello Garcia CA, Schünemann HJ, et al. Pooling of cohort studies and RCTs affects GRADE certainty of evidence in nutrition research. J Clin Epidemiol. 2022;147:151–9.

    Article  PubMed  Google Scholar 

  88. Fleming PS, Koletsi D, Seehra J, Pandis N. Systematic reviews published in higher impact clinical journals were of higher quality. J Clin Epidemiol. 2014;67(7):754–9.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

None.

Funding

Open Access funding enabled and organized by Projekt DEAL. LS received by Deutsche Forschungsgemeinschaft (DFG): Projektnummer 459430615.

Author information

Author notes

  1. Nils Bröckelmann and Julia Stadelmaier contributed equally.

Authors and Affiliations

  1. Institute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany

    Nils Bröckelmann, Julia Stadelmaier, Louisa Harms, Charlotte Kubiak, Jessica Beyerbach, Jörg J. Meerpohl & Lukas Schwingshackl

  2. Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center - University of Freiburg, Freiburg, Germany

    Martin Wolkewitz

  3. Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany

    Jörg J. Meerpohl

Authors
  1. Nils Bröckelmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julia Stadelmaier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Louisa Harms
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Charlotte Kubiak
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jessica Beyerbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Martin Wolkewitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jörg J. Meerpohl
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lukas Schwingshackl
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

NB, JJM, and LS designed the research. NB, LH, CK, JB, and LS analyzed the data and wrote the first draft of the paper. NB, JS, LH, CK, JB, MW, JJM, and LS interpreted the data. The authors read and approved the final manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Corresponding author

Correspondence to Lukas Schwingshackl.

Ethics declarations

Ethics approval and consent to participate

Not applicable since we did not include any human subject.

Consent for publication

Not applicable since we did not include any human subject.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Appendix S1.

Search strategy. Tables S1-S5. Table S1. Explanation and definition for PI/ECO similarity degree. Table S2. PI/ECO similarity degree. Table S3. Differences between published (reported) effect estimates and re-calculated effect estimates. Table S4. Reason for exclusion from the pooling scenario. Table S5. Pooling results. Figures S1-S118. Fig S1. Forest plot: Low sodium (Intervention/Exposure); All-cause mortality (Outcome). Fig. S2. Forest plot: Low sodium; Cardiovascular disease. Fig. S3. Forest plot: Intra-aortic balloon pump; All-cause mortality. Fig. S4. Forest plot: Self-administered therapy; Treatment success. Fig S5. Forest plot: Self-administered therapy; Treatment completion. Fig. S6. Forest plot: Self-administered therapy; All-cause mortality. Fig S7. Forest plot: Antiretroviral therapy; HIV infection. FigS8-Forest plot: Nonnutritive sweeteners; Body Mass Index random sequence. Fig. S9. Forest plot: Surgical abortion by mid-level providers; Failure or incomplete abortion. Fig. S10. Forest plot: Surgical abortion by mid-level providers; Complications. Fig. S11. Forest plot: Surgical abortion by mid-level providers; Abortion failure and complications. Fig. S12. Forest plot: Clopidogrel pretreatment for percutaneous coronary intervention; All-cause mortality. Fig. S13. Forest plot: Clopidogrel pretreatment for percutaneous coronary intervention; Major bleeding. Fig. S14. Forest plot: Clopidogrel pretreatment for percutaneous coronary intervention; Coronary heart disease. Fig. S15. Forest plot: P2Y12 inhibitor pretreatment in non-ST elevation acute coronary syndrome; All-cause mortality. Fig. S16. Forest plot: P2Y12 inhibitor pretreatment in non-ST elevation acute coronary syndrome; Major bleeding. Fig. S17. Forest plot: P2Y12 inhibitor pretreatment in non-ST elevation acute coronary syndrome; Main composite ischemic endpoint. Fig. S18. Forest plot: Mediterranean diet; Breast cancer. Fig. S19. Forest plot: High calcium; All fractures. Fig. S20. Forest plot: High calcium; Verterbral fractures. Fig. S21. Forest plot: High calcium; Hip fracture. Fig. S22. Forest plot: Sigmoidoscopy; Colorectal cancer mortality. Fig. S23. Forest plot: Sigmoidoscopy; Colorectal cancer incidence. Fig. S24. Forest plot: High omega-3; Cerebrovascular disease. Fig. S25. Forest plot: High α-linolenic acid; Coronary heart disease. Fig. S26. Forest plot: High omega-3; Coronary heart disease. Fig. S27. Forest plot: Omega-6; Coronary heart disease. Fig. S28. Forest plot: High calcium; Cardiovascular mortality. Fig. S29. Forest plot: High dairy; Systolic blood pressure. Fig. S30. Forest plot: Radiation therapy; Erectile dysfunction. Fig. S31. Forest plot: Radical prostatectomy; Urinary incontinence. Fig. S32. Forest plot: Radical Prostatectomy; Erectile dysfunction. Fig. S33. Forest plot: Disease-modifying drugs; Conversion to clinically definite multiple sclerosis. Fig. S34. Forest plot: Extracranial-intracranial arterial bypass; All-cause mortality. Fig. S35. Forest plot: Extracranial-intracranial arterial bypass; Stroke. Fig. S36. Forest plot: Extracranial-intracranial arterial bypass; Stroke mortality or dependency. Fig. S37. Forest plot: Transcatheter aortic valve implantation; Early all-cause mortality. Fig. S38. Forest plot: Transcatheter aortic valve implantation; Mid-term all-cause mortality. Fig. S39. Forest plot: Transcatheter aortic valve implantation; Long-term all-cause mortality. Fig. S40. Forest plot: Treating gestational diabetes mellitus; High birth weight. Fig. S41. Forest plot: Treating gestational diabetes mellitus; Large-for-gestational age neonate. Fig. S42. Forest plot: Treating gestational diabetes mellitus; Shoulder dystocia. Fig. S43. Forest plot: Treating asymptomatic bacteriuria; Pyelonephritis. Fig. S44. Forest plot: Bacillus Calmette-Guérin vaccination; All-cause mortality. Fig. S45. Forest plot: Measles containing vaccines; All-cause mortality. Fig. S46. Forest plot: Total hip arthroplasty; Reoperation. Fig. S47. Forest plot: Total hip arthroplasty; Dislocation. Fig. S48. Forest plot: Total hip arthroplasty; Deep infection. Fig. S49. Forest plot: Chest-compression-only cardiopulmonary resuscitation; Survival. Fig. S50. Forest plot: Non-calcium-based phosphat binders; All-cause mortality. Fig. S51. Forest plot: Parenteral influenza vaccine; Influenza-like illness. Fig. S52. Forest plot: Parenteral influenza vaccine Influenza. FigS53-Forest plot: Inactivated influenza vaccines; Influenza. Fig. S54. Forest plot: Inactivated influenza vaccines; Influenza-like illness. Fig. S55. Forest plot: High total flavonoids; Colorectal neoplasms. Fig. S56. Forest plot: Transfusion; All-cause mortality. Fig. S57. Forest plot: Caesarean section; Urinary incontinence. Fig. S58. Forest plot: Caesarean section; Fecal incontinence. Fig. S59. Forest plot: Antiretroviral therapy by nurses; All-cause mortality. Fig. S60. Forest plot: Antiretroviral therapy by nurses; Attrition. Fig. S61. Forest plot: Nurses for maintenance of antiretroviral therapy; All-cause mortality. Fig. S62. Forest plot: Exenatide; Acute pancreatitis. Fig. S63. Forest plot: DDP-4 inhibitors; Heart failure. Fig. S64. Forest plot: DDP-4 inhibitors; Hospital admission for heart failure. Fig. S65. Forest plot: Tamoxifen; Heart failure. Fig. S66. Forest plot: SGLT-2 inhibitors; Acute kidney injury. Fig. S67. Forest plot: Erythropoiesis stimulating agents; Venous thromboembolism. Fig. S68. Forest plot: Erythropoiesis stimulating agents; All-cause mortality. Fig. S69. Forest plot: Pneumococcal polysaccharide vaccines; Invasive pneumococcal disease. Fig. S70. Forest plot: Neoral (Cyclosporin); Acute rejection of kidney transplant. Fig. S71. Forest plot: Early intervention for NSTE-ACS; All-cause mortality. Fig. S72. Forest plot: Early intervention for NSTE-ACS; Myocardial infarction. Fig. S73. Forest plot: Early intervention for NSTE-ACS; Major bleeding. Fig. S74. Forest plot: Caesarean section; Anal incontinence; feces. Fig. S75. Forest plot: Caesarean section; Anal incontinence; flatus. Fig. S76. Forest plot: Ceramic-on-ceramic bearings for total hip arthroplasty; Harris Hip Score. Fig. S77. Forest plot: High-flexion total knee arthroplasty; Flexion in degrees. Fig. S78. Forest plot: Gender-specific total knee arthroplasty; Flexion-extension range. Fig. S79. Forest plot: Second generation antipsychotics; Sedation. Fig. S80. Forest plot: Second generation antipsychotics; Neurologic outcomes. Fig. S81. Forest plot: Surgery for achilles tendon rupture; Re-rupture. Fig. S82. Forest plot: Surgery for achilles tendon rupture; Complications. Fig. S83. Forest plot: High vitamin D; Hypertension. Fig. S84. Forest plot: Carotid endarterectomy; Ipsilateral stroke. FigS85-Forest plot: Carotid endarterectomy; Stroke. Fig. S86. Forest plot: Carotid artery stenting; Periprocedural stroke. Fig. S87. Forest plot: Nasal deconolization; Surgical site infection. Fig. S88. Forest plot: Glycopeptide prophylaxis; Surgical site infection. Fig. S89. Forest plot: Enoxaparin; All-cause mortality. Fig. S90. Forest plot: Enoxaparin; Major bleeding. Fig. S91. Forest plot: Enoxaparin; All-cause mortality or myocardial infarction. Fig. S92. Forest plot: Antiretroviral therapy; Tuberculosis infection. Fig. S93. Forest plot: High sugar intake; Weight gain. Fig. S94. Forest plot: High sugar intake; Body Mass Index. Fig. S95. Forest plot: Influenza vaccines; Influenza-like illness. Fig. S96. Forest plot: Mefloquine; Discontinuation due to adverse effects. Fig. S97. Forest plot: Mefloquine; Serious adverse events or effects. Fig. S98. Forest plot: Mefloquine; Nausea. Fig. S99. Forest plot: Live-attenuated zoster vaccines; Suspected Herpes Zoster. Fig. S100. Forest plot: High selenium; Cancer. Fig. S101. Forest plot: High selenium; Cancer mortality. FigS102-Forest plot: High selenium; Colorectal cancer. FigS103-Forest plot: Training for traditional birth attendants/ assistance by traditional birth attendants; Perinatal mortality. Fig. S104. Forest plot: Training for traditional birth attendants/ assistance by traditional birth attendants; Neonatal mortality. Fig. S105. Forest plot: Unicompartimental knee arthroplasty; Venous thromboembolism. Fig. S106. Forest plot: Unicompartimental knee arthroplasty; Flexion-extension range. Fig. S107. Forest plot: Unicompartimental knee arthroplasty; Operation duration. Fig. S108. Forest plot: Recombinant factor VII; All-cause mortality. Fig. S109. Forest plot: Recombinant factor VII; Thromboembolism. Fig. S110. Forest plot: Everolimus-eluting bioresorbable vascular scaffold; Stent thrombosis. Fig. S111. Forest plot: Everolimus-eluting bioresorbable vascular scaffold; All-cause mortality. Fig. S112. Forest plot: Everolimus-eluting bioresorbable vascular scaffold; Coronary heart disease mortality. Fig. S113. Forest plot: Percutaneous coronary intervention; All-cause mortality. Fig. S114. Forest plot: Percutaneous coronary intervention; Cardiovascular mortality. Fig. S115. Forest plot: Percutaneous coronary intervention; Myocardial infarction. Fig. S116. Forest plot: Digoxin; All-cause mortality. Fig. S117. Forest plot: Digoxin; Cardiovascular mortality. Fig. S118. Forest plot: Digoxin; Hospital admission.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bröckelmann, N., Stadelmaier, J., Harms, L. et al. An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research. BMC Med 20, 355 (2022). https://doi.org/10.1186/s12916-022-02559-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12916-022-02559-y

Keywords

BMC Medicine

ISSN: 1741-7015

Contact us