Risk of bias in observational studies using routinely collected data of comparative effectiveness research: a meta-research study

Background To assess the completeness of reporting, research transparency practices, and risk of selection and immortal bias in observational studies using routinely collected data for comparative effectiveness research. Method We performed a meta-research study by searching PubMed for comparative effectiveness observational studies evaluating therapeutic interventions using routinely collected data published in high impact factor journals from 01/06/2018 to 30/06/2020. We assessed the reporting of the study design (i.e., eligibility, treatment assignment, and the start of follow-up). The risk of selection bias and immortal time bias was determined by assessing if the time of eligibility, the treatment assignment, and the start of follow-up were synchronized to mimic the randomization following the target trial emulation framework. Result Seventy-seven articles were identified. Most studies evaluated pharmacological treatments (69%) with a median sample size of 24,000 individuals. In total, 20% of articles inadequately reported essential information of the study design. One-third of the articles (n = 25, 33%) raised some concerns because of unclear reporting (n = 6, 8%) or were at high risk of selection bias and/or immortal time bias (n = 19, 25%). Only five articles (25%) described a solution to mitigate these biases. Six articles (31%) discussed these biases in the limitations section. Conclusion Reporting of essential information of study design in observational studies remained suboptimal. Selection bias and immortal time bias were common methodological issues that researchers and physicians should be aware of when interpreting the results of observational studies using routinely collected data. Supplementary Information The online version contains supplementary material available at 10.1186/s12916-021-02151-w.


Background
Though randomized control trials (RCTs) are considered to provide the best evidence in comparative effectiveness research (CER), they also have some limitations [1,2]. They can often be resource-intensive and timeconsuming. As such, RCTs may not be able to detect effects on long-term outcomes or rare events [3][4][5]. Observational studies using routinely collected data have been used to complement RCTs [5][6][7][8]. Routinely collected health data (RCD) are generated from the daily operations of healthcare systems, recorded without a priori research question [6]. A broad range of sources (e.g., disease registries, health administrative data, quality/safety surveillance databases, electronic health records, and pharmacy data) hosts such routinely collected data and contains both drug exposure and clinical outcomes to be used to provide evidence on treatment effectiveness.
However, observational studies are limited by their susceptibility to bias [5,[9][10][11]. Hernán et al. published a framework for using observational data to emulate a target trial, a hypothetical pragmatic trial [4,12]. The framework suggested researcher explicitly specifying key components of this hypothetical trial such as eligibility criteria, treatment assignment, and the start of followup. The time when patients fulfill the eligibility criteria is assigned to one of the treatment strategies (i.e., fulfill the criteria to be classified as exposure or control), and starting the follow-up should be aligned to mimic the randomization process in an RCT [3,4,12]. By avoiding methodological pitfalls, this approach reduces the risk of bias of the effect estimate and hence produces more reliable results [13]. Cochrane has adopted this framework in the assessment of the risk of bias for non-randomized intervention studies [14].
This study aimed to assess the completeness of reporting essential information of study design and risk of bias due to failure to mimic the randomization in observational studies using routinely collected data for comparative effectiveness research. We did not aim to assess the extent that the bias could influence the conclusion of the included studies. After systematically reviewing the reporting and conducting of observational studies, we propose a checklist to help readers and reviewers to identify common methodological pitfalls of observational studies.

Study design
We conducted a meta-research study and reviewed the comparative effectiveness observational studies evaluating the therapeutic interventions with the use of routinely collected data published in high impact factor journals. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [15].

Search strategy
We identified a convenience sample of the 7 highest impact factor journals of the InCites Journal Citation Reports categories medicine, general, and internal (New England Journal of Medicine, Lancet, JAMA, BMJ, Annals of Internal Medicine, BMC Medicine, and PLoS Medicine) and 3 highest impact factor journals in endocrinology and metabolism (Lancet Diabetes & Endocrinology, Diabetes Care, and Diabetes) and cardiac and cardiovascular systems (European Heart Journal, Journal of American College of Cardiology, and Circulation) that cover research on high prevalent diseases.
As all these ten journals were indexed on PubMed, we conducted a search on PubMed to identify the observational studies evaluating a comparative effectiveness question. To reflect contemporary reporting practices and methodological conduct, the search was narrowed to studies published between 01/06/2018 and 30/06/ 2020. The full search strategy is presented in Additional file 1: Table S1.

Eligibility criteria
We included cohort studies which evaluated a therapeutic intervention by using RCD [6]. Studies were eligible for inclusion if they (1) evaluated a therapeutic intervention, defined as a treatment-related to healing a disease, i.e., pharmaceuticals, surgery; (2) used RCD as the data source; and (3) answered a comparative effectiveness question, i.e., research aiming to identify which interventions work best for improving health. Studies that did not answer CER questions, studies without an abstract, and retracted papers were excluded. The inclusion and exclusion criteria for study selection are provided in Additional file 1: Table S2.

Study screening and selection
One reviewer (ME) screened all the titles and abstracts of the studies retrieved. A second reviewer (VNT) screened a sample of 775 (57%) of 1357 articles excluded by ME. There was good agreement between the two reviewers with only 1 conflict. Then, each of the full texts was assessed by two of three reviewers (ME, VNT, MD) to ensure the eligibility of the study for data extraction. All conflicts were resolved through discussion, and a third reviewer was available to adjudicate. Literature search results were imported into Mendeley (https:// www.mendeley.com) to store, organize, and manage all references. The screening process was aided by the use of the Rayyan software [16].

Data extraction
Data from each article were extracted independently by two of the three reviewers (ME, VTN, and MD) using a standardized form created based on the framework for emulating a target trial proposed by Hernán et al. and RECORD-PE reporting guideline for observational studies using routinely collected data for pharmacoepidemiology [4,12,14,17]. The form was initially piloted and refined throughout the process (Additional file 1: Table S3 data extraction form and Additional file 1: Table 1 Situations when time points of eligibility, treatment assignment, and the start of follow-up are not aligned

Situations
Type of bias that might arise a. Follow-up starts after eligibility criteria completion and treatment assignment. This situation happens when the follow-up starts after eligible individuals have started the treatment. The follow-up time is left-truncated, and individuals who experience early outcomes after starting treatment are not captured.
Prevalent user bias b. Follow-up starts at eligibility but after treatment assignment. This situation happens when the follow-up starts after individuals have started the treatments, which means that the follow-up time is left-truncated. Additionally, individuals are selected based on post-treatment criteria (e.g., individuals have no outcome that occurred before the start of the follow-up).
Prevalent user bias and selection bias due to post-treatment eligibility c. Follow-up starts before treatment assignment and eligibility. This situation happens when individuals need to meet the eligibility criteria after the follow-up has started and individuals have started treatments. For example, patients have to receive at least 2 consecutive prescriptions of treatment to be included in the analysis, but follow-up starts from the first prescription. Those who have an outcome within this time are excluded from the analysis leading to immortal time bias, or those who stop treatment after the first prescription are excluded leading to selection bias.
Immortal time bias and selection bias due to post-treatment eligibility d. Follow-up starts at eligibility, but treatment is assigned later. This situation happens in two cases: 1) When there is a grace period, a period from when individuals meet the eligibility to when they start treatments. For example, a study compares no antibiotic use with initiation of antibiotic use within 7 days since diagnosis of urinary tract infection. If an individual starts antibiotics on day 7, it means that they have survived for 7 days leading to immortal time bias.
2) When individuals have to use the treatment for a given period to be classified in the exposed group. For example, individuals have to fill three consecutive prescriptions of aspirin to be classified as an aspirin user group, and non-aspirin users, otherwise. This also leads to immortal time bias. Another issue that might arise from this situation is the risk of misclassification of treatment. For instance, in the example of initiating antibiotics within 7 days since diagnosis, if the individual has an outcome before day 7 and has not started the antibiotic, we are uncertain to classify her/him to the no-antibiotic user or antibiotic user. Table S4 explanation of data items). Any disagreement was discussed with senior researchers (RP, IB) to reach a consensus. The following data were extracted from the selected papers:

Immortal time bias and misclassification of treatment
1. Study characteristics: title, year of publication, author, location of the corresponding author, name of the journal, study design (longitudinal study), treatment type, comparator, funding source (i.e., public, private funding), and data source 2. Research transparency practices: use of reporting guidelines, access to codes and algorithms to classify exposures and outcomes, and data sharing policy 3. Reporting of essential items: (a) Diagram to illustrate the study design (i.e., describing the time of eligibility, treatment assignment, and follow-up). (b) Eligibility criteria, and particularly whether individuals with contraindication to one of the evaluated treatments, were explicitly excluded as in an RCT. (c) Methods used to adjust for confounding (i.e., regression, propensity score, inverse probability weighting). (d) Causal contrast of interest (i.e., intention-totreat effect, per-protocol effect). (e) Time points of eligibility (i.e., when individuals were evaluated regarding their eligibility), treatment assignment (i.e., when individuals were classified to one of the treatment groups), and the start of follow-up (i.e., when individuals started outcome assessment). 4. After determining the time points of eligibility, treatment assignment, and the start of follow-up, we assessed if these time points were aligned to avoid bias. We identified the type of bias that might arise when they were not aligned (Table 1) and whether the authors described a solution to address bias.

Data synthesis
Categorical data were summarized using frequencies and percentages. Interrater reliability was tested using Cohen's kappa [18]. Descriptive analysis was completed in R (version 4.0.2).

Data sharing
Data of this study will be available on Zenodo after the publication of the article.

Patient involvement
Patients and public members were not involved in this study.

Study characteristics
Among the 1465 articles retrieved from the search, 77 articles were selected for data extraction after screening for the title, abstract, and full text ( Fig. 1).

Research transparency practices
Only seven articles (9%) mentioned the use of a reporting guideline. Fifty-three articles (69%) provided codes (e.g., ICD-10 codes) used to classify both exposures and outcomes. Ten articles (13%) indicated that data were available upon request ( Table 2). Reporting essential information of the target trial Only 18% (n = 14/77) reported a diagram to illustrate the study design and reported the three essential time points (i.e., eligibility, treatment initiation, start of follow-up). Eighteen percent (n = 14/77) did not report completely essential time points, i.e., the start of followup, when individuals completed the eligibility criteria and when patients started the treatments of interest. Regarding the inclusion criteria, only 12% (n = 9/77) reported the exclusion of patients with contraindication to one of the evaluated interventions. Only one article explained the reason for not excluding patients with such a contraindication, due to the inability to identify these patients from the dataset. Sixty-five percent of articles (n = 50/77) did not specify the type of causal contrast estimated ( Table 3).

Risk of bias due to failure of specifying a target trial
Overall, 33% (n = 25/77) raised concerns about the risk of bias. Of these, in one-fourth (n = 6/25), as the start of follow-up was not clearly reported, we could not determine if eligibility, treatment assignment, and the start of follow-up were synchronized ( Fig. 1). In 76% (n = 19/ 25), the time when patients completed the eligibility criteria, initiated the treatments, and the start of follow-up was not aligned (Fig. 1). Among these 19 articles, in four articles (n = 4/19, 21%), the follow-up started when patients met eligibility but after patients initiated treatment (Table 1 (b)), which led to prevalent user bias and selection bias due to post-treatment eligibility [19][20][21][22]. The authors did not describe any solutions to address these biases in these four articles.
In seven articles (n = 7/19, 37%), the follow-up started when patients initiated treatment but before patients met the eligibility criteria leading to immortal time bias and selection bias due to post-treatment eligibility (Table 1 (c)) [23][24][25][26][27][28][29]. Among these, one article reported handling treatment exposure as a time-dependent variable to account for immortal time bias; however, this strategy was inadequate to account for selection bias due to post-treatment eligibility [25]. One article performed a sensitivity analysis to include participants who were excluded based on the post-treatment eligibility criteria and yielded similar results to the main analysis [27].  (17) -Annals of Internal Medicine 9 (11) -BMJ 14 (18) -PLoS Medicine 8 (11) -Circulation 7 (9) -European Heart Journal 4 (5) -  In seven articles (n = 7/19, 37%), follow-up started when patients met the eligibility criteria, but patients were assigned to one of the treatment groups after the start of the follow-up, a situation both at risk of immortal time bias and misclassification of treatment (Table 1 (d)) [30][31][32][33][34][35][36][37]. Of these, four articles did not mention any solutions leading to high risk of selection bias [31,32,35,37]; three articles treated treatment exposure as a time-dependent variable [30,33,36] which was inadequate to address the risk of misclassification, and one article randomly assigned individuals who had outcomes before treatment initiation to one of the two treatment groups [34] to mitigate the risk of bias. In one article (n = 1/19, 5%), individuals could start the treatment both before and after eligibility and the start of follow-up (Table 1 (b and  d)); thus, the study was at risk of prevalent user bias and immortal time bias [38]. No solution was described in this article. Among these 19 articles that we identified biases, six articles (32%) discussed these biases in the limitations section (Fig. 2). Table 4 presents the main features of 19 studies without synchronization of eligibility, treatment assignment, and follow-up.

Discussion
Our review showed that 20% (n = 14/77) of the articles did not adequately report essential information of the study design. A third of reviewed articles had unclear risk of bias or high risk of selection bias and/or immortal time bias due to the choice of the time of eligibility, treatment assignment, and the start of follow-up that failed to mimic the randomization. In only 25% of the articles at risk of bias, a solution was described; however, these solutions were not adequate to eliminate the risk of bias completely. The lack of synchronization arises when investigators want to select individuals who might have better treatment adherence, i.e., select only individuals who adhered to the treatment for a given period (Table 5 (c)), or only individuals who have adhered to the treatment for a given period are classified as exposed (Table 5 (d)). To address the selection bias caused by using a post-treatment event to include individuals or predict treatment strategies in the future, Hernan et al. proposed creating a clone, i.e., an exact copy of the population, assign them to one of the treatment groups and censor when they deviate from the assigned treatment [12].
Another common reason for the lack of synchronization in observational studies using routinely collected data is due to having a grace period, i.e., individuals start to use treatment within a given period after the start of followup and eligibility (Table 5 (d)); thus, investigators can increase the number of eligible individuals. For example, to compare the effectiveness of hydroxychloroquine versus Fig. 2 The number of studies at risk of bias due to lack of synchronization. Nineteen (25%) studies had a high risk of bias due to the lack of synchronization. Of these, 14 proposed no solution, and 5 used inadequate methods to address the bias. Six studies inadequately reported to enable the assessment of synchronization. Fifty-two (68%) studies had low risk of bias       a. Follow-up starts after eligibility criteria completion and treatment assignment which leads to prevalent user bias.
b. Follow-up starts at eligibility but after treatment assignment which leads to prevalent user bias and selection bias due to post-treatment eligibility.
Select new users and ensure that individuals are not selected by an event that happens after the follow-up starts [12].
c. Follow-up starts before treatment assignment and eligibility which leads to immortal time bias and selection bias due to post-treatment eligibility.
Keep all individuals who start the treatment since the start of follow-up, create an exact copy of the population, assign them to one of the intervention groups from the start of the follow-up, and censor when they start to deviate from assigned treatment [39]. One strategy which is often used to account for immortal time bias in literature is to consider exposure as a time-dependent variable. However, this strategy is not adequate to address the risk of selection bias due to post-treatment eligibility, as an uncensored group might not be exchangeable with the censored group [3].
d. Follow-up starts at eligibility, but treatment is assigned later which leads to immortal time bias and misclassification of treatment.
1) Randomly assign individuals to one of the treatment strategies [12].
2) Create an exact copy of the population, assign them to one of the intervention groups from the start of the follow-up, and censor when they start to deviate from assigned treatment [39]. One strategy which is often used to account for immortal time bias is to consider exposure as a time-dependent variable. However, this strategy is inadequate to address the risk of misclassification, because if individuals have outcomes during the grace period, we are uncertain which intervention group they should be classified into. -Check if any events after the start of follow-up are listed in the eligibility criteria, e.g., complete 2 consecutive prescriptions or no outcome for the first 2 months after the start of follow-up.
3. When are individuals assigned to an exposed or nonexposed group?
-Check if the authors report clearly when individuals are classified as exposed or nonexposed group.
3a. Do individuals have to use treatment for a given period to be classified as an exposed group?
-Check if individuals have to use treatment for a given period, e.g., complete 2 consecutive prescriptions to be classified as exposed and non-exposed, if not, start the treatment or complete only 1 prescription.
3.b. Is there a grace period? -Check if individuals can start the treatment sometime after the start of follow-up and eligibility.
standard of care in the treatment for COVID-19 patients, the number of patients who initiated hydroxychloroquine immediately after hospital admission would be quite low.
To increase the number of eligible patients for the analysis, investigators allowed for a grace period to assign patients who started hydroxychloroquine within 48 h since admission to the intervention group [34,35]. However, a challenge of having a grace period is that we could not assign patients to one of the intervention groups at the start of the follow-up as in an RCT. If a patient had an outcome within 48 h since admission, it is uncertain if they should be classified as exposed or control group. To overcome the challenge of having a grace period, Hernan et al. also recommended following the strategy as above, i.e., to create an exact copy of the population, assign them to one of the intervention group, censor when they start to deviate from assigned treatment, and use inverse probability weighting to adjust for post-treatment censoring bias [12,39] (Table 5). However, the use of such an approach was never reported in our sample. Although Hernan et al. proposed this approach in 2016, there are only a few studies applying this approach due to methodological and logistical challenges. Maringe et al. provided a detailed tutorial to perform the cloning strategy [40]. Additionally, the emulated trial framework highlights the importance of the new-user design by identifying all eligible in the defined population who started the study treatments to avoid these biases. The selection of only new users, however, might reduce the sample size and the study power [41,42]. To address this challenge, sensitivity analysis could be used to assess the magnitude of potential bias related to including prevalent users [41,42].
Furthermore, some other essential information was missing in the report of observational studies in our sample, particularly specifying if patients with contraindication with one of the evaluated treatments were excluded from the analysis. This issue could be problematic as we are uncertain if patients in different treatment groups were comparable. For example, in a study, patients who had contraindication with evaluated treatments were classified as the control group [43]. It means that patients in the intervention and control groups were not exchangeable, which violated a fundamental condition of causal inference.
Previous studies have also highlighted the incomplete reporting and potential bias in the implementation of observational studies. Luijken et al. found that 6% of the evaluated observational studies did not specify if new users or prevalent users were included, and in only half of the studies using new user design, time point of eligibility, treatment initiation, and start of follow-up were synchronized [44]. Due to these avoidable methodological pitfalls, the results of observational studies could be biased and mislead healthcare decisions [45]. The emulated trial framework which relies on synchronization of eligibility, treatment assignment, and the start of follow-up to mimic the randomization of RCT can help in reducing the risk of bias. However, the approach proposed by Hernan has also some limitations particularly in some situations, synchronization of the time points of the eligibility criteria, start of treatment, and start of follow-up is not feasible. By explicitly reporting these components and the decision made when emulating the target trials, researchers could help readers in assessing the extent that results might be influenced by bias and whether the choice of methodology to address this bias was appropriate to ensure the validity of results. We propose a checklist following the framework of emulated trials to help readers and reviewers to identify the common pitfalls of observational studies ( Table 6).
Our study has some limitations. First, to ensure the feasibility of the study, we restricted the search to high impact factor journals, which might underestimate the prevalence of bias due to the lack of synchronization of eligibility, treatment assignment, and start of follow-up. However, our aim is to raise awareness of the common problems of reporting and conducting observational studies using RCD that need to be addressed in future research. Second, we were unable to determine the magnitude of the bias. For example, if there are more individuals who have outcomes during the grace period, the effect estimates would be at higher risk of bias, because these individuals are more likely to be classified in the control group. Third, we did not evaluate the risk of confounding in the included studies. Nevertheless, the emulated trial framework and the cloning strategy can address the confounding bias.

Conclusions
In conclusion, reporting of essential information of the study design in observational studies remained suboptimal. The lack of synchronization of eligibility, treatment assignment, and the start of follow-up is common among observational studies, which leads to different types of bias such as prevalent user bias, immortal time bias, and selection bias due to post-treatment eligibility. Researchers and physicians should critically appraise the results of observational studies using routinely collected data.