Public availability of results of observational studies evaluating an intervention registered at ClinicalTrials.gov

Background Observational studies are essential for assessing safety. The aims of this study were to evaluate whether results of observational studies evaluating an intervention with safety outcome(s) registered at ClinicalTrials.gov were published and, if not, whether they were available through posting on ClinicalTrials.gov or the sponsor website. Methods We identified a cohort of observational studies with safety outcome(s) registered on ClinicalTrials.gov after October 1, 2007, and completed between October 1, 2007, and December 31, 2011. We systematically searched PubMed for a publication, as well as ClinicalTrials.gov and the sponsor website for results. The main outcomes were the time to the first publication in journals and to the first public availability of the study results (i.e. published or posted on ClinicalTrials.gov or the sponsor website). For all studies with results publicly available, we evaluated the completeness of reporting (i.e. reported with the number of events per arm) of safety outcomes. Results We identified 489 studies; 334 (68 %) were partially or completely funded by industry. Results for only 189 (39 %, i.e. 65 % of the total target number of participants) were published at least 30 months after the study completion. When searching other data sources, we obtained the results for 53 % (n = 158; i.e. 93 % of the total target number of participants) of unpublished studies; 31 % (n = 94) were posted on ClinicalTrials.gov and 21 % (n = 64) on the sponsor website. As compared with non-industry-funded studies, industry-funded study results were less likely to be published but not less likely to be publicly available. Of the 242 studies with a primary outcome recorded as a safety issue, all these outcomes were adequately reported in 86 % (114/133) when available in a publication, 91 % (62/68) when available on ClinicalTrials.gov, and 80 % (33/41) when available on the sponsor website. Conclusions Only 39 % of observational studies evaluating an intervention with safety outcome(s) registered at ClinicalTrials.gov had their results published at least 30 months after study completion. The registration of these observational studies allowed searching other sources (results posted at ClinicalTrials.gov and sponsor website) and obtaining results for half of unpublished studies and 93 % of the total target number of participants. Electronic supplementary material The online version of this article (doi:10.1186/s12916-016-0551-4) contains supplementary material, which is available to authorized users.


Background
Failure to provide access to research results is a key source of wasted research [1]. The results for more than 50 % of clinical trials are never published, and publication is more likely for clinical trials with statistically significant (positive) than negative results [2][3][4][5][6]. Lack of the availability of research findings has serious consequences; it affects the results of systematic reviews and meta-analyses and distorts the evidence used for the prioritization of research questions and clinical and policy decision-making [7][8][9]. In response to this waste, in 2005, the International Committee of Medical Editors required the registration of all clinical trials, before study inception, in a publicly accessible register such as Clini-calTrials.gov [10,11]. In 2007, the US Food and Drug Administration Amendments Act also required the posting of clinical trials results for all phase II to IV trials of drugs, biologic treatments and devices having at least one site in the United State at ClinicalTrials.gov no later than 1 year after the date of final collection of data for the pre-specified primary outcome [12]. In Europe, a new law that will be implemented in 2016 will require that all clinical trials be registered on a publicly accessible European Union clinical-trials register before they can begin, with a summary of trial results posted within a year after the end of the trial.
These policies have an important impact and allowed increase research value. In fact, thanks to trials registration, unpublished studies are identified, and their results could be made available through posting. However, registration is currently mandatory only for clinical trials.
Observational studies such as cohort and case-control studies are important for assessing the intervention effect [13][14][15][16]. They are particularly useful designs when randomized controlled trials (RCTs) are not feasible or when assessing rare adverse events and long-term effectiveness. Such studies represent a large part of the published literature and outnumber published RCTs [17]. Nevertheless, prospective registration of observational studies is not currently requested [18]. Despite not being mandatory, more than 35,000 observational studies are registered at ClinicalTrials.gov.
Our hypothesis is that registration of observational studies evaluating an intervention in ClinicalTrials.gov is important to increase research value as it allows identifying unpublished studies and obtaining unpublished results.
The main objectives of this study were (1) to evaluate whether results of observational studies evaluating an intervention with safety outcome(s) registered at Clini-calTrials.gov were published and, if not, whether they were available through posting on ClinicalTrials.gov or the sponsor website; (2) to evaluate and compare the time to publication and to public availability of results after searching other sources by study funding source; and (3) to evaluate the completeness of reporting of the outcomes designated as safety issues.

Methods
We identified a cohort of observational studies evaluating an intervention with safety outcome(s) registered at ClinicalTrials.gov.

Search for relevant studies
We searched ClinicalTrials.gov on April 14, 2014, by using "completed" for recruitment, "observational studies" for study type, and date of first registration between October 1, 2007, and December 31, 2011, and "has an outcome measure designated as a safety issue" in the safety issue field of ClinicalTrials.gov. We chose October 2007 because modifications were made at this time to the design-specific data elements used for registering observational studies on ClinicalTrials.gov. These changes were strongly influenced by protocol-related items in the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement [13].
Observational studies are defined by ClinicalTrials.gov as studies where "investigators assess health outcomes in groups of participants according to a protocol or research plan. Participants may receive interventions, which can include medical products, such as drugs or devices, or procedures as part of their routine medical care, but participants are not assigned to specific interventions by the investigator (as in a clinical trial https://clinicaltrials.gov/ct2/about-studies/glossary#O)."

Identification of relevant observational studies
Among the studies retrieved, we identified all observational prospective studies assessing interventions that had a primary completion date (i.e. the date when the final patient was examined or received an intervention for the purposes of final collection of data for the primary outcome) between October 1, 2007, and December 31, 2011. We defined interventions as all pharmacologic and non-pharmacologic treatments (pharmaceutical drugs, surgery, education, rehabilitation, etc.) aimed at improving the participants' health. We chose this time period to be able to investigate public availability (i.e. at least 24 months after the study primary completion). We excluded studies with a primary completion date after December 2011, studies assessing genetics, predictors, or risk factors; not assessing interventions, pharmacodynamics, and pharmacokinetics of drugs; phase 0, I, II, I/II and II/III studies; studies with healthy volunteers, and retrospective and randomized studies. We also excluded studies for which the primary completion date was not reported at Clinical-Trials.gov. The selection process was performed by one researcher, and all records included were independently verified by a second researcher.

Extraction of data from ClinicalTrials.gov
We downloaded from ClinicalTrials.gov the following data concerning the characteristics of the studies: clinical trial number (NCT), title, study design (defined with the observational model: cohort, case-control, case-only, case crossover or other; and time perspective: prospective, cross-sectional or other), enrollment (i.e. sample size), first received date, primary completion date, results first received date, condition and intervention under study, outcome measures, locations of recruitment and funding source. The funding source was classified at Clini-calTrials.gov as "NIH" (National Institutes of Health), "US federal", "industry", and other non-industry organizations (universities, hospitals, foundations, and other government and other non-industry organizations). We secondarily categorized funding sources as non-industry (i.e. funded by NIH, US federal, other non-industry organizations) or industry (i.e. partially or totally funded by industry).
One researcher classified the following information from the full ClinicalTrials.gov record: medical field, type of intervention (drug, device, procedure/surgery, other), location of recruitment (Europe, North America, South America, Africa, Asia, Oceania) and study purpose as safety, efficacy or both. As a quality assurance procedure, a second researcher independently verified 50 % of the data.

Publication of study results in journals
For each observational study identified, we systematically searched for a publication reporting the study results (search date June 2014). First, we examined the "publication" field at ClinicalTrials.gov to search for a citation for an article that described the study results. If no citation was reported, we searched MEDLINE via PubMed by using the ClinicalTrials.gov identification number (NCT). If no publication was identified, we searched MEDLINE via PubMed by using keywords for the intervention under study and the condition. A researcher screened all citations retrieved up to the primary completion date registered at ClinicalTrials.gov and selected all citations corresponding to the selected study. A second researcher independently performed the search on PubMed for all the studies for which no publication was identified; any discrepancies were discussed until obtaining consensus.
Finally, if no publication was identified, we contacted the sponsor or the principal investigator. We searched in the "additional information" field of ClinicalTrials.gov for a link to the sponsor website for contacting the sponsor. If no link was available, we recorded the principal investigator's name from the "contacts and locations" fields of ClinicalTrials.gov and searched PubMed and Google to identify their email address. The email reminded the recipient of the study NCT number and inquired about the study publication, presentation at a congress, and plans for future publication (Additional file 1: Appendix 1). We systematically sent two reminders. If no answer was received, the study findings were considered unpublished. When the study results were reported as an abstract or poster presented at a scientific meeting, we considered that the results were not available. In fact, previous evidence showed that the quality of abstracts presented at meetings is suboptimal, frequently including results that are not the final results [19,20].
To determine whether the publication(s) corresponded to the registered observational study, we retrieved the full-text article for all citations selected and assessed from the abstract and the full text, if needed, whether a combination of information, including description of interventions and conditions, population, location, responsible party, number of participants, primary outcome measures, and primary funding sponsor partly or completely matched the information at ClinicalTrials.gov. We only selected articles that reported the results of the study. All cases were assessed by a second independent researcher, and disagreements were resolved by consensus.
If the publication highlighted that the study did not actually fulfill the inclusion criteria, while it was impossible to detect it from ClinicalTrials.gov record, the studies were still included in the analysis and considered as published to avoid bias. This occurred for eight studies (randomized n = 6; retrospective n = 1; phase 0/I/II n = 1).

Search of results posted on ClinicalTrials.gov and the sponsor website
For each observational study identified, we assessed whether unpublished results were publicly available from other sources. For this purpose, (1) we searched whether results were posted at ClinicalTrials.gov and (2) systematically searched the sponsor website to identify whether the results were available. For this, we searched Clinical-Trials.gov for a link to the sponsor website. If no link was available, we searched Google using the sponsor name to identify the sponsor website. Then, we searched the website for a section dedicated to access to study results and used the study NCT number to find the study results.

Main outcomes
The main outcomes were the time to the first publication in journals, and the time to the first public availability of the study results.
The time to the first publication in journals was the time (in months) that elapsed between the primary completion date of the study and the publication. The study primary completion date was obtained from ClinicalTrials.gov.
The publication date was the first date an article was made available online ahead of print (i.e. epub date indexed on PubMed) or published in a paper-printed version.
The time to the first public availability of the study results was the time (in months) between the primary completion of the study and the first public availability of the study results by publication or posting on Clini-calTrials.gov or the sponsor website. When results were available in different sources, we used the first date.
Secondary outcomes were the proportion of study results publicly available (via publication or posting on ClinicalTrials.gov or the sponsor website) at 12 and 24 months after primary completion.

Completeness of reporting of outcomes designated as safety issues
For all studies with results publicly available (via publication or posting on ClinicalTrials.gov or the sponsor website), we evaluated the completeness of reporting of all primary outcome measure(s) designated as a "safety issue" and when not available, all secondary outcome measure(s) designated as a safety issue as recorded from ClinicalTrials.gov. Then, for each outcome recorded, we systematically searched in the report available (i.e. publication, and for unpublished studies, results posted on ClinicalTrials.gov or the sponsor website) whether the results were adequately reported (i.e. reported with the number of events per arm), partially reported (i.e. reported with the number of events pooled or only mentioned), or not reported. When several publications were available, we selected the publication with the results more completely reported.
For all studies with results publicly available, we determined the proportion of studies with all primary outcomes designated as safety issues adequately reported and when not available, the proportion of studies with all secondary outcomes adequately reported.
For this analysis, we excluded studies when the results were not published or available in English.

Statistical analysis
Quantitative variables are described with median (quartile 1-3; Q1-Q3) and qualitative variables with number and percentages. For each outcome, we assessed Kaplan-Meier estimates of the cumulative incidence of studies (with 95 % confidence interval (CI)) at 12 and 24 months. All studies without results available in the different sources were censored on June 1, 2014 (i.e. search date). Cumulative incidence curves estimated by Kaplan-Meier methods are displayed globally and by funding source (partially or completely industry-funded; non-industry-funded

Study selection and characteristics
From the 1,243 citations screened, 489 studies were selected (Fig. 1). The characteristics of the selected studies are shown in Table 1. Only about one-quarter of the studies were registered before the start date of the study; 30 % were registered after the study primary completion date. Overall, 333 studies (68 %) were conducted in high-income countries, 111 (23 %) with at least one site in North America, and 68 % (n = 334) were partially or completely funded by industry. The median (Q1-Q3) sample size was 300.0 (75.0-1045.0); 507.0 (131.0-1794.0) for studies with industry funding and 80.0 (32-219.0) for those with nonindustry funding. The cumulative target enrollment for all studies was 1,214,218 participants. The study objective in 72 % (n = 351) of studies was to assess safety.

Publication of study results
The median time between the study primary completion date and our search was 49.0 (Q1-Q3, 41.0-62.0) months; for all studies, at least 30 months had elapsed since the study primary completion date. Among the 489 studies,   Overall public availability of study results The cumulative proportion of studies with results publicly available over time overall and stratified by funding source is shown in Fig. 4. Overall, the cumulative percentage of studies with results publicly available was 27.0 (23.1-30.9) at 12 months and 51.7 (47.3-56.2) at 24 months; the proportion for industry-and nonindustry-funded studies was 30.8 % (25.9-35.8) versus 18.7 % (12.6-24.8) at 12 months and 55.4 % (50.1-60.7) versus 43.9 % (36.0-51.7) at 24 months. After adjusting for confounding variables, the probability of having results publicly available did not differ between industryfunded and non-industry-funded studies (adjusted HR, 1.16; 95 % CI, 0.88-1.52; P = 0.30). Additional file 1: Appendix 3 reports the availability of the study results according to the date of registration (before the start of the study, between the start and the study completion, after the study completion). Overall, 43 % of the studies that were registered after the study primary completion date were published, 33 % were not published but had their data available in other sources, and 23 % had no data available.

Completeness of reporting (Table 2)
Of the 347 studies with results publicly available, 243 reported at least one primary outcome measure designated as a "safety issue". We evaluated the completeness of reporting of 242 studies (one excluded because not published in English): all these outcomes were adequately reported in 86 % (114/133) when available in a publication, 91 % (62/68) when available on Clinical-Trials.gov, and 80 % (33/41) when available on the sponsor website. Of the 96 studies that did not have a primary outcome measure designated as a "safety issue", 66 (69 %) adequately reported all secondary outcomes designated as a safety issue; 60 % when available in a publication, 92 % when available on ClinicalTrials.gov, and 61 % when available on the sponsor website.

Discussion
We evaluated the public availability of study results in a cohort of 481 observational studies evaluating an intervention with safety outcome(s) registered at Clinical-Trials.gov and completed for more than 30 months. Only 39 % (n = 189) had results published, corresponding to 65 % of the total target number of participants. The To our knowledge, this is the first large study evaluating the public availability of the results of observational studies registered at ClinicalTrials.gov in terms of publication or posting on ClinicalTrials.gov or the sponsor's website. Most evidence of the lack of availability of research results has focused on the publication of clinical trials and the posting of results on ClinicalTrials.gov [5,6,[21][22][23][24]. Large cohorts of registered clinical trials showed that the results of only 46-63 % are published [21,23]. In the field of diagnostic studies, results for 54 % of studies completed for at least 18 months were published [25].
Our study has several important implications; they clearly illustrate the need for a change in policy with a Fig. 4 Overall public availability of study results (publication available or results posted on ClinicalTrials.gov or the sponsor website) and by funding source request to also prospectively register observational studies. In many situations, observational studies are the only data available because RCTs are not appropriate or feasible. The number of published meta-analyses including observational studies in health has increased substantially and these meta-analyses could be used to inform clinical decision-making and public health policy [18,26]. Registration of observational studies is debated, as shown by recent editorials published in major medical journals [17,[27][28][29][30][31][32]. However, much of the rationale for the prospective registration of clinical trials [18,33] also applies to the registration of observational studies. Although registration at ClinicalTrials.gov does not guarantee that trial results will be published in a timely manner [22,[34][35][36], it allows knowing the existence of the study, searching for unpublished data, exploring publication bias, outcome reporting bias, and fidelity to the protocol [37,38].
Our results also highlight the need to reconsider the strategy used to identify research findings. In fact, searching the results section at ClinicalTrials.gov as well as the sponsor website increased by two-fold the number of studies with results available and allowed access to the data of 93 % of the total target number of participants. It is consequently very important that systematic reviewers search for these data in trials registries and the sponsors websites. Previous studies comparing the results posted and published for clinical trials showed that results are more completely reported in registries than in publications [21,23] and that discrepancies between the ClinicalTrials.gov results database and matching publications are common [38]. However, it is unclear which source is more accurate [38].
Publication of research findings in peer-reviewed journals is considered essential for disseminating research results. However, the publication process is long, complicated and supposes an important investment from the sponsor and investigator. Lack of or delayed publication could be related to the lack of incentives to disseminate negative results, time constraints, limited resources, changing interests, or difficulties and failure to have the results published [34]. The sponsor may prefer posting results on a website than investing in publication in a peer-reviewed journal. However, we question why sponsors post results on their website and not on Clinical-Trials.gov. In fact, ClinicalTrials.gov performs a quality control, which is not the case with the sponsor website.
Finally, ClinicalTrials.gov offers researchers the opportunity to provide access to their data if they decide not to publish them in a peer-reviewed journal. For clinical trials, the posting of results is a requirement. The World Health Organization is calling for a strict timeline for public disclosure of clinical trial results and published a new Statement on the Public Disclosure of Clinical Trial Results, which specifies that study results be reported at least 30 months after a study is completed. This requirement should also be extended to observational studies.
Our study has some potential limitations. First, as registration of observational studies is not mandatory, we could explore the public availability of the trial results only for observational studies registered at Clini-calTrials.gov. However, the publication rate and public availability of results is not likely to be higher for observational studies that were not registered. Second, our search was performed only in Medline via PubMed and we may have missed some publications. However, Medline is the largest database of biomedical journals and is the source that nearly all physicians and policymakers use to access clinical trial findings. Further, a recent study showed that searching Embase has a modest impact on the results of systematic reviews [39] and most studies evaluating RCT publication did not search Embase [21,34]. Additionally, we systematically contacted the sponsor or investigator to check whether the study was published. Finally, we used the data recorded at ClinicalTrials.gov, but this information is not always accurate, and ClinicalTrials.gov added a database of summary results allowing for reporting results of observational studies according to the STROBE statement only in September 2008.

Conclusion
In conclusion, about 39 % of observational studies evaluating an intervention with a safety outcome and registered at ClinicalTrials.gov had their results published. Searching for unpublished data allowed for access to more than two-thirds of the study results and 93 % of the total target number of participants. Given the potential important benefit of requesting the registration of observational studies, this practice should be required by research regulations.

Availability of data and materials
The data will be made available on Dryad.

Additional file
Additional file 1: Appendix 1: Emails for principal investigators indexed in ClinicalTrials.gov. Appendix 2: Flowchart of email responses. Appendix 3: Publication status according to the date of registration (before the start of the study, between the start and the study completion, after the study completion). (DOCX 31 kb)

Competing interests
Financial competing interests: none of the authors have any competing interests. Non-financial competing interests: none of the authors have any competing interests.
Authors' contributions MB participated in the study concept and design, acquisition of data, interpretation of the data, and drafting of the manuscript. PR participated in the study concept and design, interpretation of the data, and the critical revision of the manuscript. GB performed the statistical analysis and participated in the interpretation of the data, and the critical revision of the manuscript. AD participated in the interpretation of the data, and the critical revision of the manuscript. RH participated in acquisition of data. IB participated in the study concept and design, acquisition of data, interpretation of the data, drafting of the manuscript and the critical revision of the manuscript. All authors read and approved the final manuscript.