Research article | Open | Open Peer Review | Published:
Comparison of serious adverse events posted at ClinicalTrials.gov and published in corresponding journal articles
BMC Medicinevolume 13, Article number: 189 (2015)
The reporting of serious adverse events (SAEs) in clinical trials is crucial to assess the balance between benefits and risks. For trials with serious adverse events posted at ClinicalTrials.gov, we assessed the consistency between SAEs posted at ClinicalTrials.gov and those published in corresponding journal articles.
All records from ClinicalTrials.gov up to February 2014 were automatically exported in XML format. Among these, we identified all phase III or IV randomized controlled trials with at least one SAE posted. For a random sample of 300 of these trials, we searched for corresponding publications using MEDLINE via PubMed and extracted safety results from the articles.
Among the sample of 300 trials with SAEs posted at ClinicalTrials.gov, 78 (26 %) did not have a corresponding publication, and 20 (7 %) had a publication that did not match the ClinicalTrials.gov record. For the 202 remaining trials, 26 published articles (13 %) did not mention SAEs, 4 (2 %) reported no SAEs, and 33 (16 %) did not report the total number of SAEs per treatment group. Among the remaining 139 trials, for 44 (32 %), the number of SAEs per group published did not match those posted at ClinicalTrials.gov. For 31 trials, the number of SAEs was greater at ClinicalTrials.gov than in the published article, with a difference ≥30 % for at least one group for 21. Only 33 trials (11 %) had a publication reporting matching numbers of SAE and describing the type of SAE.
Many trials with SAEs posted at ClinicalTrials.gov are not yet published, omit the reporting of these SAEs in corresponding publications, or report a discrepant number of SAEs as compared with ClinicalTrials.gov. These results underline the need to consult ClinicalTrials.gov for more information on serious harms.
Randomized controlled trials (RCTs) are considered the gold standard for assessing the effects of health care interventions and form the basis for treatment decision-making. Thus, all results from clinical trials should be transparent and accessible to all [1–3]. Results should report on efficacy and also on safety to provide an estimation of the balance of benefits and risks [4, 5]. In particular, serious adverse events (SAEs), defined as adverse events that result in death, require inpatient hospitalization or the prolongation of hospitalization, are life-threatening, or result in persistent or significant disability or incapacity or a congenital anomaly or birth defect , should always be completely reported . However, inadequate and underreporting of trial results, especially safety results, is common [4, 5, 8, 9], which leads to biased evidence that can have serious consequences for patients. A notable example is the obscured reporting of cardiovascular risk with rofecoxib in the VIGOR study, in which naproxen, the control treatment, was presented as having a protective effect [10–12].
To increase transparency, the 2007 US Food and Drug Administration Amendments Act (FDAAA 801) required that, after September 2008, results from clinical trials conducted in the United States be made publicly available at ClinicalTrials.gov within 1 year of the completion of the trial [13–15]. In addition, after September 2009, the posting of adverse events was also mandatory . All SAEs as well as other non-SAEs above a specified frequency threshold must be reported per group in addition to the total number of patients at risk. In a recent study focusing on reporting, we found that trial results, especially SAEs, were more completely reported at ClinicalTrials.gov than in corresponding published articles .
In this study, we examined the consistency between SAEs posted at ClinicalTrials.gov and those published in journals. We identified a random sample of trials with SAEs posted at ClinicalTrials.gov to assess whether these safety results were published and to compare the timing of availability of SAEs between ClinicalTrials.gov and publications as well as the number and type of SAEs posted at ClinicalTrials.gov and those published in corresponding journal articles.
Search for trials with SAEs posted at ClinicalTrials.gov
On February 2, 2014, we exported all records from ClinicalTrials.gov and, using R 3.1.1  with the XML package, we identified all completed phase III or IV RCTs with at least one SAE posted at ClinicalTrials.gov. We excluded trials with only one group, trials with ≥4 groups, and phase I, I/II, II, and II/III trials. Of all eligible trials (n = 1580), we selected a random convenience sample of 300 trials to search for corresponding publications in journals.
Search for corresponding publications in journals
In June 2014, one of the authors (ET) searched for corresponding publications in journals using the link to publications provided at ClinicalTrials.gov whenever possible. Also, MEDLINE was systematically searched via PubMed by using the ClinicalTrials.gov identification number (NCT number). If no publication was identified, MEDLINE was searched again by using keywords for drug names and condition studied. The articles identified by the search had to match the corresponding trial in terms of the information registered at ClinicalTrials.gov (same objective, sample size, location, responsible party, trial phase, and funding sponsor). If there were several publications for a trial, all publications that matched the time-frame indicated at ClinicalTrials.gov were selected. Trials for which the published article reported a different time-frame or number of groups were excluded.
A second reviewer (CR) checked the matching between ClinicalTrials.gov and the corresponding published articles for all trials. All disagreements were resolved by discussion between the two reviewers with the help of a third reviewer (AD) if needed.
For the random sample of 300 trials with SAEs posted at ClinicalTrials.gov, we extracted the following characteristics from the records exported from ClinicalTrials.gov by using R 3.1.1 :
General characteristics of the trial: lead sponsor, condition, and countries where the trial was conducted; primary completion date (defined as the date of the final collection of data for the primary outcome) and the date when results were first posted. This date was extracted from the archive record and differs from the date on which results were first received, which is available under Study Results at ClinicalTrials.gov. The difference between these two dates relates to ClinicalTrials.gov production and the vetting of the results by the US National Institutes of Health.
Design of the trial: phase III or IV trial and parallel or cross-over trial.
Interventions: details concerning the interventions for the experimental and control groups.
SAEs: total number of SAEs, number of SAEs per group, types of SAEs, number of SAEs per type per group, and number of participants at risk per group. We recorded the date when SAEs were first posted from the archive record.
For all trials with corresponding publications, the following information was collected from published articles, including online supplements:
General characteristics of the publication: journal of publication, first author, date of online publication, type of journal (general medical, specialty), and whether ClinicalTrials.gov NCT number was reported in the published article.
SAEs: whether SAEs were reported and whether they were reported by number per group. If the total number of SAEs per group was not reported, we noted whether the number of the most common SAEs (those observed above a certain frequency or threshold rate), number of SAEs related to treatment, or toxicity-graded events were reported instead. We also extracted the types of SAEs reported and noted whether the number per type and per group was reported. Again, if all types were not reported, we noted whether the types reported were common event types, those observed above a certain frequency or threshold rate, those related to treatment, or toxicity-graded events. We also collected the number of participants at risk for each group.
If several publications were identified for the same trial, we extracted safety results from all corresponding publications having the same time-frame as reported at ClinicalTrials.gov. For trials with multiple phases (e.g. lead-in or induction, double-blind randomized treatment, and follow-up or extension) reported at both ClinicalTrials.gov and in published articles, we extracted SAE data only for the double-blind randomized treatment period. For published articles reporting pooled results from multiple trials, we considered the data to be missing if we were unable to extract SAE data that corresponded to the specific trial.
All data were extracted from the published article independently by two reviewers (ET, CR), independently of data collected from ClinicalTrials.gov. One of the reviewers (CR) was blinded to the hypothesis. All disagreements were resolved by discussion to reach a consensus, including intervention by a third reviewer (AD) in case of discrepancies.
Descriptive data are reported with numbers (percentages) and median (quartile 1–3 [Q1–3]). We compared time between primary completion date as reported at ClinicalTrials.gov and date of the SAEs first publicly posted at ClinicalTrials.gov or the date of the first online publication in journals reporting the number of SAEs per group by the Kaplan–Meier method. Trials for which the number of SAEs per group were not reported in a published article on June 2014 were censored at this date for the estimation of time between the primary completion date and online publication of SAEs. For trials with both results posted and published, we compared the number of SAEs reported at ClinicalTrials.gov and in the published article. To simplify, for three-group trials with two groups representing different doses of the same treatment, we combined these two groups. For trials comparing three different treatments, we selected the main comparison according to the ClinicalTrials.gov posting and without knowledge of the results. Analyses involved the use of R 3.1.1 .
Figure 1 describes the selection of trials. Briefly, from the 159,679 studies recorded at ClinicalTrials.gov on February 2, 2014, there were 1,580 phase III or IV randomized drug trials with two or three groups having results posted. We selected a random sample of 300 trials to search for corresponding publications. Among the 300 trials, 78 (26 %) had no corresponding published article. From the remaining 222 trials with results both posted and published, we excluded 20 trials for which the time-frame and/or number of groups reported in the article did not match those posted at ClinicalTrials.gov. Table 1 describes the characteristics of the 300 randomly selected trials and the 202 trials with corresponding published articles.
Time to availability of SAEs at ClinicalTrials.gov and in published articles
The difference between the median time to the availability of SAEs at ClinicalTrials.gov and in published articles was 50 months (95 % confidence interval [95 % CI]: 26–98 months) with the median time between primary completion of trials and SAEs publicly posted at ClinicalTrials.gov of 22 months (Q1–3, 15–35) and median time between primary completion date and availability of SAEs in published articles of 72 months (Q1–3, 27–119; Fig. 2).
Reporting of the number of SAEs per treatment group
From the 202 final pairs of trials, the number of SAEs per group was reported in the published article for 139 (69 %); 26 (13 %) published articles did not mention SAEs at all and 4 (2 %) reported no SAEs. Of these 30 articles, for 12 trials, the number of SAEs posted at ClinicalTrials.gov was >10 and for 6, >100. For 33 trials, the reporting of SAEs was incomplete in the published article: 10 reports (5 %) described only drug-related SAEs or SAEs of interest and 9 (4 %) reported adverse events with grade ≥3 instead. In 8 articles, the results were pooled from several trials and we were not able to extract the number of SAEs for each individual trial. Three published articles (1 %) did not report the number of SAEs per group, and 3 (1 %) reported only SAEs for the experimental group (Fig. 3).
Among the 139 trials that reported SAEs per group in the published article, for 44 (32 %), the number of SAEs per group did not match that posted at ClinicalTrials.gov; for 32 of these, the number of SAEs was different for both groups and, for 12, only one group. For the 32 trials with number of SAEs different for both groups, 22 (69 %) had more SAEs reported at ClinicalTrials.gov than in the published articles. For 15 of these trials (68 %), the difference between the number posted at ClinicalTrials.gov and the published article was ≥30 % for at least one group. For the 12 trials where number of SAEs was different for one group only, 9 trials had more SAEs reported at ClinicalTrials.gov than in the published article. For 6 of these trials, the difference between the number posted at ClinicalTrials.gov and the published article was ≥30 %.
Reporting of details of SAEs
Of the 95 trials with number of SAEs matching that at ClinicalTrials.gov, all types of SAEs that occurred were reported for 33 (35 %) in the corresponding published articles. In 34 published articles (36 %), the type of SAE was not reported at all. Other publications reported the types of SAEs for only drug-related SAEs (n = 12), the most common SAEs (n = 10), the SAEs of interest (n = 5), or SAEs leading to withdrawal (n = 1).
Overall, only 33 trials (11 %) out of the random sample of 300 trials with SAEs posted at ClinicalTrials.gov had a publication reporting matching SAE numbers and describing the type of SAE (Fig. 3). The characteristics of these 33 trials are presented in Table 2. In brief, 88 % of these trials were phase III trials, 82 % had a private funding source, 73 % had at least one site in the United States, and 73 % were published in a specialty journal.
Herein, we identified a random sample of trials with SAEs posted at ClinicalTrials.gov to assess whether these safety results were reported in published articles and, if yes, whether there were discrepancies between the publication and the registry data. Our results highlight that the reporting of SAEs in published articles remains a major problem. For a sample of 300 trials with SAEs posted at ClinicalTrials.gov, among 202 with a matching publication, 30 (15 %) did not mention SAEs or reported no SAEs in the corresponding publications. The number of SAEs per group was frequently not reported in the published articles and when it was reported, discrepancies with the numbers posted at ClinicalTrials.gov were common, with frequently more SAEs reported at ClinicalTrials.gov than in the published article.
Restricted space in articles is a frequently cited reason for incomplete reporting of harms [4, 18]. However, the assessment of the balance between benefits and risks should be the core of trial reports. Failure to report SAEs may lead to a biased safety profile and erroneous decision-making, with major consequences for patients. Despite the extension of the Consolidated Standards of Reporting Trials (CONSORT) statement published in 2004, which provides guidelines on reporting harms-related data , reporting of safety data in published articles of clinical trials continues to be suboptimal [5, 16, 19, 20], with poor adherence to the statement [21–24]. According to a recent study, only 63 % of published articles reported the total number of SAEs by group .
In a previous article focusing on completeness of reporting, we found that SAEs were significantly more completely reported at ClinicalTrials.gov than in the published articles (99 % vs. 63 %, P <0.0001) . This result was particularly troubling, but one explanation could be that SAEs were not reported in published articles because there were none.
Our results identified some trials not mentioning SAEs or reporting no SAEs in the published article, despite these being reported at ClinicalTrials.gov. Furthermore, when SAEs were reported in published articles, discrepancies with the number posted at ClinicalTrials.gov were common, with frequently more SAEs reported at ClinicalTrials.gov than in the published article. Although we do not know which the ‘true’ results are, we believe that these discrepancies clearly outline problems in the reporting of SAEs. Two studies comparing results posted at ClinicalTrials.gov and in peer-reviewed publications also showed discrepancies in the number of SAEs [18, 25, 26]. The originality of our approach was the identification of trials for which we had knowledge of SAEs to assess whether and how these safety results were reported in published articles.
Our results have important implications. Our results highlight that ClinicalTrials.gov provide more information on serious harms, whereas these events are frequently underreported in published articles. For systematic reviewers, they outline the interest of using ClinicalTrials.gov to find safety results not yet published in journals and for trials with both SAEs posted and published, to compare the rate of SAEs. In case of discrepancies, we recommend systematically contacting authors for clarification and performing sensitivity analyses in case of non-response to assess to what extent these discrepancies may affect the meta-analysis result. For journals, they question the peer-review process, in that the assessment of data recorded in registries including results and harms when available should be part of the process to assess if there are any discrepancies that could bias the results. In case of discrepancies, investigators should be contacted for clarification. They also raise questions about how reporting guidelines, especially the CONSORT harms, are implemented by journals, with a need for more active endorsement. Templates with mandatory reporting of critical elements, such as that used at ClinicalTrials.gov , could improve the reporting of safety results in journals. For policymakers, our results advocate an extension to all countries of the mandatory posting of trial results. Besides their use for limiting publication bias and selective outcome reporting, public registries may help improve transparency of results in clinical trials. Accordingly, in April 2014, the European Union voted to adopt the Clinical Trials Regulation, which requires the registration of all clinical trials conducted in Europe and posting of trial summary results in the European Clinical trials Database (EudraCT) within 1 year after trial completion [27, 28]. Nevertheless, compliance to the legal requirement in the United States is low [16, 29–33] despite civil monetary penalties (up to $10,000 a day) and, for federally funded studies, the withholding of grant funds in cases of non-compliance . Therefore, compliance must be improved. A recent article showed that sending emails to responsible parties of completed trials that do not comply with the FDAAA legal requirement to post results significantly improved the posting rate at 6 months .
We may not have identified all published articles because we searched only MEDLINE for publications. Further, for trials without publications, the results could be published at a future date because publication in journals may take time due to multiple submissions. Some trials may have multiple publications with different results reported. In this case, we did not include all reports resulting from the trial but only the reports that included safety data and matched the time frame reported at ClinicalTrials.gov. Finally, this study focused on trials assessing pharmacological treatments, but non-pharmacological treatments can also incur SAEs.
Our results reveal that many trials with SAEs posted at ClinicalTrials.gov are not yet published, omit the reporting of these SAE in corresponding publications, or report a discrepant number of SAEs as compared with ClinicalTrials.gov. Consulting safety results posted at ClinicalTrials.gov, when available, is crucial for more information on serious harms.
Consolidated Standards of Reporting Trials
Food and Drug Administration Amendments Act
Randomized controlled trial
Serious adverse event
Antes G, Chalmers I. Under-reporting of clinical trials is unethical. Lancet. 2003;361:978–9.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol. 2009;114:1341–5.
Chalmers I. Underreporting research is scientific misconduct. JAMA. 1990;263:1405–8.
Ioannidis JA, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001;285:437–43.
Pitrou I, Boutron I, Ahmad N, Ravaud P. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med. 2009;169:1756–61.
ClinicalTrials.gov “Basic Results” Data Element Definitions. 2013; http://prsinfo.clinicaltrials.gov/results_definitions.html#AdverseEventsDefinition. Accessed March 2015.
Ioannidis JP, Evans SJ, Gotzsche PC, O’Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141:781–8.
Ioannidis JPA, Contopoulos-Ioannidis DG. Reporting of safety data from randomised trials. Lancet. 1998;352:1752–3.
Scharf O, Colevas AD. Adverse event reporting in publications compared with sponsor database for cancer clinical trials. J Clin Oncol. 2006;24:3933–8.
Curfman GD, Morrissey S, Drazen JM. Expression of concern: Bombardier et al., “Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis”, N Engl J Med 2000;343:1520–8. N Engl J Med. 2005;353:2813–4.
Curfman GD, Morrissey S, Drazen JM. Expression of concern reaffirmed. N Engl J Med. 2006;354:1193.
Krumholz HM, Ross JS, Presler AH, Egilman DS. What have we learnt from Vioxx? BMJ. 2007;334:120–3.
Dickersin K, Rennie D. The evolution of trial registries and their use to assess the clinical trial enterprise. JAMA. 2012;307:1861–4.
Tse T, Williams RJ, Zarin DA. Reporting “basic results” in ClinicalTrials.gov. Chest. 2009;136:295–303.
Tse T, Williams RJ, Zarin DA. Update on registration of clinical trials in ClinicalTrials.gov. Chest. 2009;136:304–5.
Riveros C, Dechartres A, Perrodeau E, Haneef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med. 2013;10:e1001566. Discussion e1001566.
The R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2014.
Hartung DM, Zarin DA, Guise JM, McDonagh M, Paynter R, Helfand M. Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications. Ann Intern Med. 2014;160:477–83.
Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501.
Zorzela L, Golder S, Liu Y, Pilkington K, Hartling L, Joffe A, et al. Quality of reporting in systematic reviews of adverse events: systematic review. BMJ. 2014;348:f7668.
Peron J, Maillet D, Gan HK, Chen EX, You B. Adherence to CONSORT adverse event reporting guidelines in randomized clinical trials evaluating systemic cancer therapy: a systematic review. J Clin Oncol. 2013;31:3957–63.
Sivendran S, Latif A, McBride RB, Stensland KD, Wisnivesky J, Haines L, et al. Adverse event reporting in cancer clinical trial publications. J Clin Oncol. 2014;32:83–9.
Hodkinson A, Kirkham JJ, Tudur-Smith C, Gamble C. Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension. BMJ Open. 2013;3, e003436.
Smith SM, Chang RD, Pereira A, Shah N, Gilron I, Katz NP, et al. Adherence to CONSORT harms-reporting recommendations in publications of recent analgesic clinical trials: an ACTTION systematic review. Pain. 2012;153:2415–21.
Becker JE, Krumholz HM, Ben-Josef G, Ross JS. Reporting of results in ClinicalTrials.gov and high-impact journals. JAMA. 2014;311:1063–5.
Becker JE, Ross JS. Reporting discrepancies between the clinicaltrials.gov results database and peer-reviewed publications. Ann Intern Med. 2014;161:760.
Europe votes for clinical trial transparency. http://www.alltrials.net/news/europe-votes-for-clinical-trial-transparency. Accessed March 2015.
Posting of clinical trial summary results in European Clinical Trials Database (EudraCT) to become mandatory for sponsors as of 21 July 2014. http://www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/news/2014/06/news_detail_002127.jsp&mid=WC0b01ac058004d5c1. Accessed March 2015.
Anderson ML, Chiswell K, Peterson ED, Tasneem A, Topping J, Califf RM. Compliance with results reporting at ClinicalTrials.gov. N Engl J Med. 2015;372:1031–9.
Gill CJ. How often do US-based human subjects research studies register on time, and how often do they post their results? A statistical analysis of the Clinicaltrials.gov database. BMJ Open. 2012;2. doi:10.1136/bmjopen-2012-001186.
Nguyen TA, Dechartres A, Belgherbi S, Ravaud P. Public availability of results of trials assessing cancer drugs in the United States. J Clin Oncol. 2013;31:2998–3003.
Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ. 2012;344:d7373.
Saito H, Gill CJ. How frequently do the results from completed US clinical trials enter the public domain? A statistical analysis of the ClinicalTrials.gov database. PLoS One. 2014;9, e101826.
Maruani A, Boutron I, Baron G, Ravaud P. Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial. BMJ. 2014;349:g5579.
We thank Ignacio Atal for help with XML files and Elise Diard for help with figures. Philippe Ravaud is director of the French EQUATOR Centre and a member of the EQUATOR Network Steering Group.
Sources of funding
The researchers did not received external sources of funding.
The authors declare that they have no competing interests.
ET performed the literature search, selected the studies, extracted data, interpreted results, and wrote the article. PR designed the study, interpreted results, and wrote the article. CR selected the studies, extracted data, and critically reviewed the manuscript. EP performed the statistical analysis and critically reviewed the manuscript. AD generated the idea for the study, designed the study, performed consensus on selection of studies and data extraction, and wrote the manuscript. All authors read and approved the final manuscript.