Herbal remedy clinical trials in the media: a comparison with the coverage of conventional pharmaceuticals
© Bubela et al. 2008
Received: 27 October 2008
Accepted: 26 November 2008
Published: 26 November 2008
Skip to main content
© Bubela et al. 2008
Received: 27 October 2008
Accepted: 26 November 2008
Published: 26 November 2008
This study systematically compares newspaper coverage of clinical trials for herbal remedies, a popular type of complementary and alternative medicine, with clinical trials for pharmaceuticals using a comparative content analysis. This is a timely inquiry given the recognized importance of the popular press as a source of health information, the complex and significant role of complementary and alternative medicine in individual health-care decisions, and the trend toward evidence-based research for some complementary and alternative medical therapies. We searched PubMed for clinical trials, Lexis/Nexis for newspaper articles in the UK, US, Australia/New Zealand, and Factiva for Canadian newspaper articles from 1995 to 2005. We used a coding frame to analyze and compare 48 pharmaceutical and 57 herbal remedy clinical trials as well as 201 pharmaceutical and 352 herbal remedy newspaper articles.
Herbal remedy clinical trials had similar Jadad scores to pharmaceutical trials but were significantly smaller and of shorter duration. The trials were mostly studies from Western countries and published in high-ranking journals. The majority of pharmaceutical (64%) and herbal remedy (53%) clinical trials had private sector funding involvement. A minority declared further author conflicts of interest. Newspaper coverage of herbal remedy clinical trials was more negative than for pharmaceutical trials; a result only partly explained by the greater proportion of herbal remedy clinical trials reporting negative results (P = 0.0201; χ 2 = 7.8129; degrees of freedom = 2). Errors of omission were common in newspaper coverage, with little reporting of dose, sample size, location, and duration of the trial, methods, trial funding, and conflicts of interest. There was an under-reporting of risks, especially for herbal remedies.
Our finding of negative coverage of herbal remedy trials is contrary to the positive trends in most published research based primarily on anecdotal accounts. Our results highlight how media coverage is not providing the public with the information necessary to make informed decisions about medical treatments. Most concerning is the lack of disclosure of trial funding and conflicts of interest that could influence the outcome or reporting of trial results. This lack of reporting may impact the medical research community, which has the most to lose by way of public trust and respect.
Health care receives significant media attention, and complementary and alternative medicine (CAM) is no exception [1–3]. Given the continued public interest in CAM, this media attention is hardly surprising. Indeed, CAM is a multibillion dollar business . An increasingly empowered  and informed public continues to turn to CAM as an alternative or supplement to conventional medical therapies , even though the mechanisms of action are not always well understood, efficacy is often unsupported by research, and use has been associated with adverse reactions [7–9]. There are varied explanations for the appeal of CAM, such as belief in a particular philosophical orientation towards health and life, general dissatisfaction with conventional medicine, and belief that it is natural and therefore, less harmful than biomedical treatments .
In this paper we explore the nature and tone of newspaper coverage of clinical trials for herbal remedies, and compare it with coverage of clinical trials for pharmaceuticals used to treat the same medical conditions. Given the recognized importance of the popular press as a source of health information [11, 12], the complex and significant role of CAM in individual health-care decisions, and the trend toward evidence-based research for some CAM therapies [13–15], this seems a timely investigation. We examine, through a comparative content analysis, the quality of information provided by newspapers about one type of news story about CAM: clinical trials and their outcomes, recognizing that reporting the results of clinical trials does not represent the majority of news coverage on CAM. This type of study, comparing media coverage with the scientific research it covers is a well recognized method in media studies . Is the tone of reporting different for herbal remedy versus pharmaceutical clinical trials? Are there differences in the sources of trial funding and the reporting of that issue? What about the reporting of conflicts of interest? This latter question seems particularly important as real or perceived conflicts of interest may diminish public trust in medical research .
We first searched Lexis/Nexis for newspaper articles in the United Kingdom and Ireland (UK), the United States (US), and Australia/New Zealand and Factiva for newspaper articles in Canada using the search term '(herb or herbal) and remedy and "clinical trial"' from 1 January 1995 to 1 June 2005. This search strategy identified English language news articles that used both the term 'clinical trial' and 'herb/herbal'. We started with newspaper articles and not the clinical trials because few herbal remedy clinical trials receive media coverage. We retained all articles that discussed the results of an identifiable, published clinical trial. As many newspaper articles do not use the expression 'clinical trial' but instead use terms such as 'study', we also conducted specific searches on Lexis/Nexis and Factiva for additional newspaper articles on each clinical trial using different combinations of search terms, including journal name, author name(s), herbal remedy tested, and lead research institution. We omitted duplicate newspaper articles, selecting the article with highest word count, to take into account the effect of syndicated articles and newswires. We identified and collected the published clinical trials using Google searches and PubMed.
Two project-naïve, undergraduate student coders used a standardized coding frame, a common method for analyzing media coverage [18–21] for clinical trials (Appendix 1), to assess: journal name, year, type of institution where research was conducted, medical condition (using ICD-9-CM: International Classification of Diseases, 9th revision, Clinical Modification), dose, location of clinical trial, framed as controversy, benefits, risks, quality of the trial (Jadad score) , sample size, length of trial, conflicts of interest, trial funding, tone of clinical trial outcome, and overall tone of the discussion/conclusions, taking into account study limitations. For newspaper articles, the coders assessed the newspaper, country, year, word count, news format, themes, name of treatment, doses tested, use specified, main voice, location of clinical trial, framed as controversy, benefits, risks, conflicts of interest, funding of clinical trial, involvement of funding agency, how conflict of interest is viewed, tone of assessment of clinical trial outcomes, framing, accuracy, and, overall, whether the main claims in the newspaper article reflected the research findings.
The coders were first trained to use the coding frame by the lead author and then worked together under supervision for 2 weeks. If discrepancies arose, the coders reached a common interpretation and kept a log of decisions. The coders then individually coded the remaining articles (70% and 20%, respectively) with approximately 10% overlap to calculate inter-coder reliability, using Kappa scores. The coders reconvened periodically to discuss issues that had arisen during coding. The median of Kappa scores for clinical trials was 0.85 (range 0.39–1.00; N = 12) and for newspaper articles was 0.71 (range 0.47–1.00; N = 16), indicating good to excellent agreement. Independent variable Kappa scores were between 0.65 and 1.00.
One author (HB) with expertise in pharmacology identified corresponding pharmaceuticals for each medical condition identified in the coded herbal remedy clinical trials. We conducted searches on Lexis/Nexis and Factiva as above for newspaper coverage of identifiable clinical trials of the pharmaceuticals. As significantly more pharmaceutical trials received press coverage, we randomly selected up to three clinical trials per medical condition and then repeated the specific newspaper searches and coding methodology as above.
We performed a classification tree analysis, using CART Pro, version 6.0 to focus our analysis on the most important variables . The independent variables for clinical trials were 'overall tone' and the 'tone of clinical trial outcomes' and for newspaper articles were 'overall tone' and 'tone of assessment of the trial'. The classification tree used twoing for ranked data as the splitting method and determined which variables from the coding frame were most important in assigning a newspaper article or clinical trial to one of the three categories of tone (negative, neutral, positive). The most important variables for both measures of tone for clinical trials were length of trial, sample size, year, total number of benefits, total number of risks/costs, and Jadad score. The most important variables for both measures of tone for newspaper articles were word count, year, total number of benefits, and total number of risks/costs. We also selected additional variables of interest: for clinical trials, conflicts of interest and type of funding, and for newspaper articles, medical condition, conflicts of interest, and type of funding. National differences were not important and, therefore, the data from the different countries were combined.
Coding variables for clinical trials of herbal remedies and pharmaceuticals and associated newspaper articles
Coding variable: clinical trials
Herbal remedy clinical trial
(N = 57)
Pharmaceutical clinical trial
(N = 48)
Mean number of benefits
Mean number of risks
Mean sample size
P < 0.0001
Mean duration of trial (days)
P = 0.008
Mean Jadad score
P = 0.002
Trial described as randomized
P = 0.001
Trial described as double-blind
P < 0.001
Withdrawals and dropouts described
Conflicts of interest NOT specified
P = 0.003
Funding of trial NOT specified
P = 0.004
Coding variable: newspaper articles
(N = 352)
(N = 201)
Mean word count
Mean number of benefits
Mean number of risks
P < 0.0001
Source of funding NOT specified
Conflicts of interest NOT specified
At least one scientific/technical error in reporting on the trial
Duration of the clinical trial NOT specified
Sample size NOT specified
P < 0.0001
Dose NOT specified
P < 0.0001
Location of trial NOT specified
Randomization NOT specified
Double-blinding NOT specified
P < 0.0001
Use of placebo NOT specified
P < 0.0001
Withdrawals/dropouts NOT specified
Finally, we used logistic regression (multinomial for tone of assessment of the trial) adjusting for covariates that showed a significant difference using Wilcoxon tests to compare the measures of tone between herbal remedies and pharmaceuticals for clinical trials and newspaper articles. Results are presented as odds ratios and 95% confidence intervals where appropriate. All analyses were done in Stata version 7.0 (Stata Corporation, College Station, US).
Our methods have a number of limitations. First, we surveyed only the print media, while television and increasingly the Internet are likely important sources of information for the public. We did, however, survey a broad range of newspapers of varying quality. Second, our study was limited to the small subset of newspaper stories that were directly related to peer-reviewed clinical trials. However, this met our objective to assess whether media coverage of the evidence-based trend in CAM was qualitatively different from reporting on conventional medicine with a long-standing tradition of clinical trial research. Finally, our study is a content analysis using an a priori coding frame [18–21], not a more detailed qualitative textual analysis. This enabled us to survey a larger number of clinical trials and newspaper articles and to test for statistically significant differences in reporting trends.
We coded 105 clinical trials (48 pharmaceutical, 57 herbal remedy) and 553 associated newspaper articles (201 pharmaceutical, 352 herbal remedy). The main comparisons are for tone (positive, neutral, negative), quality measures, and content between the herbal remedy and pharmaceutical clinical trials. There are four measures of tone: the tone of the results/outcomes of the clinical trial; the overall tone of the clinical trial, taking into account limitations and the discussion; the tone of the assessment of the specific clinical trial in the newspaper article; and the overall tone of the entire newspaper article.
The clinical trials were published in 39 journals, primarily JAMA, the journal of the American Medical Association (21), the Lancet (14), New England Journal of Medicine (13), and BMJ (9). JAMA (10) and BMJ (8) published the greatest number of herbal remedy clinical trials reported in newspapers. Only one herbal remedy clinical trial was published in a CAM-specific journal, the Journal of Alternative and Complementary Medicine, confirming the trend for newspapers to report research in mainstream, high-impact medical journals. The majority of trials took place in North America, the UK and Europe, and the majority was conducted in public sector research institutions with only two pharmaceutical and four herbal remedy clinical trials taking place in the private sector.
We coded articles from 131 newspapers in five countries. Newspapers with more than ten articles on herbal remedy clinical trials were the Daily Mail (UK), the Kitchener-Waterloo Record (Canada), the New York Times (US), the Washington Post (US), Pittsburgh Post-Gazette (US), and The Record (US). The majority of newspaper articles were published in the US (296), followed by the UK (151), Canada (100), and Australia/New Zealand (6). Most newspaper articles were reporting on the latest news (60%) and one third had a longer, investigative reporting format where the clinical trial was only one part of the article. The predominance of straight news reporting was reflected in whether the article was framed as a controversy: 43% of articles had no controversial element, 35% were framed as a balanced controversy, and 22% as an imbalanced controversy.
The main theme of almost all the articles on pharmaceutical clinical trials was the trial itself. This contrasted with articles on herbal remedy clinical trials where 63.6% focused on the trial and the other third focused on the medical condition, the myriad of uses for an herb, and the health risks associated with herbal remedies. The main benefit cited in almost all articles (90%) was improved health or treatment options, and only 4% cited no benefits. However, 54% of articles did not quantify the likelihood of the benefit. Interestingly, 29% of articles on herbal remedy versus 4% of pharmaceutical clinical trials stated there was no benefit. The main individual interviewed or quoted in the articles for both pharmaceutical clinical trials (63%) and herbal remedy clinical (73%) trials was a university or hospital scientist or physician not specializing in CAM. Indeed CAM researchers or practitioners were cited in only 8% of articles on herbal remedy clinical trials.
There were strong associations in newspaper articles between the tone of valuation of the results of the clinical trial and the type of trial (P < 0.0001; χ 2 = 135.5; DF = 2) and the overall tone of the newspaper article and the type of trial (P < 0.0001; χ 2 = 56.1; DF = 2). Newspaper articles were more highly polarized with respect to evaluating the outcomes of herbal remedy clinical trials than for pharmaceutical clinical trials. For pharmaceutical trials, the evaluation was most likely to be positive or neutral (Figure 2), while a greater proportion of newspaper articles evaluated the outcomes of herbal remedy clinical trials negatively.
The difference in overall tone of the newspaper article, where the clinical trial may have been only one part of a larger story was even more pronounced. The overall tone of the newspaper article was most likely to be positive (68.2%) or neutral (31.8%) when reporting on pharmaceuticals. Indeed no articles on pharmaceutical clinical trials had a negative overall tone. In contrast, 21.9% of articles on herbal remedy clinical trials had an overall negative tone, while the majority were neutral (57.1%), and 21% were positive (Figure 2). This is so, even though the articles reported significantly more risks, primarily adverse effects, associated with pharmaceuticals than herbal remedies (Figure 2). However, 40% of articles overall did not specify a risk, and 53% of those which did specify a risk did not quantify the likelihood of that risk. There were no significant associations between type of trial and conflicts of interest and type of funding of the trial; 83% of articles did not mention the source of funding for the trial and 96% did not specify the involvement of the funder. Similarly, 96% did not report on conflicts of interest. However, of the 29 articles that did report on conflicts of interest, 38% viewed the conflict negatively while 48% were neutral in their evaluation of the conflict.
The significant associations for overall tone remained when we used logistic regression and adjusted for significant covariates, year, and total number of risks. We calculated significance of overall tone for positive and neutral valuation categories only because one cell (negative valuation of pharmaceutical clinical trials) had zero counts (odds ratio = 13.210; 95% Wald confidence limits 7.414–23.539; P < 0.0001). We present the probabilities for tone of assessment of clinical trial outcomes adjusted for significant covariates because of the difficulties in interpreting multinomial logistic regression outputs (type of clinical trial: P < 0.0001; χ 2 = 41.26; DF = 2; year: P < 0.0001; χ 2 = 85.84; DF = 182; total risks: P < 0.0001; χ 2 = 19.19; DF = 2).
For all newspaper articles, there is a significant association between the type of medical condition and both of the measures of tone (overall: P < 0.0001, χ 2 = 9.5, DF = 12; tone of assessment of the trial: P < 0.0001, χ 2 = 41.7, DF = 12). The largest categories of medical conditions were diseases of the genitourinary system (mainly erectile dysfunction and menopause), a range of diseases affecting organ systems and mental disorders (mainly depression). Proportionally, the most negative overall tone was reserved for disorders associated with diet and obesity. The tone of assessment of the trial itself was proportionally most negative for diseases of the genitourinary system (mainly erectile dysfunction and menopause), mental disorders (mainly depression), and infectious diseases (mainly HIV/AIDS).
As CAM use increases alongside concerns about evidence of efficacy, we are interested in how the media, a significant source of medical information for the lay public and some professionals, report on clinical trials. This is the first study to compare systematically newspaper coverage of clinical trials for herbal remedies, a popular type of CAM, with clinical trials for pharmaceuticals. There has been a large increase in the number of clinical trials for herbal remedies over the last 20 years , the vast majority published in specialized journals on traditional Chinese medicines . There has not, however, been a concomitant increase in the number of media articles reporting on clinical trials for herbal remedies; overall the media is not reporting on the trend toward evidence-based herbal medicine .
Those herbal remedy clinical trials that receive newspaper coverage are of similar quality to pharmaceutical clinical trials, likely reflecting the media's preference towards studies conducted in Western countries , and a select number of high-ranking journals for most coverage of medical research . Our Jadad scores are marginally higher than those calculated for a greater variety of CAM trials published in a broader range of journals . Another study comparing herbal remedy and pharmaceutical clinical trials conducted in Western countries also found no difference in quality of described methods . Assessing reported methodology within clinical trials, however, may be an incomplete proxy for quality since methodology is often significantly abbreviated or published in accompanying longer articles . In that case, sample size and duration of the study could be a more precise measure of trial quality , in which case, pharmaceutical clinical trials are significantly larger; many herbal remedy clinical trials had fewer than 100 patients and were of short duration.
Our results suggest high-impact journals publish a significant number of CAM studies with negative results [26, 27], even though the general trend toward favoring positive results applies to both types of clinical trial . Overall, however, the tone of both the positive and negative clinical trials is tempered in the discussion by highlighting, for example, trial limitations or conversely, potential for future research.
Despite the overall positive results and tone of the clinical trials, newspaper coverage of herbal remedy clinical trials was more negative than for pharmaceutical clinical trials. This is contrary to most published research on media coverage of CAM. Those studies consider a much broader spectrum of treatments and the media content is generally anecdotal rather than evidence based . Indeed, journalists are displaying a degree of skepticism rare for medical reporting . It is possible that once confronted with actual evidence, journalists are more critical or skeptical. It may be considered more newsworthy to debunk commonly held beliefs and practices related to CAM, to go against the trend of positive reporting in light of evidence. It is also possible that journalists who turn to press releases of peer-reviewed, high-impact journals have subtle biases towards scientific method and conventional medicine. Also, journalists turn to trusted sources in the biomedical community for comments on clinical trials, both herbal and pharmaceutical, potentially leading to a biomedical bias in reporting trial outcomes. Finally, the minority of newspaper articles were written by named journalists. Since it was not possible to identify the authors of the majority of newspaper stories, it is possible that some journalists with either a positive or negative bias contributed disproportionately to the news coverage.
Despite repeated studies, the media continues to provide insufficient information to the public largely through omission, an under-reporting of risk, and a lack of disclosure of trial funding and potential conflicts of interest. The latter is true for both pharmaceuticals and herbal remedies but may be more critical for herbals, which may be accessed by the lay public without a physician intermediary.
Media coverage is not providing the public with the information necessary to make informed decisions about medical treatments, either conventional or alternative. There are significant errors of omission of basic information such as dose, sample size, location and duration of the trial, and methods for randomized clinical trials. In addition, there is an under-reporting of risks, especially in the context of herbal remedies. This is common in the context of medical reporting where difficulties stem from a lack of comprehension, both by the public and journalists, of differences between absolute and relative risks and the nature of probabilistic analyses . Many studies have reported on such errors of omission [18–20], citing space and time constraints of medical journalists whose stories compete for publication and the attention of editors with general news and other content [19, 31].
The media is also overly reliant on narratives from satisfied patients, researchers, clinicians, and patient groups without disclosing financial ties to industry and conflicts of interest [20, 21, 32]. 'Conflicts occur when scientists are expected to exercise judgment dispassionately but instead are motivated by financial, professional, or other types of interest' . There has been an international movement toward reporting of funding and conflicts of interest in published clinical trials, yet a significant portion of herbal remedy clinical trials in our study did not disclose any funding source. Those that did showed substantial funding from the private sector, not surprising considering the current economics of CAM and interest in CAM by pharmaceutical companies. Even more concerning was the lack of disclosure of conflicts of interest, either positive or negative, for both types of clinical trials. This suggests the need for better implementation of funding and conflicts of interest disclosure policies by medical journals .
The media should be encouraged to disclose the interests of researchers and institutions involved in clinical trials that could influence the outcome or reporting of trial results . Without such information, which our study shows is largely missing from media coverage, it is not possible for the lay public to assess the credibility of the research. This lack of reporting should be of great concern to the medical research community, which has the most to lose by way of public trust and respect. Academic researchers and institutions have the most to gain from the development and enforcement of full disclosure guidelines for both medical publications and for the media.
Overall concerns have lead to suggestions for minimum standards and content for medical coverage . Gary Schwitzer and colleagues  have argued ' [j]ournalists have a special responsibility in covering health and medical news' knowing that 'readers and viewers may make important health-care decisions based on the information provided [in media stories]'. In response, solutions must be aimed at all three major stakeholders and involve guidelines for journalists  and increased training in health and science journalism , increased training in media communication skills for clinical researchers [36–38], and scientific and media literacy programs for the lay public . Given the well-established and expanding market for health coverage, it is time for journalists and editors to experiment with improving content without necessarily sacrificing narrative themes such as human-interest stories. A change for the better is unlikely to result in a reduced public appetite for health news – an appetite that is increasingly sophisticated and desirous of high quality information .
complementary and alternative medicine
degrees of freedom.
The study was funded by the Advanced Food and Materials Network (Canada) to HB and TC and a Centre for Integrated Healthcare Research New Investigator Award (MSH 69136) to HB. The study authors are independent of the funders. The funding sources had no involvement in the study. We thank Megan Koper and Thomas Moran for research assistance and coding, Nina Hawkins for administrative support, David Hik for comments on the manuscript, Alethea Adair for editing, Terry Elrod for statistical advice, and Rahim Moineddin, consulting biometrician and Assistant Professor, Department of Family and Community Medicine, Faculty of Medicine, University of Toronto for statistical analyses.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.