What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals?

Background We conducted this analysis to determine i) which journals publish high-quality, clinically relevant studies in internal medicine, general/family practice, general practice nursing, and mental health; and ii) the proportion of clinically relevant articles in each journal. Methods We performed an analytic survey of a hand search of 170 general medicine, general healthcare, and specialty journals for 2000. Research staff assessed individual articles by using explicit criteria for scientific merit for healthcare application. Practitioners assessed the clinical importance of these articles. Outcome measures were the number of high-quality, clinically relevant studies published in the 170 journal titles and how many of these were published in each of four discipline-specific, secondary "evidence-based" journals (ACP Journal Club for internal medicine and its subspecialties; Evidence-Based Medicine for general/family practice; Evidence-Based Nursing for general practice nursing; and Evidence-Based Mental Health for all aspects of mental health). Original studies and review articles were classified for purpose: therapy and prevention, screening and diagnosis, prognosis, etiology and harm, economics and cost, clinical prediction guides, and qualitative studies. Results We evaluated 60,352 articles from 170 journal titles. The pass criteria of high-quality methods and clinically relevant material were met by 3059 original articles and 1073 review articles. For ACP Journal Club (internal medicine), four titles supplied 56.5% of the articles and 27 titles supplied the other 43.5%. For Evidence-Based Medicine (general/family practice), five titles supplied 50.7% of the articles and 40 titles supplied the remaining 49.3%. For Evidence-Based Nursing (general practice nursing), seven titles supplied 51.0% of the articles and 34 additional titles supplied 49.0%. For Evidence-Based Mental Health (mental health), nine titles supplied 53.2% of the articles and 34 additional titles supplied 46.8%. For the disciplines of internal medicine, general/family practice, and mental health (but not general practice nursing), the number of clinically important articles was correlated withScience Citation Index (SCI) Impact Factors. Conclusions Although many clinical journals publish high-quality, clinically relevant and important original studies and systematic reviews, the articles for each discipline studied were concentrated in a small subset of journals. This subset varied according to healthcare discipline; however, many of the important articles for all disciplines in this study were published in broad-based healthcare journals rather than subspecialty or discipline-specific journals.


Background
Evidence on the journal-reading habits of clinicians comes from three separate groups of publications. First, several surveys have been used to ascertain the reading habits of physicians. Fafard and Snell [1] assessed house staff who reported reading an average of 8.7 hours per week, with about half of their time spent reading for specific patient situations. Reading time for family practice residents was more than three hours per week [2,3] and ranged from 1-12 hours. Dermatology residents averaged 4.2 hours reading per week and read an average of seven journals, four of which were peer reviewed [4]. Internists read an average of 4.4 hours per week [3], while surgeons reported an average reading time of 3.5 hours across 3-16 journals [5]. This average of three-four hours of reading time per week is quite consistent across disciplines, level of education, time, and nationality.
A second set of surveys and studies center on general information-seeking behaviors of clinicians. These studies show how journal reading fits in with the other types of information that clinicians use. Two systematic reviews have been done recently.
Researchers at the Australian National Institute of Clinical Studies [6] reviewed preferred information sources in many clinician groups, including physicians (primary care/general practice/family practice, hospitalists, rural physicians, diabetologists); nurses (hospital and occupational health nurses); physical therapists; dental hygienists; and policy-makers. They reviewed 34 studies and concluded that all groups used multiple information resources, with information needs answered most often by other people, followed by books and journals. Dawes and Sampson [7] evaluated 19 studies of physician information-seeking behavior. They placed books and journals in one category (print resources) and found this to be the most used information source, with colleagues being the second.
The third source of information on clinicians' use of information resources comes from marketing studies. The Association of Medical Publications [8] monitors physician use of printed journals and other information resources. Despite the rapid expansion of the Internet and all the information it contains, physicians continue to read and value journal articles, and their reliance on journals may be increasing. Data collected in 1983 and1998 shows that physician reliance on journal literature as their main source of medical information increased from 61.8% to 76.3%, an absolute increase of 14.5% in 15 years.
The importance of reading journal articles for clinical care is evident. The increasing number of journals from which important and relevant articles are found, combined with the decreasing number of personal subscriptions [9], makes it more important than ever for physicians to choose carefully which journals to subscribe to and read. This decision should not be based on intuition alone, as shown in an important study by obstetricians and gynecologists. Weiner et al. [10] sought to determine which journals had published numerical data on the relation between oral contraceptive use and cancer-information they judged to be clinically important and readily available in their subspecialty journals. Assessing 3735 articles identified by MEDLINE searches, only 27 studies reported numerical data, of which 23 were published in mainstream general medical journals. Only four were published in obstetrics and gynecology journals.
Since the publication of the study by Weiner et al. [10], several groups have tried to determine targeted journal subsets that could provide the most important clinical information to physicians in different specialties. Birken and Parkin [11] assessed journals with pediatric content. Using data from pediatric-related systematic reviews in the Cochrane Database for Systematic Reviews for 1997, as well as policy statements from the American Academy of Pediatricians and the Canadian Paediatric Society, they determined that four general medical journals and three pediatric specialty journals provided access to most of the Gehanno and Thirion [12] used MEDLINE searches and the Science Citation Index (SCI) Impact Factors to identify journal subsets in occupational health. Eight journals provided coverage of 27% their discipline content; 38 journals increased this to 52%. Coverage needed to be expanded beyond their specialty journals for them to remain current in occupational health.
Lee et al. [13] sampled research articles from 30 randomly selected journals from a list of 107 general internal medicine journals defined by SCI. They found that journals with high citation rates, SCI Impact Factors, and circulation rates; low manuscript acceptance rates; and listing on the Brandon/Hill Library List [14] were predictive of higher article methodologic scores.
Ebell et al. [15] as well as our research group [16] present an alternative approach for clinicians to keep up to date with current literature. Both groups produce summaries of important advances in areas of clinical care so that individuals do not have to read primary journals and evaluate reports. Ebell et al. provided results of a hand search of 85 core journals of interest to family/general practice. Physicians read these journals for six months and identified articles that were considered to be POEMs (patient-oriented evidence that matters). A POEM addresses a clinical question encountered by a family physician at least once every two weeks, measures patient-oriented outcomes, and presents results that will likely affect practice. The report provides summaries of which journals publish important clinical advances for general/family practice.
In this article we report on a survey of the contents of 170 core clinical journals for the publishing year 2000 to assess which journals publish the highest number of methodologically sound and clinically relevant studies in the disciplines of internal medicine, general/family practice, general practice nursing, and mental health. In the "Methods" section we describe our two-step article selection process for clinical importance and methodologic rigor, which is very similar to that used by Ebell et al. [15]. The data we provide reflects the merit of individual journal titles from a clinical perspective; it may help clinicians to choose which journals to read, and health sciences libraries to include them in their collections.

Methods
The Health Information Research Unit of the Department of Clinical Epidemiology and Biostatistics at McMaster University, Ontario, Canada, publishes several secondary "evidence-based" journals, systematically selecting, summarizing and appraising articles in a broad range of primary clinical journals. In 2000 we prepared ACP Journal Club (ACP J Club) to support internal medicine , Evidence-Based Medicine (EBM) to support general/family practice, Evidence-Based Nursing (EBN) to support general care nursing, and Evidence-Based Mental Health (EBMH) to support mental health. To identify potential candidate articles for inclusion in these journals, six Masters-level trained staff read each article in the major general healthcare journals and those in the disciplines and subdisciplines related to the content of each abstract journal. The list of these journals (see Appendix) is comprised of titles suggested by librarians, clinicians, editors, and editorial staff; SCI Impact Factors; and systematic examination of the contents of each title for at least six months. More than 400 journal titles have been assessed since the abstract journals were started in 1991.
We consider the Cochrane Database of Systematic Reviews to be a separate journal that publishes systematic reviews of the literature on a quarterly basis. This is consistent with the U.S. National Library of Medicine's decision to index the Cochrane Database of Systematic Reviews as a separate journal. We evaluate only the new reviews and those that are substantially updated each quarter. We do not con-sider the rest of the database or protocols that describe reviews that are in progress or being planned.
Original and review articles are placed in one or more of seven categories of study type-therapy and prevention, screening and diagnosis, prognosis, etiology and harm, economics and cost, clinical prediction guides, and qualitative studies [16]. All categories have a set of pass/fail rules for selection (see: http://www.acpjc.org/shared/ purpose_and_procedure.htm), except for qualitative and cost studies. Basic inclusion criteria are that the articles i) are about the healthcare of humans; ii) have at least one clinically important outcome; and iii) use appropriate statistical analyses. As an example of category-specific criteria, an article on screening or diagnosis must meet these additional criteria: • a spectrum of participants were included, some with the disease or condition of interest and some without • objective diagnoses were made using the "gold" standard or current clinical standard for diagnosis of the disease or condition • participants received both the new test and some form of the diagnostic standard • the diagnostic standard was interpreted without knowledge of the test result and vice versa.
The basic inclusion criteria are based on study design and methodology principles for evidence-based healthcare. Their use identifies studies that have data related to patients or those at risk of disease, diseases and conditions, and real-life clinical settings. Therefore, a study or review article that meets the criteria can be considered to be appropriate for possible use in patient care decisionmaking. The article readers are trained and retested annually so that they can reliably apply these selection rules for inclusion in our evidence-based journals (kappa measuring chance-adjusted agreement > 80% for all categories) [16].
With a research grant from the U.S. National Library of Medicine we intensified our data collection from the reading process related to the evidence-based journals for the publishing year 2000. All articles in 170 journals were classified as to whether they were "of interest" to the healthcare of humans and, if so, whether they reported original data or were systematic review articles. These original studies and reviews were classified into all possible categories (where more than one category could apply, for example, a therapy article that included economic data), and were then given a pass or fail methodologic designation for each category.
Articles passing methodologic criteria were assessed further for clinical interest by an editorial group of practicing clinicians for each abstract journal. These clinicians have expertise in methodology and specific areas of healthcare such as gastroenterology or neonatology nursing. At this point the clinician raters excluded all studies with preliminary results, interventions that were not readily available or proven useful, already known and applied findings, and topics addressing rare conditions or diseases. After review, often by a team of three-five clinicians (see http:// hiru.mcmaster.ca/more/RatingFormSample.htm for a copy of the rating system that was used in paper format for this study) some articles were further processed. The editors chose articles to be abstracted that they considered to have the most important message for clinicians. The remaining pass articles were listed as "Other Articles Noted" if their content was of relevance to the disciplines covered by the abstract journals.
This dual selection process (methodologic rigor and clinical importance) provided insight into which journals yielded the highest numbers of pass articles. This had major implications for clinical practice at two levels of clinical relevance. The more stringent level includes articles that were summarized in each abstract journal. The second, less stringent level includes articles that are abstracted as well as those articles that are listed in the Other Articles Noted sections. Analysis was done by abstract journal title (ACP J Club, EBM, EBN, and EBMH) to ascertain which journal titles were most important to their target clinical audience (internal medicine, general/ family practice, general practice nursing, and mental health, respectively). SCI Impact Factors were collected for each journal title for each discipline. If an SCI Impact Factor was not available we sought Social Science Index Impact Factors. These data were analyzed to determine if Impact Factors were related to yield of clinically important advances, as found by Lee et al. [13] and Gehanno and Thiron [12].

Results
For 2000, the 170 core journals we selected published 60,352 articles. The total number of pass articles was 3059 for original studies and 1073 for reviews. An article could be counted more than once if it passed for multiple categories. Six journals did not publish any pass articles. The complete list of journals and their yield appears in the Appendix.
The category breakdown of pass articles for original studies and review articles, respectively, was 1639 and 662 for therapy and prevention, 152 and 47 for screening and diagnosis, 195 and 22 for prognosis, 290 and 308 for etiology and harm, 35 and 10 for economics, 358 and 8 for qualitative studies, and 93 and 4 for clinical prediction guides.
The top 20 journals for yield of pass articles are included in Table 1. The titles varied considerably in both the total number and proportion of clinically relevant articles that they published. For example, 95.0% of the articles (all reviews) in the Cochrane Database of Systematic Reviews passed our criteria, while only 2.8% of the articles in the American Journal of Gastroenterology met standards for clinically applicable studies. (The American Journal of Gastroenterology is a specialty journal and a substantial proportion of its content is preclinical. These preclinical articles, by definition, did not meet the clinical criteria in this study.) Generally, a clinical reader would need to read in the range of 13-14 articles from these top 20 journals to obtain one that is directly clinically important in any healthcare area, although the range is substantial (1.1 to 36.9). We call this number the "number of articles needed to be read" or NNR.
The number of pass articles did not correlate with SCI Impact Factors for the top 20 journals (correlation coefficient 0.29, P = 0.24). Analysis of the top 50 titles showed a weak correlation (correlation coefficient 0.41, P = 0.004) for the same analysis.
The breakdown by discipline was done using the total number of articles that were selected for inclusion in each of the four abstract journals-internal medicine, general/ family practice, general care nursing, and mental health (Tables 2,3,4,5). Both the total number of articles abstracted and the total number of articles in each journal (abstracted and "Other Articles Noted") are included in Tables 2, 4, and 5 giving a two-level assessment of "clinical worth". EBM does not publish an "Other Articles Noted" Section.

Internal medicine content (ACP J Club)
The journals contributing articles important to the practice of internal medicine (ACP J Club) are shown in Table  2. Substantial drop-off is seen after the top three titles (New England Journal of Medicine, JAMA, and Lancet). These three journals and the Cochrane Database of Systematic Reviews provided 56.5% of the articles abstracted, with 28 additional journals providing the other 43.5%. Fifteen titles provided only one article each and, overall, 32 journals provided at least one article for abstraction. Another 51 journals provided at least one article in the "Other Articles Noted" section. Thus, 83 journals from our list of 170 published studies important to internal medicine.
The NNR to obtain one high-quality and clinically relevant study or review varied considerably across the titles. For the more stringent definition of clinical relevance (article abstracted in ACP J Club), the range of NNR for internal medicine was from 40.4 for the Cochrane Database of Systematic Reviews to 1334 for Neurology. For the less stringent definition (article abstracted or noted in ACP J Club), the NNR range for internal medicine was from 3.4 for the Cochrane Database of Systematic Reviews to 242 for Acta Obstetrica et Gynaecologica Scandinavica.
Correlating the number of articles published in ACP J Club with their SCI Impact Factor showed a large and positive correlation for both levels of clinical importance (correlation coefficient 0.786, P < 0.001 for the more stringent definition; correlation coefficient 0.688, P < 0.001 for the less stringent definition of clinical importance). These findings support the findings by Lee et al. [13] that SCI Impact Factors were correlated with quality articles for general internal medicine.

General/family practice content (EBM)
The most important articles for general/family practice (publication in EBM) were published in BMJ, Lancet, Cochrane Database of Systematic Reviews, Archives of Disease in Childhood, and Annals of Internal Medicine-these journals provided 55.6% of EBM content (Table 3). Overall, 45 titles provided abstracts for general/family practice coverage. The "shape" of the data is different for general/ family practice than for general internal medicine, with more journals providing articles for abstraction. This is consistent with the discipline because general/family practitioners must use knowledge from a broader range of health conditions (including pediatrics and obstetrics, for example) than general internists and other specialists. Only the most stringent definition for clinical worth could be evaluated for EBM content because the "Other Articles Noted" section of the journal did not exist in the year 2000. The NNR for general/family practice ranged from 55 for the Cochrane Database of Systematic Reviews to 1351 for Circulation.
Correlation analysis showed that the number of qualified articles in each journal title was associated with the journal's SCI Impact Factor (correlation coefficient 0.546, P = < 0.001). This shows substantial agreement between SCI Impact Factors and number of articles but slightly less agreement than that found using the general internal medicine data (correlation coefficients > 0.688).

General care nursing content (EBN)
Nursing content came from many journals, including journals that are considered to be primarily targeted at physicians, and was not concentrated in a small set of journal titles (Table 4). To reach 51.0% of the abstracted articles, seven titles were needed (Qualitative Health Research, Cochrane Database of Systematic Reviews, Pediatrics, JAMA, Lancet, BMJ, and Journal of Advanced Nursing). Thirty-two other journals provided articles for abstraction *The NNR is the number of articles that would have to be read in each journal to identify one with high quality methods that is clinically relevant; **The SCI Impact Factor is the Science Citation Index Impact Factor (rating of how important each journal is in relation to citations). Data are for 2000. Articles have not been screened for direct clinical relevance beyond basic criteria of having at least one clinically important outcome.  and 33 journals provided studies that were listed only in the "Other Articles Noted" section; 72 journals in total provided content for general care nursing.

Mental health content (EBMH)
Mental health content was also spread over a broader range of journals than was internal medicine (  Table 5 show that studies related to mental health are published in many journals and specialties-a reflection of the broad nature of the discipline. The NNR for mental health for the most stringent definition of clinical relevance ranged from 20.1 for Archives of General Psychiatry to 1142.7 for BMJ. Archives of General Psychiatry also has the lowest NNR for the less stringent definition (11.5), with CMAJ having the highest *Articles abstracted are those that pass methodological criteria and are deemed to be the most important by practicing internists. The other articles listed are articles with the same high-quality methods but are considered to be slightly less important clinically by practicing clinicians; **The number of articles needed to read (NNR) is a measure of the ratio of number of relevant articles (abstracted or combined abstracted or listed) divided into the total number of articles for each journal title; ***SCI Impact Factors not available for analysis; ****Canada Coordinating Office for Health Technology Assessment Reports; N.A. Not applicable. Data are for 2000. Note that the AHRQ/AHCPR and CCOHTA reports were not considered journal titles and read as such for this report.  A. † EBM does not include a listing of important but not abstracted articles ("Other Articles Noted" section); *Articles abstracted are those that pass methodological criteria and are deemed to be the most important by a team of practicing general/family practitioners; **The number of articles needed to read (NNR) is a measure of the ratio of number of relevant articles (abstracted) divided into the total number of articles for each title; ***SCI Impact Factors not available for analysis; N.A. Not applicable. Data are for 2000. Note that the AHRQ/AHCPR reports were not considered journal titles and read as such for this report.  NNR (1007) of those journals with at least one article on mental health. EBMH has a smaller "Other Articles Noted" section. Only eight additional journals provide articles for this section beyond the 61 that provide articles for abstraction.
A weak association was shown between the number of published mental health articles and SCI Impact Factors (correlation coefficient 0.386, P = 0.02 for the more stringent definition; and correlation coefficient 0.381, P = 0.01 for the less stringent definition).

All disciplines
Combining the content across the four discipline areas, we again see

Conclusions
We found that the majority of articles for each discipline were sequestered in a small subset of journals. This is consistent with Bradford's Law of Scattering for journal subsets, which states that the important articles on any topic will be concentrated in a small subset of journals with exponential drop-off in numbers of relevant articles across journal titles [17]. Across disciplines and study areas, approximately 70% of articles are often found in 30% of journals in any given area of study.
Not surprisingly, for broad-based disciplines such as mental health and nursing, the number of titles was greater than for more focused disciplines such as internal medicine. SCI Impact Factors were highly correlated with the number of important clinical articles in separate titles for internal medicine and, to a lesser extent, for general/family practice, and mental health but not for general practice nursing. This likely reflects the volume of clinically important research activity in these fields-with especially high volumes in the disorders managed by internal medicine and its subspecialties-coupled with the avidity of authors from all disciplines to submit their best studies to the high-circulation general journals.
As found by Weiner et al. [10] and others, most of the important advances in any discipline are not published in specialty journals but in the more general healthcare journals such as JAMA, Lancet, BMJ, New England Journal of Medicine, and Cochrane Database of Systematic Reviews.
Health professionals in all disciplines should be aware that major advances in any field will most likely be pub- *Articles abstracted are those that pass methodological criteria and are deemed to be the most important by practicing nurses. The other articles listed are articles with the same high-quality methods but are considered to be slightly less important clinically by practicing nurses; **The number of articles needed to read (NNR) is a measure of the ratio of number of relevant articles (abstracted or combined abstracted or listed) divided into the total number of articles for each journal title; ***SCI Impact Factor not available for analysis; N.A. Not applicable or available. Data are for 2000. Note that the AHRQ/AHCPR reports were not considered journal titles and read as such for this report.  *Articles abstracted are those that pass methodological criteria and are deemed to be the most important by practicing mental health professionals. The other articles listed are articles with the same high-quality methods but are considered to be slightly less important clinically by mental health professionals; **The number of articles needed to read (NNR) is a measure of the ratio of number of relevant articles (abstracted or combined abstracted or B-listed) divided into the total number of articles for each journal title; ***SCI Impact Factor not available for analysis; N.A. Not applicable or not available. Data are for 2000. Note that the AHRQ/AHCPR reports were not considered journal titles and read as such for this report.

Journal of American Academy of Child and Adolescent Psychiatry
lished in the main general medicine journals, while at the same time recognizing that specialty journals also publish important information. Much variation exists across journal titles in both the number and proportion of articles that are high quality, clinically important, and newsworthy. Variation also exists across disciplines. It is also interesting to note that all lists of important journals discussed in this report and also the one by Ebell et al. [15] include both North American and European titles. Reading choices for clinicians cannot be based on national or discipline boundaries alone.
Of the 45 titles that provided articles to EBM, 23 were on the list provided by Ebell  Consistent with the data from Weiner et al. [10], many advances important to general practice nursing are not published in nursing specialty or discipline-specific journals. Only four of the top 17 and eight of the top 41 journals in Table 4 are considered nursing specialty titles. Overall 39 titles provided at least one article for abstraction and an additional 33 titles provided at least one article to the "Other Articles Noted" section, again showing the broader spectrum of journals that publish articles important to general care nursing.
Clinicians in the target disciplines described here could use our findings to focus their fulltext readings. For other disciplines, a similar audit of clinical yield would be needed, either from an appropriate secondary journal that systematically reviews specified journals, or an independent audit. Another approach to staying current may be to subscribe to one or more secondary journals that highlight important clinical advances. These secondary publications have not only selected the most appropriate studies for clinical consideration, they highlight important aspects of methodology and implementation. This assessment of studies before application can be time-consuming and difficult for many clinicians, and involves a certain amount of training and practice to become proficient. Many examples of secondary publications exist in various disciplines and include the four studied in this report, POEMs [15], and Journal Watch. Use of these summaries of studies and reviews can be supplemented by access to fulltext articles.
Many academic medical centers and hospitals provide good online access to major healthcare journals. For example, the Health Sciences Library of the University of Pittsburgh, PA, USA, provides online access to 24 of the top 25 journals in this study and all 25 of the journals identified as high yielders by Ebell et al. [15]. Specialized health libraries with limited budgets may wish to focus on the journals, either in paper or electronic format, with the highest yield for the disciplines they serve.