A scoping review of competencies for scientific editors of biomedical journals

Abstract

Background

Biomedical journals are the main route for disseminating the results of health-related research. Despite this, their editors operate largely without formal training or certification. To our knowledge, no body of literature systematically identifying core competencies for scientific editors of biomedical journals exists. Therefore, we aimed to conduct a scoping review to determine what is known on the competency requirements for scientific editors of biomedical journals.

Methods

We searched the MEDLINE®, Cochrane Library, Embase®, CINAHL, PsycINFO, and ERIC databases (from inception to November 2014) and conducted a grey literature search for research and non-research articles with competency-related statements (i.e. competencies, knowledge, skills, behaviors, and tasks) pertaining to the role of scientific editors of peer-reviewed health-related journals. We also conducted an environmental scan, searched the results of a previous environmental scan, and searched the websites of existing networks, major biomedical journal publishers, and organizations that offer resources for editors.

Results

A total of 225 full-text publications were included, 25 of which were research articles. We extracted a total of 1,566 statements possibly related to core competencies for scientific editors of biomedical journals from these publications. We then collated overlapping or duplicate statements which produced a list of 203 unique statements. Finally, we grouped these statements into seven emergent themes: (1) dealing with authors, (2) dealing with peer reviewers, (3) journal publishing, (4) journal promotion, (5) editing, (6) ethics and integrity, and (7) qualities and characteristics of editors.

Discussion

To our knowledge, this scoping review is the first attempt to systematically identify possible competencies of editors. Limitations are that (1) we may not have captured all aspects of a biomedical editor’s work in our searches, (2) removing redundant and overlapping items may have led to the elimination of some nuances between items, (3) restricting to certain databases, and only French and English publications, may have excluded relevant publications, and (4) some statements may not necessarily be competencies.

Conclusion

This scoping review is the first step of a program to develop a minimum set of core competencies for scientific editors of biomedical journals which will be followed by a training needs assessment, a Delphi exercise, and a consensus meeting.

Background

“…journals, some of which have reported research for many decades, are still not producing articles that are clear enough to really judge a study’s conduct, quality, and importance—let alone to allow other researchers to reproduce it or build on it” [1].

Biomedical journals are the main route for disseminating the results of health-related research [2]. However, when examined more closely, the articles that journals publish are problematic; critical details are often missing or poorly reported, consequently reducing their quality, transparency, reproducibility, and usefulness for decision makers [3] – this is wasteful, diminishes scientific and fiscal value, and is unethical [4]. Authors and scientific journals share the majority of the responsibility for these shortcomings, as the former are accountable for the integrity of a study’s conduct and the accuracy of reporting of the content within the manuscript, while the latter are accountable for decisions regarding its publication. On the side of journals, it is scientific editors (by which we mean editors, and ultimately the Editor-in-Chief, who are tasked with making decisions about the content and policies of journals) who are accountable for all material published in their journals. Readers have a right to expect these editors to implement all reasonable actions that could lead to best practices within their journals, as well as journals having processes in place to ensure the quality of the papers they publish.

Unlike many other professional groups, such as clinicians and healthcare professionals, many scientific editors of biomedical journals operate largely without formal training and universal certification is not yet a high priority [5]. Instead, editors generally are invited to serve in their role by publishers, based on their expertise and stature in the field, since such expertise is essential for evaluating research and stature is important for establishing the reputation of the journal and attracting submissions. However, such expertise does not guarantee that editors have the background or training necessary to carry out their roles and responsibilities. Editors may or may not be paid for their role and financial support for the editorial role often does not include travel or training funds. Most editors work part-time as they continue their academic responsibilities in research and/or clinical work, with time for completing editorial responsibilities – much less training – being at a premium. Researchers and peer reviewers similarly have no international standardized formal training or certification as to research conduct, reporting, and evaluation, making the editor’s job even more demanding. This situation is highly problematic given that the consequences of deciding what gets published and the degree of quality that is acceptable impacts future research, decisions, and healthcare directly. Our view is that the lack of consistent training of editors reduces the value of the published literature, including its quality, transparency, and reproducibility, thereby reducing value for money to funders and the usability of research findings, ultimately degrading public trust in the research record [3]. However, we are unaware of any research that directly addresses this topic. Additionally, while the training of biomedical editors is an important mechanism to ensure the quality of the published literature, other important changes in tandem with this, including re-examining the training offered to peer-reviewers and training graduate students in study conduct, analysis, interpretation, and reporting, could also have a beneficial effect.

Some organizations, for example, the World Association of Medical Editors (WAME) [6] and the Committee on Publication Ethics (COPE) [7], provide rigorously-developed resources for biomedical journal editors free of charge, including guidance on the role of the medical editor, editorial policies, and listservs on which editors’ questions and issues are discussed. There are also individual websites and blogs, such as “Journalology” [8], that provide thoughtful commentary on current issues related to publication science. Several commercial groups offer short training courses for editors [9, 10]. However, for any comprehensive editor training program to work effectively and be assimilated into practice, it must be based on what the broader biomedical editor community considers to be core competencies.

We are unaware of any body of literature systematically identifying core competencies for biomedical editors, nor any agreement on or attempt at a consensus process to determine what they should be. For the purposes of this research, we borrow from the literature on competency-based continuing professional development to define competence as “the array of abilities across multiple domains or aspects of [practitioner] performance in a certain context” [11]. We thus define core competencies as the essential knowledge, skills, and behaviors necessary for the practice of scientific editing of biomedical journals. We believe it is important to develop a set of core competencies so that training programs can then be developed and tailored with the intent that all editors meet some basic globally agreed-upon standards. Other stakeholders, such as publishers (including medical associations who have their own journals), peer reviewers, and authors (researchers), also need to contribute to this effort. Herein, as a starting point, we report a scoping review of possible core competencies of scientific editors of biomedical journals.

Objectives

The objective of this scoping review was to conduct a systematic search of the literature on the competencies required for scientific editors of biomedical journals to effectively and efficiently produce transparently reported and correctly analyzed and interpreted publications. Our specific aim was to answer the research question: “What is known from the literature on the competency requirements of scientific editors of peer reviewed biomedical journals?” with the goal of summarizing the existing literature. The purpose of this scoping review is to inform the future development of a set of core competencies for scientific editors of biomedical journals, which we hypothesize will ultimately lead to improvements in the quality of the published literature.

Methods

The protocol for this project has previously been deposited in the University of Ottawa’s Digital Repository (uOttawa Research) prior to beginning the screening phase [12]. Our methodological approach was guided by the Arksey and O’Malley Framework [13], as well as the additional suggestions to this framework made by Levac [14]. Specifically, we undertook the six-step process of: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting results; and (6) consultation.

Search strategy

We searched the MEDLINE®, Cochrane Library, Embase®, CINAHL, PsycINFO, and ERIC databases, all from inception to November 10th, 2014 (Additional file 1). The specific search strategies were created by a Health Sciences Librarian with expertise in scoping review searching. The MEDLINE strategy was developed with input from the project team, then peer reviewed by a second librarian not otherwise associated with the project using the PRESS standard [15]. We also conducted a grey literature search, which included hand searching the reference lists of included articles, as well as searching key journals, in particular JAMA from 1989, and BMJ, PLoS Medicine, European Science Editing, Annals of Internal Medicine, and CMAJ from 2004 onward to identify publications related to the Peer Review Congresses [16].

Expanded scoping exercise

Given that this scoping review is part of a larger program to develop core competencies for scientific editors of biomedical journals, in addition to seeking research literature, we also incorporated an expanded scoping exercise that included non-research-based (published and unpublished) materials to fill an expected gap in research on competencies of scientific editors of biomedical journals. This expectation was based on a previous systematic review [17], which found no comparative studies on training for scientific editors of biomedical journals, and an associated environmental scan that found only a few training opportunities for these editors [18]. Additionally, our anecdotal experience was that most descriptions of editor competencies are found in editorial-type publications (e.g. commentaries), job postings, and guidance documents (e.g. COPE Guidelines for Editors) – all of which would not generally be captured in a traditional scoping review of published research.

The expanded scoping exercise included (1) searching the databases from the search strategy and grey literature for non-research-based publications, (2) searching the results of an environmental scan from a previous related project [17, 18], (3) conducting a new environmental scan with additional search terms, and (4) searching the websites of existing networks (i.e. EQUATOR Network), major biomedical journal publishers (i.e. Wiley-Blackwell, Elsevier, BioMed Central, BMJ Publishing Group, Springer), and organizations that offer resources for editors (i.e. COPE, WAME, Council of Science Editors, European Association of Science Editors, and International Committee of Medical Journal Editors). For the database searches, the full-text of all potentially relevant documents were retrieved and independently reviewed for eligibility, in duplicate, by members of the team using a priori eligibility criteria. Disagreements between reviewers were resolved by consensus or by a third member of the research team. Both environmental scans employed the same methodology, which involved the use of the Google search engine to run a series of two- and three-word keyword searches (Additional file 1). For each set of search outcomes, the first 50 Google results were screened for relevant information. If any of the last 10 results contained useful information, another 10 results were screened. This process was continued until a set of 10 results with no relevant information was found [18].

Study selection: inclusion criteria

Population

Articles with statements mentioning competencies, knowledge, skills, behaviors, and tasks (henceforth referred to as ‘competency-related statements’) pertaining to the role of scientific editors of peer-reviewed health-related journals (including Editors-in-Chief and associate/academic editors, and full-time professional journal editors) were included. Articles related to all other types of editor not directly involved in evaluation, peer review, revision, and selection of content (e.g. managing editors, technical editors, copy editors) were excluded.

Disciplines

We adopted MEDLINE’s journal selection criteria for our definition of health [19]. This definition includes journals that are “predominantly devoted to reporting original investigations in the biomedical and health sciences, including research in the basic sciences; clinical trials of therapeutic agents; effectiveness of diagnostic or therapeutic techniques; or studies relating to the behavioral, epidemiological, or educational aspects of medicine” [19]. This definition encompasses biomedical journals as well as those in the disciplines of psychology and education. For feasibility purposes, we did not include journals from the physical or natural sciences.

Study designs

The review included all study designs as well as editorials and commentaries. Economic evaluations and letters were excluded, as neither was expected to contribute useful data for the purposes of this scoping review. For feasibility purposes, we included articles written in English and French only. We did not include English or French abstracts of papers written in another language.

Screening

Following the execution of the search strategy, the identified records (titles and available abstracts) were collated in a Reference Manager [20] database for de-duplication. The final unique record set of potentially eligible studies was exported to Internet-based software, DistillerSR (Evidence Partners, Ottawa, Canada), through which screening of records and data extraction were carried out. Each title and abstract was screened by two of four reviewers (LS, JG, JT (see Acknowledgements), and MG (see Acknowledgements)) using a ‘liberal accelerated’ method [21] (i.e. one reviewer screened each record and a second reviewer screened only excluded records, independently). The full-text of all remaining potentially eligible papers was then retrieved, uploaded into DistillerSR, and reviewed for eligibility, independently, by two members of the team (LS and JG) using a priori eligibility criteria. Disagreements between reviewers at this stage were resolved by consensus or by a third member of the research team.

Charting the data

A data extraction form was developed a priori to capture information on each document included in the review. It was piloted and refined based on feedback from the exercise. Three people (JG, KDC, LS) carried out data extraction in the following manner: data were extracted by one reviewer and a second reviewer conducted verification of the data for all records. Disagreements between reviewers were resolved by consensus. General study characteristics extracted for the database search were: first author name and contact information (of corresponding author), year of publication, institutional affiliation of first author, country, language of publication, study design, and funding source. For the environmental scans we extracted the URL, title of the document, language of publication, and who produced the document (affiliation). For all documents, we collected descriptions of any statements potentially relating to the competencies of scientific editors of biomedical journals, such as descriptions of particular skills, knowledge, attitudes, behaviors, tasks, and training.

Collating, summarizing, and reporting the results

In an effort to create a useful summary of the data for the next steps of our program to develop core competencies for scientific editors of biomedical journals, we combined the competency-related statements retrieved from all sources. First, two people (JG, KDC) classified all statements pertaining only to Editors-in-Chief into a single category, since these would be considered to be beyond the core competencies of scientific editors more generally. They then collated overlapping or duplicate statements to produce a list of unique statements. Finally, they grouped statements into emergent themes to make them more manageable for future use (e.g. in an upcoming Delphi exercise), and so that they would be understandable to readers. While some of the wording of particular statements was modified to assimilate overlapping statements, where statements were expressed as knowledge, skills, behaviors, or tasks that implied competencies, but not as competencies themselves, we did not edit or translate them to express competencies, in order to preserve the original intent. The relationships between behaviors, tasks, and competencies will be the subject of discussion and translation undertaken as part of the consensus meeting phase of this project.

Results

Database search

We screened 5,837 titles and abstracts, of which 360 were screened in full-text (Fig. 1). Of these, 206 were excluded, leaving 154 publications meeting the inclusion criteria. Twenty-five of these publications were research based (Table 1) and the remaining 136 were editorial in nature (Additional file 2).

Fig. 1

Study flow diagram

Table 1 Included research-based publications

Charting the data

General characteristics

Research-based publications

A total of 18 publications from the database search presenting the results of research (subsequently termed ‘research-based publications’) were considered relevant to core competencies for scientific editors of biomedical journals, along with another seven articles found in the grey literature search (including six conference abstracts for which there appears to be no full text publication) (Table 1). None of these 25 articles had an outline or description of core competencies of scientific editors as an objective of the research. Fifteen studies were survey-based research, three were descriptive studies, three were case studies, and two were final project reports (one from a task force and the other from a study funded by the US National Institute of Education). The remaining two studies were a systematic review and a mixed-methods study. Five studies reported receiving funding. Nineteen studies were published in different journals and the remaining six were part of the Peer Review Congress Proceedings. Publication dates ranged from 1975 to 2014; 2009 was the year with the most publications (n = 3), followed by 2011 (n = 2) and 2013 (n = 2). Twelve of the studies’ first authors were from the USA, 10 from the United Kingdom, and one each from Iran, India, and Canada. The publications produced a total of 40 competency-related statements (i.e. possible competencies), with individual publications yielding between zero and 14 statements and a median of one statement per publication.

Expanded scoping exercise

Non research-based publications

A total of 136 non-research-based publications were considered relevant to core competencies for scientific editors of biomedical journals, yielding a total of 537 competency-related statements (Additional file 2). Similar to the research-based literature, none of these publications had the explicit goal of outlining a set of core competencies for scientific editors. Overall, 133 publications were editorial in nature, while the remaining three included a lecture, a job description, and an interview. Seventeen journals had multiple included publications, with three of them (Australian and New Zealand Journal of Obstetrics and Gynaecology, Croatian Medical Journal, and Indian Journal of Medical Sciences) producing three publications each while the other 14 journals had two publications each. JAMA also had four publications across three journals in its collection (JAMA, JAMA Internal Medicine, JAMA Ophthalmology). The rest of the sample consisted of a single publication per journal. The date of publication ranged from 1954 to 2015; 2011 (n = 20) and 2012 (n = 17) were the 2 years with the most studies. The sample included 66 studies with first authors originating from the USA and 19 from the United Kingdom, with representation from another 18 countries as well among first authors. The individual publications yielded between zero and 15 competency-related statements, with a median of two statements per publication.

Environmental scan of training in Journalology

We reviewed all 258 documents listed in the Repository of Ongoing Training Opportunities in Journalology [20], which houses all of the data from an environmental scan of training in Journalology carried out by members of our team in 2013. From this repository, we extracted 11 relevant non-research-based articles from which we were able to retrieve 556 competency-related statements relating to scientific editors of biomedical journals (Table 2). Seven of these documents were from organizations that provide guidance to editors (e.g. WAME, COPE, International Committee of Medical Journal Editors) and the remaining documents were from a variety of other sources.

Table 2 Included documents from expanded scoping exercise

Environmental scan of core competencies for scientific editors of biomedical journals

The environmental scan carried out for this project consisted of a total of 40 keyword searches (Additional file 1) that yielded 48 relevant documents, of which 35 were deemed to meet the eligibility criteria after screening in duplicate. These 35 documents produced 426 competency-related statements (Table 2). Among the sample, 18 documents were produced or published by journals, nine were from associations and societies, six were from organizations providing guidance to editors, and two were from publishers.

Search of networks, publishers, and meetings

A search of networks, publishers, and meetings produced an additional five documents, three of which were documents from publishers’ websites and two from the EQUATOR Network (Table 2). We extracted seven competency-related statements from these additional documents.

Of the 64 documents in the expanded scoping exercise, the European Association of Science Editors Editor’s Handbook (2nd Edition) [22] had the highest number of competency-related statements (n = 166). The median number of competency-related statements across the scoping exercise documents was six.

Collating and summarizing the data

The combined 1,566 competency-related statements originating from the 225 total documents were collated and then de-duplicated, producing a list of 203 unique statements (i.e. possible competencies of scientific editors) (Table 3). The statements were organized into seven categories that emerged from the data: (1) dealing with authors; (2) dealing with peer reviewers; (3) journal publishing; (4) journal promotion; (5) editing; (6) ethics and integrity; and (7) qualities and characteristics of editors.

Table 3 Competency-related statements ^a

Discussion

This scoping review identified 225 relevant publications, spanning more than 60 years, and involving authors from more than 20 countries. It produced a comprehensive list of possible competency-related statements for the scientific editor position within biomedical journals. This categorized list of statements will be used in a subsequent Delphi exercise aiming to ask a broad spectrum of scientific editors of biomedical journals to rate the importance of each statement in relation to performing their duties as a scientific editor. This data will then help inform a consensus meeting in which a select group of editors will collaborate to outline a set of core competencies.

Despite our finding that the competencies of editors have been discussed in the published literature since the mid-1950s, a recent systematic review [17] found no comparative studies of the effectiveness of training for editors. This is concerning, given the gate-keeping role that scientific editors play as guardians of the scientific record [23]. Trends in the number of publications annually would seem to indicate that, while the overall number of publications in this area has grown since the topic first emerged in the literature, interest peaked around 2011 and is now waning. The trend, however, is more reflective of editorial-type articles in journals, which have declined since 2011, while the number of research-based publications has remained relatively stable (yet sparse) during the same timeframe.

One possibility for the decline in total publications is that organizations may believe these issues have already been identified and they are working on training materials for them rather than publishing more research or commentaries about them in medical journals. A possibility for the continued lack of research in this area may be that the focus of many major funding agencies is to fund by disease (e.g. heart, kidney, cancer, diabetes). As journal editing, and the field of journalology more broadly, is a domain that spans across the spectrum of research on diseases, it is often difficult to find appropriate funding opportunities and even more difficult to convince specific disease-based funding agencies that the research merits their funding investment.

A major challenge that we anticipated as part of this scoping review was that a large proportion of the evidence may not be in the traditional research-based literature. For this reason, we placed a heavy emphasis on extensively searching the grey and non-research-based literature by including two related environmental scans as part of the larger program on core competencies for scientific editors of biomedical journals. However, it is still possible that we may not have captured every aspect of a biomedical editor’s work in our searches. In particular, some of the more tacit (e.g. difficult to describe) aspects of the work of editors may simply not be documented, or may reside in documents that may not necessarily be found in a database or Google search (e.g. resources residing behind membership or password-protected webpages, paywalls). We expect that any critical missing items will be brought forth in the training needs assessment, the Delphi, or the consensus meeting. Additionally, because of the subjective nature of this type of data extraction, it is possible that some competency-related statements were missed within the included publications. However, we feel that the likelihood of this was reduced as several members of the research team are editors. Finally, due to the broad inclusion criteria and the decision to preserve the wording used by authors to describe potential competencies as much as possible, it is likely that some items may not actually be competencies per se, but may instead describe tasks, behaviors, and knowledge related to competencies. However, these items are still useful in describing important aspects of editors’ work and will therefore contribute valuable information for the development of core competencies.

With the large number of competency statements and our desire to create a manageable list for use downstream in our program of research, our efforts to remove redundant and overlapping items in order to streamline the list may also have led to the elimination of some nuances between items that were subtly different from one another. While we implemented measures to ensure consistency in our methods (i.e. piloted forms, duplication of the classification exercise), ultimately there is a degree of interpretation and selectivity embedded in this process. Thus, our list of possible competencies may not include all of the competencies of biomedical editors. As noted above, the next phases of this project are designed to elicit any missing items.

Another limitation is that for feasibility purposes we only considered English and French articles, which raises the possibility that relevant information published in another language was missed. Similarly, the databases searched may not have included some journals from outside fiscally resourced countries.

Conclusion

To our knowledge, this scoping review is the first attempt to systematically identify possible competencies of editors. On its own, the review will serve to inform readers on the extent and nature of existing literature in this area, as well as the breadth of skills, abilities, tasks, knowledge, and training that may be necessary to fulfill the position of scientific editor at a biomedical journal. More importantly, the review is part of a larger program to develop a minimum set of core competencies for scientific editors of biomedical journals.

The purpose of the 203 competency-related statements generated here is to be the central tool used for a Delphi exercise involving scientific editors of biomedical journals from around the world. Subsequently, these statements will serve to stimulate discussion at a consensus meeting in which the goal will be for relevant stakeholders to agree upon a minimum set of core competencies for scientific editors of biomedical journals. This evidence-based approach will ultimately lay the groundwork for the development of specific competency-based training and certification for scientific editors of biomedical journals [5]. The development of core competencies and subsequent training represent critical steps toward ensuring that the publication of biomedical research truly represents a hallmark of quality and trustworthiness, both within and beyond the research community.