We followed Core Outcome Measures in Effectiveness Trials (COMET) guidelines  for this COS development study and report on it in accordance with Core Outcome Set–STAndards for Reporting . To commence the item generation process required for a COS, we conducted a systematic review of outcomes reported in published trials (1980 to December 2016) and registered trial protocols (January 2014 to December 2016) via (1) search term development in collaboration with two senior information specialists and conduct of the search across ten publication databases and grey literature; (2) two authors independently screening citations and extracting data on study characteristics, outcomes, and measures (with a third author as arbiter if needed); and (3) assignment of outcomes according to COMET taxonomy . We included randomized, quasi-randomized, or non-randomized intervention studies of pharmacological (e.g., haloperidol) or non-pharmacological (e.g., reorientation, music) interventions for delirium prevention, treatment, or both, conducted in adults or children experiencing an acute hospital admission. We excluded studies conducted in ICUs and those reporting interventions to treat pediatric or adult agitation on emergence from general anesthesia. In addition, our item generation process included semi-structured qualitative interviews exploring outcomes important to delirium survivors and family members.
Item reduction and consensus methods comprised a two-round, web-based modified Delphi consensus process. To gain final consensus, this Delphi process was followed by an in-person consensus meeting, hosted by the European Delirium Association, using a modified nominal group technique .
Recruitment of participants for qualitative interviews, Delphi panel, and consensus meeting
We sought a purposive and international sample from three stakeholder groups: (1) clinical researchers, (2) clinicians, and (3) delirium survivors and family. We recruited delirium survivor and family participants using a multi-modal strategy, including a designated study Twitter account, snowballing (i.e., research participants passing on recruitment materials to other potential participants), and personal contacts. Our multi-modal strategy to recruit expert clinicians and delirium researchers included recruitment flyers sent through membership lists of the American Delirium Society and Australian Delirium Association and to attendees of the European Delirium Association 2019 meeting (in-person consensus meeting), announcements at the American Delirium Society 2019 meeting, personalized recruitment emails sent to corresponding authors of studies included in our systematic review, flyers posted in UK National Health Service organizations, snowballing, and personal contacts.
Semi-structured interviews with delirium survivors and family members were conducted by telephone by a single experienced interviewer (LR). The interview guide incorporated COMET plain language  to orient participants to the terms “study outcomes” and “COS.” Interviews were audio recorded, transcribed verbatim, and content analyzed by one author .
Item reduction for identified outcomes occurred via de-duplication (i.e., removing redundant outcomes), removing outcomes related to aggregate population data rather than individual patient outcomes (e.g., number of patients receiving analgesia), and grouping similar outcomes . We grouped into a single outcome those describing adverse events, side effects, and complications, and those describing study-related feasibility or process outcomes. As more items are associated with lower COS Delphi response rates , we further reduced outcomes by removing those identified in < 5% of studies, unless specifically mentioned in survivor/family member interview transcripts. The final list of outcomes was then reviewed for wording clarity (with lay descriptions of medical terms to aid understanding) and for domain grouping.
To conduct the Delphi, we used the bespoke DelphiManager software, Version 4 (COMET Initiative, Liverpool, UK). Participants were directed to self-select their key stakeholder group (i.e., patient/family; clinician; researcher) and to score the importance of each outcome for COS inclusion, without consideration of measurability or feasibility. Importance was scored using the Grading of Recommendations Assessment, Development and Evaluations (GRADE) Scale . This is a 9-point Likert scale with scores 1 to 3 considered not important, 4 to 6 important but not critical, and 7 to 9 as critical for inclusion. This scoring method is recommended by COMET to facilitate maximum discrimination between questionnaire items [32, 33]. Participants were provided an “Unable to Score” response option and the opportunity to suggest additional outcomes. To avoid presentation bias, the DelphiManager software randomized outcome domain presentation order.
For Delphi round 1 scores, we calculated mean and standard deviation (SD) of GRADE importance scores and determined the proportion of participants rating each outcome with scores of 7 to 9 (critically important), 4 to 6 (important but not critical), and 1 to 3 (not important) for the entire expert panel, and separately for each stakeholder group. Additional suggested outcomes were deduplicated and worded appropriately for inclusion in round 2. For round 2, participants received their own round 1 scores and summarized scores, with visual representation using histograms. Participants were asked to re-score outcome importance. If a participant changed their scoring so that it moved into a new category (e.g., from “important but not critical” to “critical for inclusion”), participants were requested to provide a free-text reason for this change. For both rounds, we sent three email reminders regarding completion using the DelphiManager software.
In-person consensus meeting and nominal group technique
To inform our in-person consensus meeting, we calculated mean (SD) Delphi round 2 importance scores and determined the proportion of participants rating each outcome as critical for inclusion overall and by stakeholder group. As recommended by COMET , outcomes brought to the consensus meeting met the following criteria: scored as “critical for inclusion” by ≥ 70% of respondents and “not important” by < 15% considering all participants and for each of the three key stakeholder groups. No outcome that was rated by < 70% of participants as critically important overall or within a stakeholder group was brought forward to the consensus meeting.
For pragmatic reasons, we timed our consensus meeting with the 2019 European Delirium Association annual conference. We provided an overview of our meeting’s aim and structure and the Delphi results. We provided the importance scoring for the outcomes by stakeholder group to consensus meeting participants, for consideration during outcome ranking. Using nominal group technique methods, we held iterative rounds of small group and then whole group discussion. To avoid negating the Delphi process, participants were not permitted to suggest new outcomes. Participants ranked outcomes from most critical to least critical for COS inclusion at the end of each discussion.
The study was funded by the Canadian Institutes of Health Research. It received approval from the Research Ethics Boards of the University of Toronto, King’s College London, Sunnybrook Health Sciences Centre (Toronto, Canada), and the UK Health Research Authority (HRA) and Health and Care Research Wales (HCRW). Ethics approvals to recruit via social media, snowballing, and networking methods enabled recruitment from multiple countries including the USA, Europe, Asia, Oceania, and South America. Written informed consent was obtained from all study participants. The Del-CORs project is registered with the COMET initiative (http://www.comet- initiative. org/studies/details/796). We previously published the study protocol .