Skip to main content

Practical guidance for planning resources required to support publicly-funded adaptive clinical trials

Abstract

Adaptive designs are a class of methods for improving efficiency and patient benefit of clinical trials. Although their use has increased in recent years, research suggests they are not used in many situations where they have potential to bring benefit. One barrier to their more widespread use is a lack of understanding about how the choice to use an adaptive design, rather than a traditional design, affects resources (staff and non-staff) required to set-up, conduct and report a trial. The Costing Adaptive Trials project investigated this issue using quantitative and qualitative research amongst UK Clinical Trials Units. Here, we present guidance that is informed by our research, on considering the appropriate resourcing of adaptive trials. We outline a five-step process to estimate the resources required and provide an accompanying costing tool. The process involves understanding the tasks required to undertake a trial, and how the adaptive design affects them. We identify barriers in the publicly funded landscape and provide recommendations to trial funders that would address them. Although our guidance and recommendations are most relevant to UK non-commercial trials, many aspects are relevant more widely.

Peer Review reports

Background

Clinical trials are a vital part of improving the treatment and care of patients. Due to the increasing costs of trials [1, 2] and the need to answer important research questions as rapidly and robustly as possible, new trial methods that can increase operational and statistical efficiency are of great interest. Adaptive trial designs [3] are one such class of methods; they provide pre-planned opportunities to use accumulating trial participant outcome data to make changes to the course of the trial, whilst ensuring the statistical properties of the trial remain intact and results credible. Adaptive designs (ADs) have different features that can, for example, (1) improve the statistical power of the trial; (2) reduce the time taken and the number of participants required to evaluate treatments, thus potentially saving money and other resources; and (3) reduce exposure of trial participants to insufficiently effective, or more harmful, treatments by stopping recruitment to them early [4]. ADs are typically more statistically and operationally complex than traditional trials and require high-quality interim analyses undertaken (including implementation of decisions) rapidly; they may therefore require higher levels of (and exact timing of) effort, resources and expertise to design, set-up, deliver, analyse and report. The sample size and study length of an adaptive trial are often unknown at the outset, which can further complicate their resourcing. There has been little guidance aimed at non-commercial organisations and researchers who conduct clinical trials on appropriate resourcing of adaptive trials. If adaptive trials are inadequately resourced, their advantages may be compromised, leading to increased risk of operational or statistical biases [3, 5, 6].

The costing adaptive trials project

The Costing Adaptive Trials (CAT) project investigated the additional resources, as compared to similar non-ADs, required to support effective adaptive trials. Full details of the methods and results are reported in Wilson et al. [7]. Briefly, this research was undertaken in the UK in 2020 through a mock costing exercise. Research staff in seven academic UK Clinical Research Collaboration (CRC) registered Clinical Trials Units (CTUs) provided the staff and non-staff financial costs that they estimated were required to support an adaptive version of a trial, and a non-adaptive version of the same trial. This was undertaken for five different trial scenarios covering different types of ADs based on real trials run in the UK (see Additional file 1). The level of practical experience of the designs varied across the CTUs. The mock costing exercise was followed by a qualitative research component to understand the factors influencing the estimated resource requirements and differences between the non-adaptive and adaptive trial designs, as well as between CTUs.

Results demonstrated wide variability in the staff and non-staff resources anticipated across scenarios and CTUs, dependent, for example, on availability of core infrastructure programme funding or in-house IT systems. On average, there was a modest increase (2–4%) in resources anticipated for the AD, compared to the non-AD, within each scenario. This is consistent with comparisons using alternative methods [8]. The highest percentage increase was for statistical staff, followed by data management staff. There was inconsistency in whether additional resources for trial management staff were required across CTUs.

An important objective of the CAT project was to use results from the research to develop guidance for non-commercial organisations and researchers who design, plan, coordinate and deliver clinical trials. Here, we outline a five-step approach to aid, and potentially shorten, the time-consuming planning of adequate resourcing of adaptive trials (both staff time and non-staff costs). This approach was informed from our CAT research results that focussed on CTU resources (excluding per-patient costs). We did not explore other research costs such as intervention supply, or other methodological groups that may be involved in a trial, for example, health economists or researchers using qualitative methods. Thus, these are not fully considered in this guidance.

The process is shown in Fig. 1, with each step described in further detail below. It is predominantly aimed at academic organisations and researchers running clinical trials, but also may be relevant to other organisations and funders. Specific recommendations to funders are provided in the ‘ Guidance for funders’ section. Although we focus on ADs, the process may be useful for other innovative designs such as master protocols [9] and seamless designs (e.g. phase II/III) which are often adaptive, but not always.

Fig. 1
figure 1

Outline of process for considering and justifying resources for an adaptive design

Step 1 – What tasks are required to support the trial?

We start with the assumption that there is a proposal for a clinical trial, following the PICOS (Participants, Intervention, Comparator, Outcomes, Statistical analysis) framework [10]. Before following the process here, it is recommended to consider first whether an AD is suitable. For example, if the outcome measure on which the adaptations are based is not observed sufficiently quickly, then an AD is unlikely to provide improved utility [11].

Assuming an AD is suitable, it is helpful to first map out a recruitment strategy (including estimated sample size, number of sites, length of recruitment), a Gantt chart and the tasks that are required to support a clinical trial, regardless of whether an AD is used. We show some major tasks common to trials in Table 1 and how ADs impact on staff and non-staff resources in Table 2.

Table 1 Major tasks required to run a clinical trial and how adaptive designs may affect them
Table 2 Resources where adaptive designs increase use

For each task, it is useful to consider the workload required by broad categories of staff. The broad staff categories identified in the CAT project were trial management, data management and statistics. However, some institutions will have staff that might cross several categories (such as a statistical programmer) or be a cross-cutting role (e.g. quality assurance) and some trials will require staff from other methodological areas such as health economics not considered here. Some tasks will predominantly cause workload for one staff category (e.g. writing a Statistical Analysis Plan will be the responsibility of statistical staff with substantially less input from staff from other categories) whereas others may involve more than one (e.g. setting up and testing the randomisation system).

Some CTUs who took part in CAT described using their own costing template (also known as a budget planning tool [8]) that captured the amount of work or time required, which makes the costing process easier and more transparent. We have provided an example costing tool, implemented in a spreadsheet, in Additional file 2providing the list of tasks mentioned in Tables 1 and 2.

Some resources are also affected by the institution itself. Some CTUs participating in CAT had their own in-house randomisation and clinical data management system (CDMS); others outsourced these tasks to third parties. These differences in available infrastructure will influence staff workload (across categories) as well as non-staff costs required depending on the adaptive features considered. Some institutions had core infrastructure programme funding that will influence staff resources and non-staff costs required.

Once the trial team has a good understanding of the resource required for the proposed non-adaptive study, it is time to progress to Step 2. We would note that in some cases there may be no equivalent non-adaptive study to serve as a baseline, such as a phase I dose-finding trial, in which case this step may be more difficult.

Step 2 – How will the adaptive design features affect these tasks?

The next step in the process is considering the implications of the AD features (Table 3) on the tasks identified in Step 1 and how this influences the resources required. Clearly, this will depend on the proposed AD and adaptive features or adaptations considered.

Table 3 Additional implications of adaptive design features on resource use

We first consider elements affecting resource requirements that ADs have in common, across the life cycle of a trial: (1) whilst in set-up; (2) during recruitment and follow-up; and (3) at the time of the final analysis and reporting. We then provide some considerations for specific types of ADs.

Trial set-up

Trial set-up involves several tasks that may be affected by the AD given adaptive trials are typically more complex than non-adaptive trials, including writing the protocol, discussion with regulators, applying for regulatory and ethical approvals, development of the database, development of contracts with industry partners, design and development of randomisation systems, and setting up recruitment sites. This means an AD protocol may require more effort (including, for example, fully describing the design and its statistical properties, potentially with additional documents summarising simulations) and extra time may be needed for approvals and site set-up. Based on the experience of the authors, ethics committees and (if applicable) regulators may have queries about the design that require clarifications or amendments and resubmission; extra effort may be required to develop patient information sheets (PIS) that explain the design in a clear, concise and accurate way without giving away too much information that could potentially introduce biases in the conduct of the trial.

During recruitment and follow-up: conducting interim analyses and implementation of adaptations

A common property of ADs is that they involve one or more interim analyses whilst recruitment is ongoing and, depending on the results, implementing changes.

Interim analyses require high-quality data on the variables to be used to make adaptation decisions in a timely manner. In some cases, this may be a single outcome variable, but other designs might use a range of outcome variables. As an example, the decision-making at the interim analysis in MIDFUT, an adaptive multi-arm multi-stage trial in diabetic foot ulcer [17], involves an efficacy outcome, as well as safety data and early cost-effectiveness. Resources required for ensuring data are as accurate and complete as possible, known as data cleaning, must be done by the time of each interim analysis; any additional burden on site staff, trial managers, and data managers in the time leading up to the interim analyses should be considered.

Interim analyses also require additional statistical resource to undertake the analyses on the cleaned dataset. As well as the time to do this, additional tasks may be required. For example, an interim statistical analysis plan (SAP) would be required to unambiguously lay out the planned analysis methods in advance of data being available. This is a formal requirement for a Clinical Trial of an Investigational Medicinal Product (CTIMP), and may also require a statistician not involved in the design or conduct of the trial to prepare the interim analyses, allowing the Trial Statistician to remain blind [18] to arm allocation; in other circumstances where this is not a formal requirement, it still may be desirable to ensure that trial results do not influence the conduct of the final analysis. The time for any additional statisticians to understand the complexities of the design should also be considered. Statisticians may also be involved in the data cleaning process, such as in identifying outliers to be checked with sites. All of these factors would increase the resources required for statistical staff.

In some cases, experts in other areas might be required for the interim analysis. An example is the STOP-OHSS group sequential trial [19], assessing the clinical and cost-effectiveness of early active management of ovarian hyperstimulation syndrome compared to usual care. One interim analysis for non-binding futility early stopping is planned. A health economist was involved in the trial design by developing a preliminary model that informed the choice of an appropriate interim futility stopping rule. They will develop interim and final Health Economic Analysis Plans (HEAPs) and update the preliminary health economic model at an interim analysis [20].

After an interim analysis is completed, there may be pre-planned changes required to the sample size and trial systems. This may involve: changes to the CDMS and randomisation systems; making substantial amendments to the protocol and other trial documents and associated regulatory approvals [21, 22]; and implementing changes in trial sites. Depending on the nature of the changes required, this may increase resources required for CTU staff, or require increased costs to make changes to any outsourced systems.

Final analysis and reporting

Analysis and reporting will be influenced by the AD, adaptive features, and interim decisions made. The CONSORT extension for randomised adaptive trials [23], which provides guidance on clear reporting of all parts of the trial, is a useful resource to consider.

Overall numbers of data queries and the amount of cleaning around the time of the final analysis may be reduced due to having been brought forward to interim analyses. However, statistical analyses for ADs, especially estimation of treatment effects and related quantities such as confidence intervals, can be more complex than for traditional designs—see Robertson et al. [24] for a recent review. Other analyses, such as health economics, might also become more complex to account for the AD [25, 26].

Implications of specific features of the designs

The above considerations are applicable generally in ADs, but their impact may be affected by the specific design and its adaptive features. The number of interim analyses will clearly have an impact. The consequences of interim analyses can be split into ‘fixed costs’ and ‘variable costs’. Fixed costs are present regardless of the number of interim analyses; variable costs will increase (although not necessarily linearly) with the number of interim analyses. Examples of fixed costs would include interim SAP development; variable costs would be the statistical resource required for the interim analyses and effects of implementing changes on trial systems.

Bespoke, complex, novel ADs may require more resource increase for set-up and analysis than more commonly used ADs (such as group sequential or sample size re-estimation designs).

Step 3 – How does this affect the resources required across all possible scenarios?

Once the impact of the AD on the tasks required is mapped out, the next step is to estimate any impact on the resources required. Adaptive trials are often characterised by their flexibility, leading to uncertainty about how they will unfold. Although some ADs have a fixed sample size, most do not. It is important to consider how resources required for tasks will change if adaptations occur.

Different non-commercial organisations may have various approaches to costing which influence this. Here, we have identified some principles that we recommend are considered by those estimating resource requirements and by funders. We first consider the effects on resources that ADs have in general, and then the effects of specific ADs.

General resource impacts of ADs

First, the impact of some tasks on staff resources will likely be at different points in the trial and the timings may be hard to predict upfront. Larger organisations, with larger numbers of staff and more experience in running adaptive trials, may handle this workload planning more easily than those with smaller teams and less experience. For example, if an interim analysis requires one month’s work from a second, blinded, statistician this will be easier to accommodate if the institution has many statisticians than if it only has one. Further examples are provided in Table 4 within the context of the Graves-PCD trial.

Table 4 Graves-PCD

Second, some additional costs of running adaptive trials may be shared across other trials. Examples may be the purchase of specialised software, the development of standard operating procedures (SOPs), or staff training. These types of costs are needed for specific aspects of work; their costs may be attributed to a single trial, split across multiple trials (e.g. with costs calculated ‘per use’), or may be met internally in organisations that have core institutional support.

Third, consideration of how ADs impact National Health Service (NHS) Research Costs, Service Support Costs and Treatment Costs may be required. The informed consent process may take longer due to increased trial complexity requiring more site staff time. Treatment costs may vary due to uncertain sample size, increasing complexity in treatment cost negotiations and requiring additional trial management input.

Impact of specific ADs

A common AD incorporates pre-specified criteria for early trial stopping due to lack-of-benefit. If the trial is stopped early this will have an impact on the project duration and resources required to recruit participants. This may lead to a reduction in costs compared to the trial continuing, without cost to the statistical properties of the study. We note that stopping for lack-of-benefit does not immediately bring a trial to an end: following up those who have already been enrolled is likely to continue alongside the closure of trial sites and undertaking a final analysis on all outcomes in order to disseminate the results. Therefore, the remaining staff resource required may be reduced but is not removed.

In other ADs, such as multi-arm multi-stage (MAMS) designs, recruitment to the overall trial may not stop early but recruitment to individual arms may. Until an interim analysis is performed it will remain unknown which, and in some cases, how many arms will continue. There are some tasks required for stopping recruitment to an arm, such as: amending the randomisation system; implementing changes to the PIS and at individual sites [21, 22]; and conducting final analyses for closed arms. There may also be an impact on the time needed to recruit remaining participants if the sample size is specified per arm. Early stopping of arms or the trial may lead to a reduction in overall costs required without compromising the trial’s integrity and validity.

Other ADs, like sample size re-estimation, may potentially increase the target sample size, and therefore trial duration. The change in resources required in this circumstance will depend on how this influences the tasks required. All CTUs who took part in the CAT mock costing exercise provided estimated resources based on the maximum target sample size, and the time needed to recruit to this. This is useful to inform the maximum resource needs. However, one must consider all potential scenarios that can occur, as well as the likelihood of each scenario, and how this will influence the resources required by the trial. In the more theoretic ADs literature, it has been a rule of thumb that quantities such as the ‘average sample size’ (i.e. the sample sizes associated with all possible scenarios, weighted by probabilities of each scenario occurring) can be a good representation of the efficiency of an AD. These metrics are based on multiple hypothetical trials and are not necessarily helpful for the specific trial being costed. They also do not take into account the fixed costs of starting and stopping the trial and the need to have funds available to cover all possible scenarios.

For designs that may lead to a high variation in the resources required, the host institution must carefully consider the implications of this. As in Step 2, larger organisations with many projects and staff may be better able to deal with uncertainty than smaller organisations.

It is important to remember that uncertainty is not unique to ADs. In most trials, there is uncertainty around feasibility aspects such as recruitment rates, and whether specified milestones will be met (e.g. as monitored in the internal pilot phase of a trial [29]). Most non-adaptive trials can also be stopped early, e.g. due to safety concerns.

Step 4 – What are the funder expectations?

Currently, most funders of academic or public sector trials (in the UK) provide limited flexibility in specifying the costs requested in a funding application. A single figure is required which is considered by the funding panel deciding on whether there is value for money provided by the research. This figure would generally be the maximum required by the trial. If an adaptive trial is highly variable in its cost (e.g. depending on how many arms stop early, or whether the recruitment target increases following a sample size re-estimation) then this maximum may make the research look expensive, but would minimise the risk of needing to return to a funder for additional resources to complete a trial, and also minimise the risk of an inconclusive trial result. In the CAT project, we found that CTUs presented this maximum amount to ensure the research could be delivered in the ‘worst-case scenario’ that requires the maximum sample size.

There should be opportunities to insert more details in the application form to provide an estimate of cost savings if a particular change happens. Most funders have a justification of costs section, which would allow including estimation of cost savings in certain scenarios. In addition, some funders of trials (e.g. the Medical Research Council (MRC) Developmental Pathway Funding Scheme, DPFS [28]) currently require projects to be split into milestones, each with an associated cost. This may allow better specification of the uncertainty in the cost of an AD if milestones are linked to interim analyses and encourage organisations to plan ahead. This last approach has some limitations, such as not allowing alternative paths of milestones depending on the results of an early milestone.

An alternative approach is that funders allow requesting funded extensions of research or variations to contracts. In our experience with UK funders, this has not been commonly encouraged as a way to handle uncertainty in ADs and does not provide certainty of funding in the worst-case scenario. It may, however, be a useful way to handle sample size re-estimation designs or platform trials that can add in new arms.

It is vital to bear in mind the funder requirements and flexibility of the application form when deciding how to best present the uncertainty of the resources required.

Step 5 – Justifying and refining the design

Considering Steps 2–4 will determine how the design influences the resources required and how this can be communicated to the funder. It is important then to consider how different aspects of the design could be modified to retain most or all of the benefits whilst minimising any statistical or resource issues that arise. This may include considering the number of interim analyses, reconsidering the types of adaptations that are implemented, and the specific decision rules that are specified. As well as considering the resources required by the trial design, it is important to consider the quality of evidence provided by the trial also.

The methods of the value of information analysis (VOIA) may be useful to help consider the benefits and the associated costs of collecting more information to inform (and reduce the uncertainty of) a technology adoption decision [30, 31]. Currently, there is limited use of this approach in practice in the context of ADs [32] as well as non-adaptive designs [33]. However, VOIA methods can be used to quantify the value of non-ADs and alternative ADs allowing the comparison of multiple designs. This can help the research team to refine and justify their chosen design [34, 35]. A clear understanding of the costs of the trial is required for these analyses, which is facilitated by following Steps 1–4.

Example

Table 4 presents an example of how a funded adaptive trial could have followed the five-step process.

Guidance for funders

Funders of clinical trials benefit from the appropriate use of ADs as they provide higher efficiency and more robust evidence that ultimately benefits patients. In our experience, some funders have been more encouraging than others in the use of innovative designs [36]. Nevertheless, there are some barriers imposed that may stifle methodological innovation; some of these were raised earlier in this article.

Currently, some aspects of public funding of trials may penalise innovative designs. By only allowing limited space for specifying the requested funding, it may be necessary to focus on the maximum ‘worst-case scenario’ cost which may make the trial look expensive. Having distinct funding panels that typically focus on a particular phase of trial makes getting funding for seamless designs, spanning multiple phases, difficult. Funding agreements that penalise investigators for stopping a trial early because they have answered the research question quicker (i.e. by completely cutting off funding) make some efficient designs unappealing to academic organisations. Furthermore, the additional resources required pre-funding application are usually unfunded, often making organisations reliant on core funding when exploring an adaptive trial design. Funders could promote more use of ADs through making infrastructure awards that would allow further capacity to develop them.

There are some ways in which funders could encourage more innovation by allowing more flexibility in their awards and application processes (Table 5).

Table 5 Recommendations to funders to encourage increased appropriate use of innovative designs

We would also advise funders, in collaboration with applicants, to consider whether overly complex ADs could be simplified (with resource savings) without loss of benefit [11]. Conversely, funders should encourage applicants to add adaptive elements if they would be beneficial to information or patients.

We would like to highlight that several funders have made great progress in addressing barriers to the wider use of innovative designs. It is also understandable that some funders operate under considerable constraints (e.g. annual budgets that cannot be carried forward) that make it more difficult to address some barriers.

Discussion

Once there is a compelling reason that an AD brings better and quicker evidence, resourcing and justifying it is a time-consuming process. Our five-step approach outlined here provides structure to the process. This complements literature covering the process of designing adaptive trials such as [3, 4, 38], addresses barriers raised in the implementation of ADs [39, 40], resourcing of clinical trials [41,42,43] and investigation of additional resources required to support adaptive trials [7, 8].

We have restricted attention to resourcing the trial after it is funded. Like any clinical trial, the process of designing an adaptive trial requires a substantial amount of input from a multidisciplinary team. However, more specialist expertise and a greater amount of time is generally required for an AD. This is difficult to resource other than from institutional core funding for trial development infrastructure. Some funders may offer development grants which would help cover this additional work upfront.

We would emphasise that ADs can provide many benefits that may outweigh cost considerations. These include lower average time taken to complete the trial, better outcomes for participants recruited to the trial, and higher-quality evidence provided by the trial. In addition, the apparent higher costs of supporting an AD may just be a ‘worst-case-scenario’ and be offset by substantially reduced costs if the trial finishes early. The only work we are aware of that investigated differences in cost between adaptive and non-adaptive trials in practice is Martin et al. [8], which investigated several different cost-drivers using regression models. The authors did not find a significant cost difference between adaptive and non-adaptive designs although it is likely to be difficult to estimate reliably as AD use may be different by phase and indication.

The key factor that justifies an AD is the ratio of benefit it provides (both to the efficiency and patient benefit of the trial itself and the long-run impact of the evidence generated) to the additional cost it incurs [44, 45]. Research that can inform and improve both parts of this ratio is needed. Trial methods that can maintain or improve the speed and quality of interim analyses whilst reducing the resources required would be very useful; methods that may improve the benefit provided without requiring more resources would similarly be of great interest. Overall, a framework for better quantifying the benefit of an AD in the presence of real-world issues such as delay in assessment of outcomes [11] would help justify this to funders.

ADs provide advantages and complexities for other types of analyses too. In our CAT research, and this guidance, we have not highlighted areas such as health economics, qualitative research and evidence synthesis. Previous work, e.g. Flight et al. [32] has noted the impact of ADs on health economic considerations. However, further work is needed for investigating how ADs affect the resources required for health economists and other methodological disciplines. Additional resources may be required for the design of the adaptive trial, contributing to interim decision making and appropriately analysing the final data to account for the AD.

As the recommendations in this paper are based on investigation of UK academic CTUs, we acknowledge they are most relevant to academic trials run in the UK. Co-authors of this paper have experience with international academic funders including the European Commission, the Deutsche Forschungsgeneinschaft (DFG, Germany), National Institutes of Health (NIH, USA), National Health and Medical Research Council (NHMRC, Australia) and Fight Kids Cancer (Belgium) suggesting that the issues identified here are very similar. With trials funded by large international pharmaceutical companies, some of the guidance will be relevant but it is likely that there is much more of a focus on average cost, power of trials, and portfolio optimisation. We would welcome further papers that consider how our recommendations may be best tailored to fit trials outside of the UK and run by industry including smaller companies.

The evolution of application forms by funders that would enable a rationale for costs to sit alongside the design choices, paired with greater flexibility in the way costs are presented to funders, could improve transparency whilst enabling the many benefits of innovation in trial design to be realised more broadly in clinical research through the funding of more trials using ADs. It is also important that innovation in trial designs that can lead to improved efficiency, quality of evidence, and patient benefit are incentivised by funders.

Although we have concentrated on ADs, the process could be used for other innovative approaches. For example, master protocols [9] (including basket trials and umbrella trials, and platform and living protocols) are not necessarily adaptive but may require similar considerations of appropriately resourcing them. Platform trials, which offer the opportunity to add in new arms, may require considering the additional costs incurred from the additional arm compared with the fixed costs of the platform.

Through better guidance on appropriately resourcing ADs, we hope that their use can continue to increase, which will play an important role in improving patient outcomes and improving research productivity.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Abbreviations

AD:

Adaptive design

CAT:

Costing Adaptive Trials

CDMS:

Clinical Data Management System

CRC:

Clinical Research Collaboration

CTIMP:

Clinical Trial of Investigational Medicinal Product

CTU:

Clinical Trials Unit

DFG:

Deutsche Forschungsgemeinschaft

DPFS:

Developmental Pathway Funding Scheme

HEAP:

Health Economic Analysis Plan

IMP:

Investigational Medicinal Product

MAMS:

Multi-arm multi-stage

MRC:

Medical Research Council

NHMRC:

National Health and Medical Research Council

NHS:

National Health Service

NIH:

National Institutes of Health

NIHR:

National Institute for Health Research

PIS:

Patient Information Sheet

SAP:

Statistical analysis plan

SOP:

Standard operating procedure

VOIA:

Value of information analysis

References

  1. DiMasi JA, Grabowski HG, Hansen RW. Innovation in the pharmaceutical industry: New estimates of R&D costs. J Health Econ. 2016;47:20–33.

    Article  Google Scholar 

  2. Bentley C, Cressman S, van der Hoek K, Arts K, Dancey J, Peacock S. Conducting clinical trials—costs, impacts, and the value of clinical trials networks: A scoping review. Clin Trials. 2019;16:183–93.

    Article  Google Scholar 

  3. Pallmann P, Bedding AW, Choodari-Oskooei B, Dimairo M, Flight L, Hampson LV, et al. Adaptive designs in clinical trials: Why use them, and how to run and report them. BMC Med. 2018. https://doi.org/10.1186/s12916-018-1017-7.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Burnett T, Mozgunov P, Pallmann P, Villar SS, Wheeler GM, Jaki T. Adding flexibility to clinical trial designs: an example-based guide to the practical use of adaptive designs. BMC Med. 2020;18:352.

    Article  Google Scholar 

  5. Gallo P. Operational challenges in adaptive design implementation. Pharm Stat. 2006;5:119–24.

    Article  Google Scholar 

  6. Gallo P. Confidentiality and Trial Integrity Issues for Adaptive Designs. Drug Inf J. 2006;40:445–50.

    Article  Google Scholar 

  7. Wilson N, Biggs K, Bowden S, Brown J, Dimairo M, Flight L, et al. Costs and staffing resource requirements for adaptive clinical trials: quantitative and qualitative results from the Costing Adaptive Trials project. BMC Med. 2021;19:251.

    Article  Google Scholar 

  8. Martin L, Hutchens M, Hawkins C, Radnov A. How much do clinical trials cost? Nat Rev Drug Discov. 2017;16:381–2.

    CAS  Article  Google Scholar 

  9. Park JJH, Siden E, Zoratti MJ, Dron L, Harari O, Singer J, et al. Systematic review of basket trials, umbrella trials, and platform trials: a landscape analysis of master protocols. Trials. 2019;20:572.

    Article  Google Scholar 

  10. Saaiq M, Ashraf B. Modifying, “Pico” Question into “Picos” Model for More Robust and Reproducible Presentation of the Methodology Employed in A Scientific Study. World J Plast Surg. 2017;6:390–2.

    PubMed  PubMed Central  Google Scholar 

  11. Wason JMS, Brocklehurst P, Yap C. When to keep it simple – adaptive designs are not always useful. BMC Med. 2019;17:152.

    Article  Google Scholar 

  12. Wason JMS, Trippa L. A comparison of Bayesian adaptive randomization and multi-stage designs for multi-arm clinical trials. Stat Med. 2014;33:2206–21.

    Article  Google Scholar 

  13. Grieve AP. Response-adaptive clinical trials: case studies in the medical literature. Pharm Stat. 2017;16:64–86.

    Article  Google Scholar 

  14. Wason JMS, Jaki T. Optimal design of multi-arm multi-stage trials. Stat Med. 2012;31:4269–79.

    Article  Google Scholar 

  15. Wason J, Stallard N, Bowden J, Jennison C. A multi-stage drop-the-losers design for multi-arm clinical trials. Stat Methods Med Res. 2017;26:508–24.

    Article  Google Scholar 

  16. Park JJH, Harari O, Dron L, Lester RT, Thorlund K, Mills EJ. An overview of platform trials with a checklist for clinical readers. J Clin Epidemiol. 2020;125:1–8.

    Article  Google Scholar 

  17. Brown S, Nixon J, Ransom M, Gilberts R, Dewhirst N, McGinnis E, et al. Multiple Interventions for Diabetic Foot Ulcer Treatment Trial (MIDFUT): study protocol for a randomised controlled trial. BMJ Open. 2020;10: e035947.

    Article  Google Scholar 

  18. Day SJ. Statistics Notes: Blinding in clinical trials and other studies. BMJ. 2000;321:504–504.

    CAS  Article  Google Scholar 

  19. STOP-OHSS. https://fundingawards.nihr.ac.uk/award/NIHR128137. Accessed 6 Jan 2022.

  20. Thorn JC, Davies CF, Brookes ST, Noble SM, Dritsaki M, Gray E, et al. Content of Health Economics Analysis Plans (HEAPs) for Trial-Based Economic Evaluations: Expert Delphi Consensus Survey. Value Health. 2021;24:539–47.

    Article  Google Scholar 

  21. Schiavone F, Bathia R, Letchemanan K, Masters L, Amos C, Bara A, et al. This is a platform alteration: A trial management perspective on the operational aspects of adaptive and platform and umbrella protocols. Trials. 2019;20:264.

    Article  Google Scholar 

  22. Hague D, Townsend S, Masters L, Rauchenberger M, Van Looy N, Diaz-Montana C, et al. Changing platforms without stopping the train: experiences of data management and data management systems when adapting platform protocols by adding and closing comparisons. Trials. 2019;20:294.

    Article  Google Scholar 

  23. Dimairo M, Pallmann P, Wason J, Todd S, Jaki T, Julious SA, et al. The Adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design. BMJ. 2020;369: m115.

    Article  Google Scholar 

  24. Robertson DS, Choodari-Oskooei B, Dimairo M, Flight L, Pallmann P, Jaki T. Point estimation for adaptive trial designs. ArXiv210508836 Stat. 2021.

  25. Flight L, Julious S, Brennan A, Todd S, Hind D. How can health economics be used in the design and analysis of adaptive clinical trials? A qualitative analysis. Trials. 2020;21:252.

    Article  Google Scholar 

  26. Laura Flight. The use of health economics in the design and analysis of adaptive clinical trials (PhD Thesis). 2020.

  27. Macdougall J. Analysis of Dose--Response Studies---Emax Model. In: Dose Finding in Drug Development. New York: Springer New York; 2006. p. 127–45.

  28. Medical Research Council MRC. Biomedical Catalyst: Developmental Pathway Funding Scheme (DPFS) submission deadlines. 2021. https://mrc.ukri.org/funding/browse/biomedical-catalyst-dpfs/biomedical-catalyst-developmental-pathway-funding-scheme-dpfs-submission-deadlines/. Accessed 30 Jun 2021.

  29. Herbert E, Julious SA, Goodacre S. Progression criteria in trials with an internal pilot: an audit of publicly funded randomised controlled trials. Trials. 2019;20:493.

    Article  Google Scholar 

  30. Fenwick E, Steuten L, Knies S, Ghabri S, Basu A, Murray JF, et al. Value of Information Analysis for Research Decisions—An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value Health. 2020;23:139–50.

    Article  Google Scholar 

  31. Rothery C, Strong M, Koffijberg H (Erik), Basu A, Ghabri S, Knies S, et al. Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value Health. 2020;23:277–86.

  32. Flight L, Arshad F, Barnsley R, Patel K, Julious S, Brennan A, et al. A Review of Clinical Trials With an Adaptive Design and Health Economic Analysis. Value Health J Int Soc Pharmacoecon Outcomes Res. 2019;22:391–8.

    Article  Google Scholar 

  33. Mohiuddin S, Fenwick E, Payne K. Use of value of information in UK health technology assessments. Int J Technol Assess Health Care. 2014;30:553–70.

    Article  Google Scholar 

  34. Chick S, Forster M, Pertile P. A Bayesian decision theoretic model of sequential experimentation with delayed response. J R Stat Soc Ser B Stat Methodol. 2017;79:1439–62.

    Article  Google Scholar 

  35. Flight L, Julious S, Brennan A, Todd S. Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials. Med Decis Making. 2021;:0272989X2110450.

  36. Blagden SP, Billingham L, Brown LC, Buckland SW, Cooper AM, Ellis S, et al. Effective delivery of Complex Innovative Design (CID) cancer trials—a consensus statement. Br J Cancer. 2020;122:473–82.

    Article  Google Scholar 

  37. Studies within a trial (SWAT). https://www.nihr.ac.uk/documents/studies-within-a-trial-swat/21512. Accessed 5 Oct 2021.

  38. Thorlund K, Haggstrom J, Park JJ, Mills EJ. Key design considerations for adaptive clinical trials: a primer for clinicians. BMJ. 2018;360: k698.

    Article  Google Scholar 

  39. Jaki T. Uptake of novel statistical methods for early-phase clinical studies in the UK public sector. Clin Trials. 2013;10:344–6.

    Article  Google Scholar 

  40. Dimairo M, Boote J, Julious SA, Nicholl JP, Todd S. Missing steps in a staircase: A qualitative study of the perspectives of key stakeholders on the use of adaptive designs in confirmatory trials. Trials. 2015;16:1–16.

    Article  Google Scholar 

  41. Speich B, Gloy V, Schur N, Ewald H, Hemkens LG, Schwenkglenks M, et al. A scoping review shows that several nonvalidated budget planning tools for randomized trials are available. J Clin Epidemiol. 2020;117:9–19.

    Article  Google Scholar 

  42. Speich B, von Niederhäusern B, Blum CA, Keiser J, Schur N, Fürst T, et al. Retrospective assessment of resource use and costs in two investigator-initiated randomized trials exemplified a comprehensive cost item list. J Clin Epidemiol. 2018;96:73–83.

    Article  Google Scholar 

  43. Speich B, von Niederhäusern B, Schur N, Hemkens LG, Fürst T, Bhatnagar N, et al. Systematic review on costs and resource use of randomized clinical trials shows a lack of transparent and comprehensive data. J Clin Epidemiol. 2018;96:1–11.

    Article  Google Scholar 

  44. Emerson SC, Rudser KD, Emerson SS. Exploring the benefits of adaptive sequential designs in time-to-event endpoint settings. Stat Med. 2011;30:1199–217.

    Article  Google Scholar 

  45. Mahlich J, Bartol A, Dheban S. Can adaptive clinical trials help to solve the productivity crisis of the pharmaceutical industry? - a scenario analysis. Health Econ Rev. 2021;11:4.

    Article  Google Scholar 

Download references

Acknowledgements

We thank individuals who contributed to the mock costing exercise. We thank Elizabeth Pilling for useful comments.

Funding

This report is independent research supported by the National Institute for Health Research (NIHR130351). JMSW is funded by a NIHR Research Professorship (NIHR301614). The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care (DHSC).

National Institute for Health Research, NIHR130351, NIHR301614, James Wason, BRC-1215-20014, Thomas Jaki, Medical Research Council, MC_UU_00002/14, Thomas Jaki, MC_UU_00002/15,Sofia Villar, MC_UU_00004/08, Matthew Sydes.

The Centre for Trials Research, Cardiff University receives infrastructure funding from Health and Care Research Wales and Cancer Research UK. CJW was supported in this work by NHS Lothian via Edinburgh Clinical Trials Unit. TJ received funding from UK Medical Research Council (MC_UU_00002/14). SSV received funding from UK Medical Research Council (MC_UU_00002/15). This research was supported by the NIHR Cambridge Biomedical Research Centre (BRC-1215–20,014). The Leeds Institute of Clinical Trials Research receives infrastructure funding from the NIHR and CRUK.

The Cancer Research UK Clinical Trials Unit (CRCTU) receive infrastructure funding from Cancer Research UK.

MRS is supported for this work through the Medical Research Council (MC_UU_00004/08).

Sheffield Clinical Trials Research Unit receives infrastructure funding from the NIHR.

Newcastle Clinical Trials Unit receives infrastructure funding from the NIHR.

ICR-CTSU is in receipt of a core programme grant from CRUK (C1491/A25351).

Author information

Authors and Affiliations

Authors

Contributions

JW, MD, HH and RM led the drafting of the manuscript. KB, SB, JB, LF, JH, TJ, RL, PP, MP, CS, MS, SV, CW, NW and CY provided critical review and rewrote sections of the manuscript. The authors read and approved the final manuscript.

Corresponding author

Correspondence to James M. S. Wason.

Ethics declarations

Ethics approval and consent to participate

This research received an ethical waiver by Newcastle University’s ethics committee due to it not involving patients. Consent was obtained for all participants who took part in the interviews.

Consent for publication

Not applicable.

Competing interests

SSV reports family member employment by a commercial organisation that could have an interest in the content of the manuscript. No other relevant conflicts were reported by any author.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Brief overview of each scenario used in the Costing Adaptive Trials mock costing exercise.

Additional file 2.

Costing spreadsheet.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wason, J.M.S., Dimairo, M., Biggs, K. et al. Practical guidance for planning resources required to support publicly-funded adaptive clinical trials. BMC Med 20, 254 (2022). https://doi.org/10.1186/s12916-022-02445-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12916-022-02445-7

Keywords

  • Adaptive designs
  • Adaptive clinical trials
  • Clinical trials
  • Efficiency
  • Resource requirements
  • Trial coordination