Statistical documentation for multi-disease, multi-domain platform trials: our experience with the Staphylococcus aureus Network Adaptive Platform trial

Abstract

Platform trials have become widely adopted across multiple disease areas over recent years, however, guidelines for operationalising these trials have not kept pace. We outline a series of documents that summarise the statistical components, and implicit processes, of the Staphylococcus aureus Network Adaptive Platform (SNAP) trial to provide an informal template for other researchers and reviewers of platform trials. We briefly summarise the content and role of the core protocol, statistical appendix, domain-specific appendices, simulation report, statistical implementation guides, data safety and monitoring committee (DSMC) reports, and domain-specific statistical analysis plans and final reports, and a transparent governance structure that ensures separate blinded and unblinded statistical teams. In the absence of guidelines or checklists for platform trial statistical documents, we hope to provide useful guidance to others in terms of what has worked so far for the SNAP trial, stimulate discussion, and inform a future consensus.

Trial registration NCT05137119. Registered on 30 November 2021.

Background

Platform trials have been widely adopted across multiple disease areas in recent years [1]. Broadly speaking, a platform trial is a randomised clinical study that can evaluate multiple therapeutic interventions simultaneously, under a single core (or ‘master’) protocol, with the flexibility to add or remove interventions over time. Although first proposed in the mid-2000s, only 16 platform trials existed by 2019 [1]. That number rose rapidly to at least 98 by 2022 [2], with the COVID-19 pandemic undoubtedly serving as a catalyst for growth. Efficiency, speed, treatment personalisation, and economies of scale are reasons why platform trials have become so widely used [3].

Platform trials are complicated, making their design, implementation, and analysis challenging. In addition to the usual clinical trial requirements, platform trials require investigators to justify the number, type, and timing of statistical analyses and associated decision rules; the grouping of interventions into domains; and disease strata (where relevant). Investigators must also justify the overall adaptive strategy, e.g. the appropriate response adaptive randomisation algorithm [4], early stopping criteria [5], when to introduce new interventions [6], and choosing strategies to deal with non-concurrently randomised cohorts [7].

In any clinical trial, ensuring prespecified, detailed, comprehensive, and, where practical, openly available statistical documentation is key to holding investigators to account and promoting trial integrity. Given platform trials are complicated designs that may change over time in ways that cannot be foreseen at trial initiation, documentation must be both comprehensive and structured to minimise the maintenance burden over time.

In the spirit of transparency, and to provide an informal template for other researchers considering platform trial designs, here we summarise how the statistical components of the Staphylococcus aureus Network Adaptive Platform (SNAP) trial are documented and operationalised.

The SNAP trial

The SNAP trial is a ‘whole-of-life’ multi-disease, multi-domain adaptive comparative effectiveness platform trial, and includes multiple different interventions nested within different treatment modalities (i.e., ‘domains’), disease strata [8, 9], and incorporates all age ranges across both paediatric and adult populations [10]. Note that many aspects of the SNAP trial approach have been pioneered by similar trials, in particular, the Randomised Embedded Multifactorial Adaptive Platform for Community-Acquired Pneumonia (REMAP-CAP) [11] and the Australasian COVID-19 Trial Adaptive Platform Trial (ASCOT ADAPT) [12].

Key statistical documentation

Table 1 outlines the purpose, producers, and primary intended audience of key statistical documents. To support platform trial implementation, the governance structure requires separate blinded and unblinded statistical teams, known respectively as the statistical committee and the analytic team. The statistical committee consists of investigators and independent advisory members who are responsible for directing the ongoing blinded trial design and analysis strategies, whereas the analytic team consists of data managers, statistical programmers, and statisticians who are responsible for the scheduled analyses, evaluation of adaptation decision criteria and closed (unblinded) reporting to the data safety and monitoring committee (DSMC). Maintaining the confidentiality of the closed report between only the members of the DSMC and analytic team is essential to the integrity of the trial.

Table 1 Key statistical documents

Figure 1 outlines the information flow for a hypothetical realisation of the SNAP trial. This hypothetical realisation is illustrated at its third scheduled analysis, having previously satisfied a decision rule at the second scheduled analysis leading to an additional analysis to support a domain-specific conclusion.

Fig. 1

Flow of statistical information in the Staphylococcus aureus Network Adaptive Platform trial. This hypothetical realisation is at its third scheduled analysis, having previously satisfied a decision rule at the second scheduled analysis leading to analyses to support a domain-specific conclusion

Note: the flow of information also approximates the chronological order that the documents are produced. SAP, statistical analysis plan; SIG, statistical implementation guide; DSMC, data safety and monitoring committee. * Simulation report may be updated upon trial adaptation. † Additional domain-specific appendices may be added where required. Documents produced solely by the blinded team are unshaded, documents solely produced by the unblinded analytic team are shaded grey

For simple clinical trials, a single comprehensive statistical analysis plan (SAP) is a key pre-emptive strategy to ensure that there is transparency and clarity in the definition of estimands, planned analyses, ordering of endpoints, and hypothesis testing. The SAP aims to avoid ambiguity and deviations, intentional or otherwise, from the intended design and analyses. Because platform trials are complicated there is greater potential for the integrity of data analysis to be compromised, or simply for the original intent to be unclear; therefore, additional statistical documents are required that go beyond typical SAPs. By taking a modular approach to documentation and avoiding duplication of information across documents, the potentially daunting task of maintaining the accuracy and internal consistency of the trial documentation over time can be greatly facilitated.

The key statistical documents used to operationalise the SNAP trial are the core protocol, statistical appendix, domain-specific appendices, simulation report, statistical implementation guide, data safety and monitoring committee (DSMC) reports, domain-specific SAPs, and domain-specific final reports. It is important to note that statistical documentation that contributes to, or defines criteria for changes to, the trial design is developed by the blinded team (namely the statistical committee and other blinded investigators) without direct input from the unblinded team (namely the unblinded analytic team and DSMC), unless otherwise described in their respective charters, to ensure that the design is not changed to favour, for example, a particular treatment or population (see Table 1 for detail). For transparency, apart from the DSMC report and domain-specific final reports, all these key statistical documents are readily available on the SNAP website [13]. Many of the same documents have also been published in peer-reviewed journals including the core protocol [8], domain-specific and paediatric appendices [10, 14, 15], and the combined statistical appendix and simulation report [9].

The core protocol details the central aims of a trial along with core trial eligibility criteria, endpoints, decision rules, and trial governance structures that will be used throughout the trial and across different domains. The core protocol is complemented by domain-specific appendices, which detail domain-specific eligibility, interventions, endpoints, and analysis or design considerations. New domain-specific appendices can be added as new domains are added, and existing domain-specific appendices can be updated if additional interventions are added within a domain. Statistical content within the core protocol and domain-specific appendices is broadly descriptive to allow the reader to understand the underlying statistical concepts without being unnecessarily burdened by technical detail. The core protocol and domain-specific appendices are written and maintained by the blinded investigators with contributions from the blinded statistical committee. Note that a related concept is a population-specific appendix, for example in SNAP there are appendices related to paediatric populations or pregnant women, that fulfil a similar role to domain-specific appendices. For simplicity, in this commentary, we can safely assume that what applies to domain-specific appendices also applies to population-specific appendices.

For SNAP, the dedicated statistical appendix to the core protocol is the foundational statistical document and encapsulates the statistical models and features underpinning the platform trial with the requisite technical detail. The SNAP trial’s statistical appendix is similar to a traditional SAP. For example, both the statistical appendix and a typical SAP document have the same items recommended by current guidelines [16] and require a priori specification and extensive internal and, ideally, external review. However, the SNAP statistical appendix is not interchangeable with a traditional SAP. Instead, the appendix describes the trial design and analysis strategy in statistical generality with the aim of futureproofing the SNAP trial with respect to downstream adaptations (e.g., adding a new domain) and serves as a blueprint from which to develop multiple, often domain- or intervention-specific, analysis plans [9]. The statistical appendix is produced by the blinded statistical committee and requires both trial governance and ethics approval, including for any amendments.

The design choices specified in the statistical appendix are supported by an extensive simulation report. The simulation report documents the trial design assumptions and the trial operating characteristics under a range of plausible scenarios. For example, a scenario of identical lack of response for all interventions across strata (i.e. ‘no effect’) and multiple scenarios describing different degrees of homogenous effects, as well as some unanticipated scenarios, such as heterogenous effects across strata, sometimes in different directions. The simulation report provides justification for the platform design and decision rule criteria, including the maximum and expected sample sizes for anticipated scenarios. The simulation report is regularly updated to account for changing circumstances, including but not limited to the inclusion of new domains, interventions, or population subgroups; unexpected recruitment rates overall or by subgroup; different than simulated response rates in the control group; and higher than expected loss to follow-up or protocol deviations. The simulation report is produced and updated by the blinded statistical committee.

Specific guidance for scheduled analyses is provided within a statistical implementation guide which incorporates elements of the statistical appendix and the, possibly updated, simulation report. The statistical implementation guide is produced, and continually updated, by the blinded statistical committee to provide specific instructions to the unblinded analytic team for a scheduled analysis and inform them of the current trial design such as the inclusion of new domains or treatments. It provides the specific form of the models and decision rules that must be used for an upcoming scheduled (but not terminal) analysis. Where the analysis is a terminal analysis to support a domain-specific conclusion, a domain-specific SAP is used (see next paragraph). Scheduled efficacy analyses are often restricted to the primary estimand, but each analysis may also include important secondary estimands related to safety outcomes. Unblinded results from analyses specified within the statistical implementation guides are produced by the unblinded analytic team and summarised within the ‘closed’ DSMC report. In SNAP, investigators receive an ‘open’ DSMC report (also produced by the unblinded analytic team) that maintains blinding by aggregating data summaries over treatments and strata and excluding efficacy results.

A domain-specific SAP is used to guide the terminal analysis of a particular domain, for example after a scheduled analysis satisfies a domain-specific decision rule to cease randomising participants to the domain-specific interventions. The domain-specific SAP is effectively the same as a traditional SAP in that it provides a complete a priori statistical strategy for analysing all relevant trial data including core and domain-specific estimands (in the SNAP trial, for a particular trial domain). Domain-specific SAPs are developed by the blinded statistical committee prior to unblinding the research team to domain-specific outcomes (noting that the analytic team are already unblinded) and will be made publicly available before their implementation. Unblinded results from the analyses specified in the domain-specific SAP, including unblinded efficacy data, are summarised in a domain-specific final report. To maintain the blinded status of the other trial domains, the results may be aggregated, or the models marginalised across interventions in other domains. The domain-specific final report is produced by the unblinded analytic team. Results of a domain-specific final report are provided to the DSMC for additional information concerning secondary safety and efficacy estimands and summarised for publication.

Ancillary statistical documentation

The key statistical documents are also complemented by a suite of ancillary documents that are essential for transparency, governance, and implementation. These ancillary documents generally support statistical trial integrity and define logistics and processes around data transfer, storage, external requests, and confidentiality. Statistical components of trial integrity are supported by a data management plan detailing data management procedures and related responsibilities; a data sharing policy detailing procedures for data sharing and availability both within and across regions; terms of reference outlining membership, responsibilities, and communication channels for each of the statistical committee, unblinded analytic team, DSMC, and other trial committees and management groups. The data management plan, data sharing policy, and other relevant ancillary documents are made publicly available [13].

A note on statistical software

All statistical code for the scheduled analyses is encapsulated in R packages that are hosted on Github, an online software development platform, to ensure version control and seamless collaboration among members of the analytic team. All analysis code and unblinded outputs are reviewed by at least one member of the analytic team (in addition to the primary analyst) prior to reporting results in the open and closed DSMC reports and domain-specific final reports. To prevent inadvertently unblinding the ongoing trial, potentially through comments in the code or the temporary implementation of assumptions about the data to resolve issues while data collection is ongoing, only current members of the analytic team have access to working code and the associated Github repositories. Upon publication of domain-specific results, the R package associated with the domain-specific analysis will also be published to ensure software transparency. The code used to produce the simulation report was developed by Berry Consultants, a commercial consultancy specialising in platform trial design, and is therefore not publicly available.

Conclusions

To publish trial results, most journals require a comprehensive and prespecified SAP that is congruent with current guidelines [16]. For platform trials, a consensus on what is required to document platform trial designs (and the nomenclature for said documentation) does not yet exist. Given the complexity and design heterogeneity of most platform trials, this lack of consensus may lead to confusion and inconsistency in the structure and effectiveness of documentation.

In the absence of a guideline or checklist for platform trial statistical documents, such as those which currently exist for traditional SAPs, we hope that this commentary provides a useful guide to others in terms of what has worked so far for the SNAP trial and stimulates discussion around the documentation requirements of platform trials that may inform a future consensus.