Clinical Study Reports—a systematic review with thematic synthesis: Part 1. History, contents and structure, definitions, and terminology

Abstract

Background

Clinical study reports (CSRs) are standardized full reports of the protocols, results, and other pertinent details of clinical studies that are typically submitted by pharmaceutical companies to regulatory authorities, as part of the drug approval process. Their recommended contents and structure were described in 1995 in a document of the International Conference on Harmonisation, ICH E3, although companies can choose how to present the data. Until 2015, such reports were not readily available to the public, but since then some regulatory authorities have made them available, as have some pharmaceutical companies, albeit often in abbreviated or redacted versions. The apparent benefits of pharmacological interventions are not as impressive when they are calculated using data from clinical study reports compared with published trial reports, and more information emerges about harms the interventions can cause.

Results

Our methods are described in Part 2 of this systematic review with thematic synthesis, in which we have summarized the uses of CSRs, as described in 349 publications of various sorts, including analyses of clinical trials, data analyses, commentaries, and official documents. We have specifically concentrated on how CSRs affect assessments of benefits, harms, and the benefit-to-harm balance, and other factors that affect it. In Part 1, we discuss the history of the development of CSRs, their contents and structure, definitions of CSRs and qualifying terms, and relevant terminology (including the availability of CSRs, data sharing systems, and transparency and confidentiality).

Conclusions

Our conclusions are listed in Part 2 of this review.

Background

As users of clinical study reports (CSRs), which provide more comprehensive data on clinical trial protocols and results than publicly available accounts of clinical trials, as published in peer-reviewed journals [1, 2], we are interested to know how often they have been and are being used for research, concentrating on systematic reviews and meta-analyses of the benefits, harms, and benefit-harm balance of pharmacological interventions. We are also interested in knowing about factors that affect the benefit-harm balance, including cost-effectiveness, adherence, the quality of CSRs, and prepublished protocols. We have therefore carried out a systematic review of publications in which clinical study reports have been mentioned.

In this first part of the review, we discuss the history of CSRs, definitions, and relevant terminology. In the second part, we shall outline the methods we have used in compiling this systematic review and discuss the uses to which CSRs have been put, focussing in particular on their uses in studying the benefits and harms of pharmacological interventions and the benefit to harm balance.

The complete list of references we have reviewed in preparing these two parts is given in the Appendix to Part 2.

History of CSRs

During the 1960s, after the introduction of the 1962 Kefauver–Harris Amendment to the US Food Drug and Cosmetics Act and promulgation of the 1968 Medicines Act in the UK, pharmaceutical companies in those jurisdictions were required to provide evidence that any new pharmacological agent that they hoped to market as a pharmaceutical product met specific standards of quality, efficacy, and safety, a set of concepts that had been introduced in the early 1970s [3]. As part of the process, they started to include in the dossiers of information presented to regulatory authorities, when applying for marketing authorization, details of the clinical trials of therapeutic interventions, so-called pivotal trials, that they had performed during drug development.

In those clinical trials, information about the study and the individuals taking part (individual patient data, IPD) is collected and can be summarized in a final report of the findings of the study, with varying amounts of detail. Those details comprise the clinical study reports (CSRs). Although CSRs are primarily associated with studies performed in phase 3, they can also be prepared for studies carried out at any phase of drug development that involves a clinical study, both before and after marketing authorization.

At first, knowledge of the existence of CSRs was largely limited to pharmaceutical manufacturers and regulators. For example, they are not mentioned in general texts such as Spilker’s encyclopaedic 1991 review of clinical trials [4] and Meinert’s 1996 Clinical Trials Dictionary [5]. However, after events in the 1990s had combined to make the existence of CSRs better known in the wider academic community, they were mentioned in the 4th edition of Stephens’ Detection of New Adverse Drug Reactions in 1998, in the context of pharmacovigilance [6].

Developments in the 1990s

The first important development was the foundation by the European Union in 1990 of the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH). [It later became the International Conference on Harmonisation of Technical Requirements for Pharmaceuticals for Human Use.] Among the several documents that the ICH issued was one that is known colloquially as ICH E3, published in 1995. Its full title was “Structure and Content of Clinical Study Reports” [7].

The objective of ICH E3 was to instruct pharmaceutical manufacturers on how to compile a single document that would be acceptable to all regulatory authorities that came under the ICH’s umbrella, i.e. those in the member states of the European Union, in Japan, and in the USA, and the document was recommended to them for adoption. In a later document [8], ICH confirmed that “E3 is a guideline, not a set of rigid requirements or a template, and flexibility is inherent in its use.” The ICH also gave regional regulators the freedom to add specific local requirements in appendices to the document. The policy that the European Medicines Agency (EMA) formulated as a result of ICH E3 was adopted by the US Food and Drug Administration (FDA) but not by the Japanese Pharmaceuticals and Medical Devices Agency (PMDA) [9]. It was also later adopted by other regulators, such as Health Canada.

The second important development was the introduction of evidence-based medicine in the 1990s, following the invention of meta-analysis of combined data from different published studies, introduced in the late 1970s [10]. It was known that publication bias [11], failure to publish the results of a study because of the direction and/or strength of the findings, typically when there are null or negative outcomes [12], was giving some interventions the appearance of greater efficacy and fewer harms than they actually possessed. Authors of systematic reviews therefore started to look for the unpublished results of such studies to include in their meta-analyses [13].

Varying estimates of efficacy

Thus, when a group of researchers found that the positive results of a Cochrane systematic review and meta-analysis that they had published in 2009 [14] could not be replicated and became aware of the existence of clinical study reports, they sought to obtain such reports from the manufacturers of the intervention they were assessing. The manufacturers were the pharmaceutical companies Roche and GSK, and the interventions were oseltamivir and zanamivir, neuraminidase inhibitors that were being used to mitigate the adverse clinical effects of influenza. When they finally obtained the unpublished data in CSRs, they found, in 2014, that the drugs had much less efficacy than had previously been supposed [15, 16].

Contents and structure of CSRs

Although ICH E3 gives details of recommended contents and structure of CSRs, so-called “static listings”, it is not incumbent on pharmaceutical companies to stick rigidly to a single format, and many do not, largely because the resultant documents are time-consuming to prepare—by one estimate anything from 3 to 120 working days (median 27) [17].

The contents of a CSR, as described in ICH E3, are as follows:

• 1. Title page

• 2. Synopsis

• 3. Table of contents

• 4. List of abbreviations

• 5. Ethics

• 6. Study administrative structure

• 7. Introduction

• 8. Study objectives

• 9. Investigational plan

• 10. Study subjects

• 11. Results (efficacy and/or pharmacokinetics/pharmacodynamics)

• 12. Results (safety)

• 13. Discussion & overall conclusions

• 14. End of text tables and figures

• 15. References

• 16.  Appendices

We believe that this format would satisfy the needs of any of the different types of audience who might want access to CSR data (e.g. regulators and authors of systematic reviews). However, pharmaceutical companies are free to organize the information in different ways; here is an example of a possible structure [18]:

• §1 A summary section, including the background and rationale, objectives, materials and methods; a summary of the benefits and harms; discussion, conclusion, and appendices.

• §2 The protocol of the study and a list of amendments; examples of the case report forms used to record demographic and other details about each participant and consent forms; glossaries of terms used; information on the method of randomization; a reporting analysis plan; certificates of analysis; and lists of investigators and the members of the ethics committee.

• §3 Lists of demographic and efficacy data.

• §4 Lists of [so-called] safety data, i.e. harms.

• §5 A statistical report and appendices.

Sample outlines for different sections of CSRs have also been published [19].

CSRs submitted to the EMA are of highly variable length, some of them enormously long (median 9629 pages per submission, IQR 2711–26,673, in a study of 142 medications [20]). In contrast, synopses are considerably shorter (median 5 pages out of nearly 900 pages per report, in a study of 78 CSRs [21]).

Simpler alternatives of preparing CSRs have been proposed, based on the availability of interactive data review tools [22]. These have been recommended [23] and are likely to make production of CSRs easier for pharmaceutical companies. “Lean and mean” writing has been advocated [24]. However, it is not clear that this would benefit the transparency of the data to be shared.

ICH E3 does not include instructions on preparing abbreviated CSRs, for which other guidelines are available [25]. Guidelines have also been suggested for preparing lay summaries of CSRs [26].

A group of editors of laboratory medicine journals have urged the ICMJE to adopt and enforce certain basic requirements for reporting laboratory tests for biomarkers in clinical study reports [27]. However, their language was ambiguous, and it is likely that they meant reports in published papers rather than CSRs. However, presumably their recommendations would be equally applicable to the latter. Likewise, documentation of immunogenicity results when anti-drug antibodies are formed [28].

Definition of a clinical study report and qualifying terms

We define a clinical study report (CSR) as follows:

Clinical study report n

A standardized full report of the protocols, results, and other pertinent details of a clinical study that is typically submitted by a pharmaceutical company to a regulatory authority or authorities when they apply for marketing authorization of a pharmaceutical product.

Talbot et al. have described the purpose of a CSR as being to provide a complete, clear, and accurate account of the study and to serve as a logical link between the rationale, objectives, variables, and the results and conclusions of the study [29].

Thus, a CSR has been succinctly defined as “a standardised full report of a clinical study submitted by a pharmaceutical company to a regulatory authority during the drug approval process” [30].

A CSR was defined at greater length in ICH E3 as follows:

An “integrated” full report of an individual study of any therapeutic, prophylactic or diagnostic agent (referred to herein as drug or treatment) conducted in patients, in which the clinical and statistical description, presentations, and analyses are integrated into a single report, incorporating tables and figures into the main text of the report, or at the end of the text, and with appendices containing the protocol, sample case report forms, investigator-related information, information related to the test drugs/investigational products including active control/comparators, technical statistical documentation, related publications, patient data listings, and technical statistical details such as derivations, computations, analyses, and computer output etc.”

The definition given by the FDA is almost identical [31], and a similar definition has been given by Health Canada [32].

However, since ICH E3 is a guideline and not a statutory document, it is open to those who prepare CSRs to vary the format according to their perception of what is required by regulators, which may differ from one jurisdiction to another. There is a case for legislating for international uniformity.

Qualifying terms

The ICH E3 definition of a CSR given above includes the word “full”, and others have also qualified the term “clinical study reports” with the word “full”, to stress that pharmaceutical companies have only fully committed to providing synopses. Shortened versions of CSRs have also been referred to using the terms “abbreviated” [31] and “condensed” [33]. Occasionally, so-called “supplemental” reports may be issued, containing material not included in other forms of reports [34].

While the drug development process is under way, periodic interim reports may be written to keep the developers informed. These are called “development safety update reports (DSURs)” [35].

Other terminology

The ambiguity of “clinical study report”

If the results of a clinical study of any sort are published, the resulting publication may be referred to as a “clinical study report”, i.e. the report that details the results of that particular study in a publicly available format. That is not what the phrase “clinical study report” means in the context of this review, as the definitions discussed above show. It is unfortunate that the phrase carries this ambiguity, and it would have been better had a different term been chosen for ICH E3. However, the term “clinical study reports” is now too well entrenched for any change to be proposed. Nevertheless, we recommend that the phrase “clinical study report” should be avoided when it does not refer to a CSR; e.g. instead of writing “here we present a clinical study report”, authors could write “here we report the results of a clinical study” or something similar.

Availability of CSRs

By “availability” we mean access to full unredacted CSRs online without restriction.

In 2014, the EMA announced, in accordance with its previously published policies on the availability of documents that were not publicly available [36,37,38,39], that it would make available on their website for downloading the clinical data that had been submitted to the agency by pharmaceutical companies. This came into effect in January 2015 and CSRs are available from the EMA at https://clinicalstudydatarequest.com. In the USA, the FDA introduced a similar pilot program in 2018, although it ended in 2020 [40, 41].

Similar requirements are being applied in other jurisdictions. In Canada, for example, Vanessa’s law imposes an obligation on authorization holders to make information about their clinical trials publicly available [42].

Data sharing

In July 2013, The European Federation of Pharmaceutical Industries and Associations (EFPIA) and the Pharmaceutical Research and Manufacturers of America (The PhRMA Foundation) published a joint statement describing the intention of pharmaceutical companies to “dramatically increase the amount of information available to researchers, patients, and members of the public”, including “patient-level clinical trial data, study-level clinical trial data, full clinical study reports, and protocols from clinical trials … regardless of the outcome” [43]. However, the main commitment seems to have been to make publicly available, “at a minimum” [44], synopses of clinical study reports for clinical trials in patients “within a reasonable period of time after approval of the product and indication”. There was no commitment to provide full CSRs automatically. For example, in a 2015 study of the ways in which data were shared (“data sharing measures”) by large pharmaceutical companies whose products had been approved by the FDA, gleaned from 10 sets of published guidelines, only 25% fully met the measures [45], and in a 2020 study of the benefits and harms of human papillomavirus (HPV) vaccines, the authors were unable to obtain a single complete unredacted CSR [46].

Most of the publications that deal with the availability or non-availability of CSRs have been calls of one sort or another for greater availability. Studies of the availability of CSRs via regulators have shown that reports are generally available from that source [47, 48].

In recent years, data-sharing platforms of various kinds have been launched. They provide information about CSRs [49] in websites of pharmaceutical companies and academic/company consortia. In a study of six of these platforms, the following results were found [the name of the platform is followed by the number of clinical studies whose data were listed on the website as being available on request]:

• Vivli Center for Global Clinical Research Data—5426;

• ClinicalStudyDataRequest.com (CSDR)—2897;

• Yale Open Data Access (YODA) Project—395;

• Biological Specimen and Data Repository Information Coordinating Center (BioLINCC)—219;

• Project Data Sphere—154;

• Supporting Open Access to Researchers-Bristol-Myers Squibb (SOAR-BMS)—0.

The existence of these platforms is not widely known, and they are generally underused, even when data can be obtained from them on request [50].

Some pharmaceutical companies include a statement about data sharing in their publications [e.g. [51]. However, in a formal study of the data-sharing policies of commercial and non-commercial funders, only 41% of the former shared data, whereas of the latter 60% had a mandatory policy and the rest a voluntary one for sharing.

Transparency and confidentiality

Transparency in the context of CSRs has been defined as “the conduct of [regulation] in a fashion that makes decisions, rules and other information visible from the outside” [52].

Although transparency policies are closely linked to the availability of CSRs that underpin decision-making about licensing of medicines, we have used two categories to describe them (see Part 2), based on the language that authors have used in their papers. In doing this, we note that availability does not guarantee transparency, because that will depend on the extent to which the available CSR has been abbreviated and/or redacted. Most of the papers that we have categorized under the heading of transparency have been calls for increased transparency.

In one case, after a company had been granted marketing authorization for a product by the EMA, the company challenged the EMA’s transparency policy of making the CSRs openly available. The European Court of Justice (ECJ) ruled that there was no obligation for an EU institution, body, office, or agency to apply a general presumption of confidentiality in such cases, and that CSRs could therefore be made available [53]. However, it also ruled that in such cases the Agency should request information from a company that might determine that a particular case fell within the commercial interest exception, and the onus was on the company to explain how disclosure of individual passages in a report could reasonably and foreseeably undermine its commercial interests. Indeed, in a study of two drugs (carfilzomib and lesinurad) it had approved, the EMA enabled access to about 260,000 pages of detailed clinical trial information, of which it considered only two pages contained confidential commercial information [54]. The ECJ’s judgement was similar to one that the European Ombudsman had previously reached in a similar case [55].

Since regulators started to make CSRs freely available, transparency has improved. For example, in a study of 19 new drugs, sponsored by 11 large companies, approved by the FDA, and involving 553 trials, an analysis of 505 relevant trials showed that most had been registered in advance (median 100%, IQR 86–100%) [56]. In 71% (IQR 57–100%), results were reported or a CSR synopsis provided, 80% (70–100%) were published, and 96% (80–100%) were publicly available in some form by 13 months after FDA approval. Disclosure rates were lower at the time of approval (65%) but improved significantly by 6 months later. Half of the drugs had publicly disclosed results for all trials. The authors concluded that although clinical trial transparency was high, there was room for improvement.

However, not all pharmaceutical companies observe such high degrees of transparency. In a study of 42 companies, transparency policies were highly variable [57]. Of 23 companies from the top 25 by revenue, 21 (91%) committed to register all trials and 22 (96%) committed to share summary results; however, their policies commonly lacked timelines for disclosure, and trials of unlicensed medicines and off-label uses were included in only 26%. Only 17 companies committed to share the summary results of past trials. Twenty-two companies had a policy on sharing CSRs, mostly on request, two committed to share only synopses, and only two policies included unlicensed treatments. Twenty-two companies had a policy to share IPD; 14 included phase 4 trials and only one included trials on unlicensed medicines and off-label uses. The smaller companies made fewer transparency commitments. Two fell short of industry body commitments on registration and three on summary results. In some cases contradictory and ambiguous forms of language were documented.

The results of a study in 2021 were similar [58]. Of 316 industry-sponsored clinical trials of 30 medicines that had been approved by the FDA, CSRs were available for public download in 70 cases (22%), 37 were available from the EMA, and 40 from Health Canada. Although the companies did not offer direct downloads of CSRs, they confirmed that the CSRs from 183 of the 316 clinical trials (58%) were eligible for independent request by submitting a research proposal. Overall, 218 of the trials (69%) had CSRs available for public download and/or could be requested from the company.

Other types of reports

We have not here discussed other types of reports as sources of information, such as clinical investigator brochures, charters of data monitoring committees, endpoint adjudication charters, or regulatory approval packages, all of which may provide useful information on benefits and harms. We have not found any studies in which information available in those types of documents has been compared with that found in CSRs; such comparisons would be of interest.

Conclusions

Our conclusions from this review are listed at the end of Part 2.