When researchers reanalyze patient-level data from randomized controlled trials (RCTs), the trial outcomes change in approximately 1 of 3 cases, according to a study published in the September 10 issue of JAMA. The new study provides more fuel for an already heated debate on access to clinical trial data.
To learn what type of effect access to patient-level raw data from clinical trials might have, Shanil Ebrahim, PhD, from Stanford University in California, and colleagues searched MEDLINE for published studies (through March 9, 2014) that were reanalyses of trial data. Specifically, the group aimed to find studies that retested the same hypothesis as the original RCT. Overall, they identified 37 such publications.
Of those, 35% of the reanalyses produced results that were significantly different from the initial analysis. In 3 (8%) instances, the newer analysis changed which patients should be treated, in 1 (3%) instance it reduced the proportion of patients that should be treated, and in 9 (24%) instances it increased the proportion of patients who should be treated.
The authors also report that 2 reanalyses changed the magnitude of the treatment effect initially reported, 4 led to a change in statistical significance, and 4 actually changed the direction of the effect.
"It is difficult to assess whether these changes in trial conclusions led eventually to major changes in clinical practice and, if so, how large these changes were," the authors write. "Clinical practice choices depend only partly on trial evidence, and sometimes multiple additional trials exist that inform the same question. Nevertheless, when contradicting messages exist, it is unclear which of the 2 discrepant articles will have more influence: the original is usually published in more influential journals, but the subsequent reanalysis may be viewed as a more correct appraisal of the data."
Dr. Ebrahim and colleagues note that all but 5 reanalyses were carried out by authors included on the original publication.
The most common difference between the newer analysis and the initial one was a change in statistical or analytical methods, followed by changes in the definition or measurement of outcomes, how missing data were handled, and use of an intention-to-treat study population. Four of the reanalyses addressed errors in the initial publication.
Access Is Crucial
In an accompanying editorial, Harlan M. Krumholz, MD, from Yale University School of Medicine, New Haven, Connecticut, and Eric D. Peterson, MD, MPH, from Duke University Medical Center, Durham, North Carolina, note that one of the "discomforting" findings from the current study is how infrequently reanalyses occur.
"Searching the literature from 1966 to present, the authors found only 37 reports that met their criteria as an RCT reanalysis," they write. "Of these few reanalyses performed, the majority (84%) had overlapping authors from the original report. Thus, reanalyses are not only rare, but the majority that were reported were not fully independent of the original research group."
To increase access to trial data and improve transparency, Dr. Krumholz and Dr. Peterson recommend several steps be taken. For example, trial aims, analyses, and methods should all be prespecified and the results "registered with the medical community." In addition, all raw data and metadata should be available to other investigators interested in reanalyzing it.
"The culture of clinical research in medicine will need to evolve for open science to succeed," they write. "The recognition that one trial can potentially lead to different findings and conclusions depending on many discretionary decisions that are made about the data and reanalyses almost mandates that those choices are transparent and described in detail — and that others have the chance to replicate them. Rather than the rare exception, open science and replication should become the standard for all trials and especially those that have high potential to influence practice."
The study authors and the editorialists acknowledge that although not everyone is enthusiastic about more open data policies, there have been some hopeful efforts in that direction from academic and commercial groups.
Dr. Ebrahim reports receiving support from Mitacs Elevate and SickKids Restracomp Postdoctoral Awards. Dr. Mills reports receiving support via a Canadian Institutes of Health Research Canada Research Chair. Dr. Krumholz reports board membership with UnitedHealthcare and grants or pending grants with the Centers for Medicare & Medicaid Services, Medtronic, and Johnson & Johnson. The Duke Clinical Research Institute, of which Dr. Peterson is the executive director, conducts clinical trials and other research with a number of governmental and industry partners.
JAMA. 2014;312:1002-1003, 1024-1032. Article abstract, Editorial extract
Δεν υπάρχουν σχόλια:
Δημοσίευση σχολίου