The FDA has authorised remdesivir for use in COVID-19 patients: but there’s no good evidence it reduces mortality
May 12, 2020
Henry M Drysdale, Nicholas J DeVito, Jeffrey K Aronson
Correspondence to henry.drysdale@phc.ox.ac.uk
Lay Summary by Mandy Payne, Health Watch
Two weeks ago, the National Institutes of Health (NIH) announced in a press release that their clinical trial on the experimental antiviral drug remdesivir, run by the National Institute of Allergy and Infectious Diseases (NIAID), shows that it “accelerates recovery from advanced COVID-19”. Patients in the trial who received remdesivir had a 31% faster time to recovery than those who received placebo. The same day, Dr. Anthony Fauci announced in the Oval Office that the trial demonstrates “a clear-cut, significant positive effect”, and that remdesivir “will be the standard of care”, but he gave no information about the findings that had led him to reach that conclusion. Two days later, the FDA issued an “emergency use authorization”, meaning that remdesivir can now be “distributed and used by licensed health care providers to treat adults and children hospitalized with severe COVID-19”. The World Health Organization (WHO) is now engaging with the US Government to make remdesivir widely available.
This may look like important progress at a difficult time. However, there are problems with the NIAID trial, and with how the results have been used. Apart from the press release in which the NIH mentioned two results, the trial itself remains unpublished, so we have no information on the methods, other outcomes (like the need for ventilation), full results, or how the data were analysed. This context is essential for doctors, patients, and policymakers to interpret the results of the trial accurately and make an informed judgement about the benefits and harms of remdesivir, but instead a single isolated positive result is being used to inform policy and treatment decisions on a massive scale. A more detailed analysis is promised in a “forthcoming report” with no specific timeline provided. This is indefensible: the NIAID should publish the full methods and results of the trial, including all outcomes and adverse events, for critical peer review and discussion, before results are used to inform the widespread treatment of patients.
According to the Washington Post Dr. Fauci said “he received a report from the trial’s leaders on Monday. He said he announced preliminary results early, before they were published, out of ethical concerns, so that patients receiving a placebo can opt to get the drug instead.” Given that no mortality benefit has been demonstrated for remdesivir, the decision to stop the placebo arm based on these data may be premature. There is another problem with stopping the trial early: the effect of remdesivir on mortality in this trial currently shows a non-significant result (P=0.059); further results may provide the power necessary to give a more definitive answer, but there is no guarantee that the P value will continue to fall as the trial proceeds. The apparent trend towards significance may reverse, but we won’t know this if the trial is stopped early.
There are further concerns about the NIAID remdesivir trial: 13 days before the NIH press release and Dr. Fauci’s statement in the White House, the researchers running the remdesivir trial switched their primary outcome on ClinicalTrials.gov. They originally intended to evaluate remdesivir using a 7-point scale, including how many patients receiving remdesivir or placebo died, and how many required mechanical ventilation. Instead, on 16 April the trialists switched the primary outcome registered on ClinicalTrials.gov to “time to recovery”: this is the positive and statistically significant outcome reported by the NIH that prompted rapid authorisation of remdesivir by the FDA. The ClincialTrials.gov entry for the trial also lists 27 additional outcomes in addition to the two reported in the press release, including mortality at both day 14 and day 29: none of these 27 additional outcomes has yet been reported.
Outcome switching is not necessarily a problem: the CONSORT guidelines on reporting clinical trials state that it is acceptable to switch outcomes during the course of a trial, but that any changes must be declared, and reasons given, when the trial results are reported. The problem comes when outcome switching is not declared by the researchers conducting the study, because this misleads the reader and increases the chances of false-positive results. Unfortunately, this is common; the largest systematic review shows that around a third of primary outcomes are switched. More recently, the COMPare study showed that around a quarter of primary outcomes in major medical journals were switched without any declaration of the change. Ultimately, undeclared outcome switching can lead to patient harm: in 2001, “study 329” reported positive results for the use of paroxetine in adolescents with depression. However, these outcomes had been silently switched, and when the full data were analysed in 2014, paroxetine was found to be ineffective for depression, and associated with significant increases in suicide. This was not discovered until millions of prescriptions had been issued.
In the case of the NIAID remdesivir trial, the problem is that this new primary outcome has been used to inform major policy and treatment decisions without full reporting of the other outcomes (such as the need for ventilation) or a proper declaration of the switch. Following questions from the public and press about the outcome switching in this trial, the NIAID replied in an email to a CNBC reporter, stating “NIAID statisticians performed modeling of what happens if the right day is not picked for assessment, which revealed that meaningful treatment effects could be missed with that primary endpoint. Time to recovery avoids this issue”. NIAID also stated “This change in primary endpoint was made without any knowledge of data from ACTT, before any interim data was available”, implying that there could be no bias in the switch. However, the NIAID response is problematic: it implies that the new primary outcome was chosen because it was likely to yield a positive result, rather than because it is an important outcome for patients with COVID-19; and it contains no explanation of why this result was carried forward before any other trial information or results were released for critical peer review.
This raises a further point: the outcome used to drive the rapid authorization of remdesivir, time to recovery, is far less useful than other outcomes, particularly death, but also the need for mechanical ventilation and adverse effects of the drug. Reducing the time to recovery for patients with COVID-19 may represent benefits such as minimising the after effects of the disease, or reducing healthcare costs. However, time to recover is at best a surrogate marker of effectiveness. With a global pandemic of a new virus that causes severe respiratory illness, doctors and patients need to know whether remdesivir reduces the numbers of people dying from COVID-19; whether it reduces the need for ventilation, which has significant adverse effects and can overwhelm health services; and whether the drug itself causes adverse reactions. As Dr. Peter Bach of Memorial Sloan-Kettering Cancer Centre said in a recent STATNews piece about the trial “The reason we have shut our whole society down is not to prevent Covid-19 patients from spending a few more days in the hospital. It is to prevent patients from dying. Mortality is the right endpoint.”
This brings us on to the most concerning aspect of remdesivir’s rapid authorization: there is still no good evidence that remdesivir reduces the number of people who die from COVID-19. In a recent trial of remdesivir published in the Lancet, remdesivir did not reduce mortality, time to clinical improvement, duration of mechanical ventilation, or viral load. The NIAID stated that their trial of remdesivir showed no significant reduction in mortality (8% vs 11%, P = 0.059). Neither trial was able to reach full completion. This is worrying: rapid widespread dissemination and use of remdesivir to treat patients with COVID-19 will direct funding away from other vital healthcare resources and research efforts, and it may not save a single life.
COVID-19 is a new disease, and we do not have the luxury of good-quality evidence to guide diagnosis, prognosis, or treatment. In making current treatment decisions we must therefore rely on rapid evaluations of emerging results from a range of research efforts, in order to make the best clinical and policy decisions possible with the data available to us. However, undeclared outcome-switching, using isolated results without publishing the trial, and wishful thinking around modest results in surrogate outcomes does not help patients suffering and dying with COVID-19 today. It certainly will not help the patients who become infected later, when another epidemic of COVID-19 comes along.
Henry M Drysdale is a clinical researcher with the DataLab, University of Oxford, and a medical doctor in the NHS. His academic work focuses on integrity in academic and NHS research. Henry co-founded the COMPare project, which monitored outcome switching in 5 major medical journals.
COI: HMD has been employed on grants from the Laura and John Arnold Foundation.
Nicholas J. DeVito is a doctoral student in the Nuffield Department of Primary Care Health Sciences, University of Oxford. He studies transparency in biomedical research and runs covid19.trialstracker.net as part of the DataLab and Centre for Evidence-Based Medicine.
COI: NJD receives a doctoral studentship from the Naji Foundation, grant support from the Fetzer Franklin Fund, and has been employed on grants from the Laura and John Arnold Foundation and the Good Thinking Society.
Jeffrey K Aronson is a physician and clinical pharmacologist working in the Centre for Evidence-Based Medicine in the Nuffield Department of Primary Care Health Sciences, University of Oxford. He is an Associate Editor of BMJ EBM and a President Emeritus of the British Pharmacological Society.
COI: JKA has written papers on the subject of clinical trials and has written papers and edited textbooks on clinical pharmacology, pharmacovigilance, and adverse drug reactions.
