Mark Jones, Senior Lecturer, University of Queensland
“There are a number of possible methodological reasons why the conclusions could be so different.”
Muthuri et al published a systematic review in the Lancet in 2014, which included observational data for 29,234 patients from 78 studies of which 64% were treated with neuraminidase inhibitors (NI) and 10% died. The study concluded that NIs protect hospitalised patients with 2009A/H1N1 influenza from mortality with adjusted odds ratio 0.81 (95% CI 0·70–0·93; p=0·0024).
Heneghan et al, published a HTA systematic review in 2016, which included summary data on 30 studies, with 11,013 patients, and 1301 deaths (12%). Individual patient data was obtained from four studies with 3071 patients and 242 (8%) deaths. Analysis showed insufficient evidence that oseltamivir reduced the risk of mortality [hazard ratio (HR) 1.03, 95% CI 0.64 to 1.65]. (You can also find a summary at our tamiflu timeline here and here.
Why are the conclusions so different?
There are a number of possible methodological reasons why the conclusions could be so different. Some are outlined below:
The studies are based on different data.
It is true that the studies are largely based on different data and this could explain why the studies have different conclusions. However descriptive analysis for each of the studies shows quite similar results. For example, Muthuri et al reported slightly higher proportion of treated patients died compared to untreated patients (9.7 vs 9.2%). Furthermore a crude comparison that takes account of clustering by study showed odds ratio = 0.92 (95% CI 0.81–1.05). Similarly, Heneghan et al reported similar percentage of deaths receiving oseltamivir to that of survivors (83% vs. 82%) with meta-analysis showing odds ratio = 0.90 (95% CI 0.67 to 1.20). Crude analysis of the IPD also showed a similar result with odds ratio = 0.93 (95% CI 0.65 to 1.33).
A perplexing aspect of Muthuri et al is data presented in the supplementary appendix 6 shows that of the 78 studies included, all patients in 27 studies received neuraminidase inhibitors and 16 studies had no mortality. It is unclear how these studies were included in a mortality comparison between neuraminidase inhibitors and no treatment because, in meta-analyses, studies with no events, and studies without a control group are typically dropped from the analysis as they cannot provide any useful information?
The statistical model used for analysis is different.
The main analysis in Muthuri et al was conducted using generalised linear mixed models. This method does not take account of the time-dependent nature of treatment, leading to immortal time bias. An analysis that does not take account of the time-dependent nature of treatment misclassifies time at risk of outcome prior to treatment as being associated with treatment when in fact it is associated with no treatment. Methods such as multivariable adjustment of confounding variables and propensity score matching do not address time-dependent bias because they do not correct the misclassification of time at risk.
The time-dependent nature of treatment can be accounted for correctly using Cox proportional hazards regression and including treatment as a time-dependent exposure variable. Muthuri et al report that they have done this type of analysis on a subset of the data however the results (adjusted HR = 0.51, 95% CI 0.45–0.58; p<0·0001) are not consistent with what would be expected given the direction of immortal time bias. Immortal time bias makes the treatment appear better than it really is. Also note the very tight confidence interval despite the analysis being conducted on a subset of the data where 35% of the treated patients were dropped due to missing data on timing of treatment. Furthermore excluding these patients may have introduced a selection bias because we found treated patients with missing data on timing of treatment had poorer outcomes. In a second blog I speculate on how Muthuri et al may have conducted a biased survival analysis.
In contrast, Heneghan et al use a time-dependent Cox regression model that also takes account of the competing risk of hospital discharge. Furthermore, the results from this analysis suggest immortal time bias is in the direction consistent with that expected based on previous research.
Confounding by indication can occur.
Confounding by indication can occur when the reasons for treating particular patients and not treating other patients are associated with likelihood of mortality. For example, it could be that sicker patients tend to be more likely to get treatment. Unfortunately this is a limitation of all observational studies including Heneghan et al and Muthuri et al. Both studies have attempted to address this limitation by adjusting for potential confounders however it is uncertain whether residual confounding remains. Muthuri et al used propensity scores whereas Heneghan et al investigated variables associated with receiving treatment and included those variables in multivariable analysis. Heneghan et al further showed that, compared to patients treated early, patients treated late with oseltamivir had increased likelihood of a number of comorbidities, including infection, cardiovascular, diabetes and obesity. This finding provides (at least) a partial explanation for late treatment being associated with poorer outcomes.