Kamal Mahtani: Deputy Director of CEBM
David Nunan, Senior Research Fellow
As part of our work in the Centre for Evidence-Based Medicine we take an active interest in the quality of systematic reviews of topics of public and patient importance, particularly when they generate bold headlines.
Perhaps the most common, and recent, eye catching headlines relate to diet and nutrition.
For example, there has been recent widespread media reports with headlines such as “Eating fat is good for you: Doctors change their minds after 40 years” and “Saturated fat ‘ISN’T bad for your heart’”. If correct, these headlines challenge current dietary recommendations from a number of organisations such as the American Heart Association, the British Heart Foundation, and the World Health Organization, all of which advise limiting dietary saturated fats.
So should there be a major shift in the public health advice we receive and give out on this topic? For this to happen the evidence underpinning these headlines should be at least equivalent with current evidence in terms of its quality, reliability and trustworthiness.
Unfortunately, the quality of the evidence does not appear to justify such bold headlines.
As one example, a recent systematic review in the BMJ Open journal concluded that, based on a summary of their evidence, current recommendations to lower blood cholesterol for cardiovascular protection was wrong. However, the study itself highlighted several competing interests amongst the authors which may limit the interpretation. Furthermore, our post-publication analysis of the study highlighted several weaknesses in the methods, limiting the conclusions drawn from the study. The review authors, despite a press release to the public, did not convey the weaknesses we highlighted.
In a similar vein, a 2015 systematic review in BMJ Open Heart emphatically concluded (in its declaratory title) that “Evidence from randomised controlled trials did not support the introduction of dietary fat guidelines in 1977 and 1983”. The review set out to specifically analyse the association between saturated fat intake and mortality due to all causes and coronary heart disease (CHD) from RCTs that had been carried out before key national recommendations were made in the USA (1977 and the UK (1983).
Interestingly, a Cochrane systematic review, also published in 2015, looked at reduced saturated fat intake for cardiovascular disease from RCTs. The review also included studies published before the introduction of the 1977 USA and the 1983 UK guidelines.
You might think that both reviews would come to similar conclusions. However, not all systematic reviews are equal. A recent analysis concluded that non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, a systemic difference that indicates that such reviews (non-Cochrane) “may be overstating the evidence,” the author’s comment.
A quick look at some of the differences between the 2015 BMJ Open Heart systematic review (hereafter referred to as ‘non-Cochrane review”) and the Cochrane systematic review reveals some interesting findings.
For example, authors of Cochrane reviews are required to publish their protocols before starting their review. They must outline their methods for searching, including, and appraising the studies that will be included (and excluded). Cochrane reviewers also have to outline the outcomes they will look at, and categorise them as to their clinical relevance/importance. The 2015 Cochrane review identified eight outcomes that they considered to be clinically important.
Stating and then publishing systematic review protocols in this way is important, as it reduces the risk of reporting only selected outcomes when the review is published (known as outcome reporting bias). It also reduces the risk that the reviewers will simply run lots of analyses for lots of different outcomes and find a statistically significant difference for one of them by luck (known as “p-hacking”). For this reason researchers of non-Cochrane reviews are also encouraged to prospectively register their systematic review protocol on the PROSPERO database.
The 2015 non-Cochrane review did not appear to have a pre-defined, published protocol. And because the final published review only included two outcomes – deaths due to any cause or due to coronary heart disease (CHD) – the reader has little idea if these were the outcomes the reviewers set out to examine from the start.
The 2015 non-Cochrane reviewers stated in their review, that they only combined studies published at or before the USA (1977) and UK (1983) recommendations on saturated fat intake were made. They showed there was no statistical difference between people randomised to a low-fat diet and those randomised to a “usual” diet for death from any cause (relative risk [RR] 0.996, 95% confidence interval [CI] 0.87 to 1.15) and death due to CHD (RR 0.99, 95% CI 0.78 to 1.25). This finding was used to underpin their (definitive) title.
However, as the Cochrane review also included studies published before 1983, it’s possible to compare this data directly with those of the non-Cochrane review.
For all-cause and CHD deaths, the Cochrane review showed similar results but pooled risk estimates were between 1-3% larger (i.e. higher risk of death in the usual diet group).
On a closer look, we have observed discrepancies in the data extraction between the two reviews that might explain these differences in risk estimates (see figure 1). For example, lets compare the data extracted on all cause mortality from the “Rose Olive Oil (1965)” study.
- Non-Cochrane review
- = RR 3.00, 95% CI 0.33 to 26.99
- Cochrane review
- = RR 1.50, 95% CI 0.17 to 13.05
That’s a 2-fold difference, between the reviews, in the data extracted from the same study. Such discrepancies can start to put doubts into a reader’s mind, especially given the evidence for less precision in non-Cochrane reviews.
Figure 1. Relative risk estimates for all-cause mortality from included studies in the two reviews (Harcombe = lead author of the non-Cochrane review; Hooper = lead author of the Cochrane review). NI = study not included in the review.
Although it was not obviously pre-specified, the non-Cochrane review only looked at deaths as their main outcome.
However, the number of cardiovascular events (such as myocardial infarction, stroke, heart failure) a patient is at risk of having is also a critically important outcome to measure.
Therefore, we reanalysed data from studies published before 1983 that were included in the Cochrane review. This showed a statistically significant 21% reduced risk in cardiovascular events that was in favour of participants randomised to a low saturated fat diet (RR 0.79, 0.69 to 0.91; figure 2). This roughly translates to a number needed to treat of 14, meaning that for every 14 people who follow a low saturated fat diet, 1 additional cardiovascular event would be prevented.
Although we would still need to take into account the quality of the evidence, the data available at the time of the 1977 US and 1983 UK guidelines appear to support a low saturated fat diet for the reduction of major cardiovascular events.
So why the discrepancy? This may be difficult to fully elicit.
The non-Cochrane review initially stated that there were no competing interests amongst the authors of the review. However, the journal later published an accompanying correction to this, which highlighted that the corresponding author receives income from writing diet books. The correction was also accompanied by an editor’s comment, which stated that the competing interests should have been made from the outset.
Figure 2. Meta-analysis for cardiovascular events based on studies available pre-1983 from the Cochrane review.
While they are often used to stimulate scientific debate, contrasting findings and conclusions can be confusing to wider audiences.
Our preliminary results appear to suggest that there may be elements in at least one non-Cochrane review that contradicted the definitive statements its authors have made. This warrants an independent and more detailed analysis.
We’ll be sure to let you know what we find, whatever the results.
Dr David Nunan – Departmental Lecturer in Evidence Based Medicine
Dr Kamal R. Mahtani – GP and Deputy Director
Centre for Evidence Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford.
Dr David Nunan and Dr Kamal R. Mahtani are both members of the Royal College of General Practitioners (RCGP) steering committee to support the new Physical Activity and Lifestyle clinical priority.
Dr Nunan and Dr Mahtani have received funding for research from the NHS National Institute for Health Research School for Primary Care Research (NIHR SPCR) and the RCGP for independent research projects related to physical activity and dietary interventions. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, the Department of Health, the RCGP or any other institution named in this post.
They declare no other relevant conflicts of interest.