Outcome reporting bias: if you say you’re going to do something, do it!

Selective reporting of outcomes is just one type of reporting bias and there are a number of ways in which it can arise. In the previous blog we gave an example of the effect of selective reporting bias through under-reporting of data. There are other ways in which selective reporting of outcomes may arise. These are summarised in the Cochrane handbook but include:

Do you consider any of them when reading a paper?

Prevalence and impact of selective reporting

A 2009 study showed that in a cohort of registered trials, 31% (46 of 147) had some form of discrepancy between the outcomes registered and the outcomes published. Furthermore, when it could be assessed, the studies with changes were more likely to report a statistically significant result.  In a cohort of Cochrane systematic reviews, over a third were suspected to have at least one RCT containing selective outcome reporting bias. The authors also demonstrated the impact of this bias and found that selective outcome reporting can produce a median change in the treatment effect size of 39% (IQR 18% to 67%).

The authors of a 2015 systematic review of studies that examined selective outcome reporting found 27 relevant inclusions. The median proportion of trials with an identified discrepancy between the registered and published primary outcome was 31% (although there was large variability between them). Four studies observed outcome changes in more than 50 % of trials.

Bottom line, selective reporting of outcomes is still very much around.

Reducing selective reporting of outcomes

Recently The Compare Project was launched. The aim is to prospectively audit the presence of outcome switching in all RCTs published in the top five medical journals (NEJM, JAMA, The Lancet, Annals of Internal Medicine, BMJ). The outcomes presented in each published trial are compared with the stated outcomes in the clinical trial registry and protocol. When any outcome switching is noted, a letter is sent to the journal editors to inform them. As stated in the project approach: “Through increased awareness of misreported outcomes, individual accountability, and feedback for specific journals, we hope to fix this ongoing problem”. The results are being posted and updated live, and as I write 61 trials have been checked and only 9 were deemed perfect. Look and see if the results have changed.

But the long term “fix” to reduce selective reporting of outcomes shouldn’t actually be that difficult. Reviewers and editors should make it routine practice (and many do) to check that the published outcomes match those in the trial protocol or registry (e.g. the ISRCTN or ClinicalTrials.gov). If the protocol is not available it should be requested directly from the corresponding author. If all that is not possible (which would be a concern in itself), then as a reader, simply checking the outcomes reported in the methods section with the outcomes reported in the results section of the published article should be a minimum.

The CONSORT statement is a minimum set of recommendations for the reporting of randomized trials to increase transparency. The recommendation specifically includes the following:

Outcomes 6a Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed
6b Any changes to trial outcomes after the trial commenced, with reasons

 

So cross-checking of CONSORT statements completed by authors is also good practice, to ensure accurate reporting.

However, although many journals endorse the CONSORT statement, not all enforce it.

Trialists should be ensuring that all data are as transparent as humanly possible. Open access journals and open repositories for data deposition aim to ensure transparency and access to all data. There will be little place to hide should you choose otherwise.

But the bottom line, if you want to be a good experimenter, is that if you say you’re going to do something: (a) do it, (b) show that you did it, and (c) show how you did it.

Kamal Mahtani

About Kamal Mahtani

Kamal R. Mahtani is a GP and and Clinical Lecturer at the University of Oxford

View more posts by Kamal Mahtani

One comment on “Outcome reporting bias: if you say you’re going to do something, do it!

  1. Pingback: Outcome reporting bias: is it ok to be a little selective? - CEBM

Leave a Reply

Your email address will not be published. Required fields are marked *