Do journals really care about bias and reporting in clinical trials?
July 25, 2019
“…we hope to have highlighted one particular aspect of RCT reporting that is often lacking, at the same time gaining a much more detailed understanding of how to appraise medical evidence both informally and within a more rigid analysis”
Who are we, and why did we choose this SSM?
We are Oliver and Miranda, two final (6th) year medical students at Oxford. During our time at medical school, we have at various points touched on the necessary academic and clinical skills involved in the practice of evidence-based medicine (EBM). Having both enjoyed this aspect of our course, and keen to further explore the principles and methods underpinning EBM, we elected to take part in a three-week special study module (SSM) at the Centre for Evidence-Based Medicine (CEBM).
What did we do?
Our time at CEBM focused on the assessment of how authors of randomised controlled trials (RCTs) report their methodology. This is an important topic as it is through adequate standards of reporting that readers can determine how likely trial results are to be influenced by bias, and therefore judge the validity of the results. Guidelines exist, including structured checklists, which can be used by authors when writing up an RCT, in order to ensure adequate reporting. The most widely endorsed of these guidelines is the Consolidated Standards of Reporting Trials (CONSORT). First published in 1996, with the latest edition published 2010, CONSORT was developed specifically to guide reporting of RCTs, covering key elements of trial methodology that can introduce bias including the reporting of randomisation sequence generation and concealment prior to (allocation concealment) and after (blinding) randomisation.
Despite existing for over 20 years, and experiencing widespread endorsement, adherence to CONSORT guidelines has been shown to be poor. In our project we wanted to assess the current state of reporting, particularly comparing reporting of trials published in CONSORT-endorsing versus non-endorsing journals.
To do this, we decided to take as our sample, every RCT included in systematic reviews of interventions published in the Cochrane Library during a one-month period (23/03/19 to 23/04/2019 inclusive). All Cochrane reviews include as a figure, a risk of bias summary. This figure shows for each RCT included in the review, the Cochrane authors judgement of the risk of bias, which can be either high, low, or unclear. If CONSORT has been followed correctly, and reporting is adequate, then no RCTs should be judged as unclear.
We used the Cochrane reviews to identify all the RCTs judged as having ‘unclear’ risk of bias for one or more of three key elements used in the risk of bias assessment: random sequence generation, allocation concealment and blinding. We then looked at these RCTs to see (1) when they were published (2) the journal they were published in. This analysis was performed by both MS and OB; to ensure consistency between us we cross-checked a random selection of 5 reviews each: no discrepancies were found.
Next, for the RCTs published from 1997 onwards, so after the first iteration of the CONSORT statement, we went to the journal website to see whether they endorse CONSORT. We did this as we wanted to see whether these RCTs with ‘unclear’ risk of bias were being published in CONSORT-endorsing journals or elsewhere. Finally, we also looked in detail at the author guidelines for the CONSORT-endorsing journals to identify whether they recommend or require adherence to CONSORT and how many require authors to submit a completed CONSORT checklist with their manuscript.
What did we find?
Cochrane reviews and randomised controlled trials
We found 46 Cochrane reviews during the period 23rd March – 23rd April 2019 (inclusive), of which 3 had, as of 24/04/2019, been withdrawn. This left 43 Cochrane reviews including a total of 702 RCTs spanning a publication period from 1971 to 2018. We excluded 16 RCTs from our analysis as they were not published in a journal (i.e. thesis, pharmaceutical trial, conference abstract), leaving 686 in our final analysis. Almost 70% (476) of these were judged by Cochrane authors to have an unclear risk of bias for random sequence generation, allocation concealment or blinding. Of these 476 RCTs, 108 were published before and 368 were published after the first iteration of the CONSORT guidelines in 1996 (see Table 1).
(% of total RCTs / % RCTs with unclear risk of bias)
|Total RCTs included in 1-month of Cochrane reviews
|Excluded as not a journal publication
|Total RCTs included in final analysis
|RCTs with unclear risk of bias
Table 1: Total and proportion of RCTs with an ‘unclear risk’ of bias for random sequence generation, allocation concealment or blinding published in journals, included in 43 Cochrane reviews.
CONSORT endorsement assessment
The 476 RCTs with an unclear risk of bias assessment were published in 307 unique journals, of which 259 (84%) were the source of trials published after the first iteration of the CONSORT guidelines (i.e. post-1996). We assessed these 259 journals for their policy in relation to CONSORT guidelines (Figure 1). In order to qualify as endorsing CONSORT, a specific mention of CONSORT with an expressed expectation that authors refer to, or adhere to the guidelines must have been made on journal’s website. We were unable to check 40 (15.4%) journals due to them either being discontinued or difficulty locating a specific journal website or author guidelines. Of the remainder, 123 (47.7%) expressed endorsement of CONSORT on their website, whilst 96 (37.1%) did not.
For the 123 journals endorsing CONSORT, we looked at the author guidelines to assess the actual expectations of authors submitting an RCT (Figure 2). 51 (41.5%) of the journals recommend adherence to CONSORT, 19 (15.4%) require adherence but without needing to submit a completed checklist, and 53 (43.1%) require authors to submit a completed CONSORT checklist.
Figure 1. CONSORT endorsement status of journals from which the RCTs published post-1996, judged to have an unclear risk of bias, were taken.
Figure 2. Author requirements of the 123 CONSORT endorsing journals from which RCTs were judged to have an unclear risk of bias.
Cochrane reviews of interventions published between March-April 2019 judged it was not possible to determine the risk of bias for key methodological aspects in over two-thirds of included trials due to poor (under) reporting in trial publications. The findings from these trials may be inaccurate, potentially giving overestimated or underestimated effect estimates which could ultimately impact on patient care.
There are two logical explanations for our findings:
- Journals endorsing CONSORT do not check to see if authors are adhering, even if they require a checklist:
- Incorrect judgement by the Cochrane review authors
We postulate it is more likely to be a result of the first rather than second explanation, though we cannot offer certainty as we have not checked the original studies which is one of the limitations of our assessment. There are others, a key one being that we do not know the specific date or year when included journals began endorsing CONSORT.
That said, policymakers and clinicians are expected to seek out up-to-date systematic review evidence for treatment recommendations and decisions and many will turn to the Cochrane Library as their source of such evidence. At present, they will be faced with reviews containing high degrees of uncertainty due to the quality of reporting of included trials.
It’s a problem that is fixable if we (the evidence pipeline community) have the will. A potentially effective way to assess this will would be to prospectively identify published trials for CONSORT adherence around, say allocation concealment, and submit correction letters when we find non-adherent reporting. However, even this will generate mixed outcomes as previously witnessed for outcome reporting.
What did we learn?
We found our time in the CEBM both enjoyable and entertaining and were pleased to have been able to complete this project in the relatively short time available. Under the supervision of Dr David Nunan, we were able to get a glimpse into the limitations in written reports from RCTs – while reading research reports, an important part of learning and practising medicine, it is easy to miss what information is missing. Through this project we hope to have highlighted one particular aspect of RCT reporting that is still often lacking, at the same time gaining a much more detailed understanding of how to appraise medical evidence both informally and within a more rigid analysis.
Many thanks to Dr David Nunan for his excellent mentoring, supervision and teaching.