A Composite Reference Standard for COVID-19 Diagnostic Accuracy Studies: a roadmap
May 18, 2020
Sara Graziadio1, Timothy Hicks1, A. Joy Allen2, Jana Suklan2, Samuel G. Urwin1,
Amanda Winter1, D. Ashley Price1, Richard Body3
1NIHR Newcastle In Vitro Diagnostics Co-operative
Newcastle upon Tyne NHS Hospitals Foundation Trust
Newcastle upon Tyne, NE1 4LP
2NIHR Newcastle In Vitro Diagnostics Co-operative
Newcastle University, Newcastle upon Tyne, NE2 4HH
3Division of Cardiovascular Sciences, The University of Manchester
The University of Manchester, Manchester, M13 9PL
Correspondence to firstname.lastname@example.org
The aim is to develop a composite reference standard for COVID-19 diagnosis that will support a standardised approach across research groups to decrease the high false negative rate of rRT-PCR that could penalise the evaluation of diagnostic accuracy of new tests if rRT-PCR is used as a stand-alone reference standard. Currently there is a paucity of robust evidence for the use of blood biomarkers in COVID-19 diagnosis, therefore it is proposed that an extensive list of biomarkers is recorded during the diagnostic accuracy evaluations of the new COVID-19 tests in the summer months in order to help bridge this knowledge gap. To analyse this data, a temporary Composite Reference Standard (CRS) which includes radiological findings will be used to support the identification of biomarkers discriminatory for COVID-19. These will be included in the next iteration of CRS which could be used in the winter studies to discriminate COVID-19 from other respiratory diseases, highly prevalent in the winter months and characterized by similar radiological features to COVID-19.
Clinical studies are in planning or currently underway and aim to evaluate the in-context diagnostic accuracy of novel diagnostic tests. These novel tests have the potential to be useful diagnostic tools during the next phases of the pandemic and for the differential diagnosis during the endemic phase. Real time reverse transcriptase polymerase chain reaction (rRT-PCR) is currently the reference standard for the diagnosis of COVID-19, but a high false negative rate has been reported.[1-3] Many of these diagnostic accuracy studies compare the new tests with rRT-PCR, an imperfect reference standard. The effect of this is underestimation of the specificity of the new tests.
The authors propose the development and use of a composite reference standard, with predefined diagnostic criteria or levels of diagnostic uncertainty, to allow a comparison with evidence from up to date literature with regards to diagnostic accuracy, but also with current clinical practice in disease management. Multiple composite reference standards can be developed/adapted for specific clinical settings in order to take into account feasibility of use in practice (e.g. evaluations in the community settings cannot rely on radiological examinations). However, the focus in this article was on the development of a composite reference standard to be used in hospital studies.
Rapid reviews were carried out to identify signs and symptoms, biomarkers and radiology features that could support a diagnosis of COVID-19 (Appendix I). These reviews followed a Traditional Literature Review (TLR) approach  and supported the discussion of an expert panel who met on the 29th of April 2020 to identify the elements of the CRS for the diagnosis of COVID-19.
The panel comprised of:
Methodologists that are part of the National Institute for Health Research (NIHR) MedTech & In Vitro Diagnostics Co-operatives (MIC), with expertise in (from):
Clinicians with expertise in (from):
- Ambulatory Care, Hospital at Home (University Hospitals Birmingham NHS Foundation Trust and Oxford Health NHS Foundation Trust)
- Emergency Medicine (Manchester University NHS Foundation Trust)
- Infectious Diseases (The Newcastle upon Tyne Hospitals NHS Foundation Trust)
- Primary Care (Oxford Health NHS Foundation Trust)
The bias was minimised in the development of the composite reference standard by aligning our methodology to the following recommendations:
- the use of the logical `OR` operator to link tests with perfect sensitivity, and the use of the `AND` operator to link tests with perfect specificity;
- minimization of the number of tests needed;
- maximization of independency among component tests (by choosing tests that measure biologically different aspect of the disease);
- potentially assigning weights to tests in order to incorporate the information about their different clinical value and accuracy in the reference standard.
Presentation of evidence
Evidence from studies evaluating the potential different components of the reference standard were presented as the main biological categories of testing. Data were extracted from the BMJ-Best practice for COVID-19 diagnosis and identified through rapid literature reviews of primary and secondary diagnostic accuracy studies. The areas of interest were:
- Sign and Symptoms (Tables 1 and 2)
- Blood Biomarkers (Table 3)
- Radiological Examinations (Table 4)
Signs and Symptoms
Table 1 Summary of evidence for signs and symptoms which are characteristic of COVID-19
The authors could not find articles reporting diagnostic accuracy of signs and symptoms in non-hospitalized populations. These would be beneficial in informing a reference standard for the community setting and help to understand how signs and symptoms develop during the course of the disease.
The only source identified is an article by Tostmann (2020) who reported signs and symptoms from hospital staff for early detection of COVID-19 (see Table 2):
Table 2 Summary of signs and symptoms for early detection of COVID in a hospital staff population in Netherland (Table reprinted with permission from Tostmann, 2020).
During the panel, clinicians reported that these symptoms are not specific to COVID-19 and the only one more characteristic of COVID-19 (i.e. anosmia) could, in future, be subject to bias due to extensive reporting and the potential for that symptom to be caused by nasal congestion of any aetiology. Indeed, since anosmia was recognised as a characteristic feature of COVID-19 and the information was disseminated in the general news, its discriminatory value is decreasing.
Articles evaluating the diagnostic accuracy of blood biomarkers are rare. A systematic review currently ongoing within our team has highlighted that the most comprehensive review to date being is by Soraya & Ulhaq, 2020.
A summary of their laboratory findings is given in Table 3:
Table 3 Diagnostic accuracy of common biomarkers at detecting COVID-19 (Table reprinted with permission from Soraya & Ulhaq 2020).
Clinicians commented that reduced leukocytes (WBC), lymphocytes and neutrophils are very common signs of infections that will not help differentiate COVID-19 from other infectious respiratory diseases during the winter months when these diseases are prevalent. Thrombocyte is a less common finding in infection and has the potential to be useful in the differential diagnosis.
Similarly, the features on the chest X-ray and chest CT scans are non-specific of COVID-19 but Ai (2020) reports that the positive rates of rRT-PCR and chest CT for patients diagnosed with COVID-19 were 59% (601/1014) and 88% (888/1014) respectively, with an overall sensitivity of chest CT of 97% compared to PCR. Table 4 summarizes the current evidence:
Table 4 Radiological characterisation for patients with suspected COVID-19.
Ai (2020) also showed the progression of ground glass opacity and consolidation in patients over time, indicating that the prevalence of a presenting feature is dependent upon the time at which the x-ray or CT is taken during the disease progression.
The expert panel decided that, even though the proposition of a composite reference standard for COVID-19 would be interesting to support standardisation of protocols for diagnostic accuracy studies and may provide better estimates of the accuracy of novel tests, there is currently not enough robust evidence around diagnostic accuracy of signs and symptoms and biomarkers for these to be used as part of a CRS. It was also recognized that during the winter, when other respiratory diseases are circulating, these symptoms and characteristics will be less specific. Therefore, a more comprehensive CRS will be needed for the winter season, with a simpler CRS currently more acceptable due to the lower prevalence of other respiratory illnesses in the summer months.
It was concluded that the best approach at this point of the pandemic is to use a temporary CRS, which includes radiological features and rRT-PCR, to support evaluations of new diagnostic tests during the summer season when there is a low prevalence of other respiratory conditions with similar radiological signature to COVID-19. During these evaluations robust, standardized data will be collected that will allow us in early Autumn to assess statistically which clinical features are discriminative of COVID-19. Another expert panel meeting will follow to develop another CRS, which incorporates the new results on clinical features and biomarkers, to be used prospectively during the winter season evaluations when other respiratory diseases are also circulating, and radiological evidence will not be as discriminatory.
Temporary Composite Reference Standard for the summer season
The panel agreed that the following CRS could be used until more robust evidence on symptoms and biomarkers emerge:
- DEFINITE COVID-19: any positive rRT-PCR result during the course of the disease
- POSSIBLE COVID-19: negative or undetermined PCR/ELISA AND any radiological evidence of pneumonia with characteristic symptoms of COVID-19
- UNLIKELY COVID-19: negative or undetermined PCR/ELISA AND no radiological evidence of pneumonia even with characteristic symptoms of of COVID-19
It should be noted that this reference standard refers to a population of suspected COVID-19 based on a symptomatic presentation (i.e. sustained cough and/or fever).
Since a more widespread screening approach in the hospital seems to be the next stage in UK testing strategy, and with the emerging potential of new tests to capture the non-symptomatic population, it is important to include patients with positive molecular results but no symptoms characteristic of COVID-19 in diagnostic accuracy studies. It is advised to recruit and analyse this patient group separately with exploratory analysis.
The definite non-COVID-19 patients, i.e. patients with no symptoms and no rRT-PCR test positivity, are excluded in this late stage evaluation of the accuracy of the new tests because it is unlikely that future testing strategies will incorporate this population. Also, this population is usually included in earlier phases of evaluation (e.g. analytical validation studies).
Recommendations for research
When designing diagnostics accuracy studies for novel COVID-19 diagnostics, we advise researchers to collect the data, where possible, shown in Table 5 below (in addition to any molecular and serological testing results). This data combines both the observed indicators from the literature review and the listed symptoms, biomarkers and indicators laid out by ISARIC.
Table 5 Recommended key characteristics/indications to record during the next round of diagnostic accuracy studies as aligned with ISARIC Case Reporting.
In addition to the biomarkers and symptoms highlighted in the literature reviews, supplementary biomarkers and symptoms identified by ISARIC should also be captured where possible as shown in Table 6.
Table 6 Recommended additional characteristics/indications included in the ISARIC Case Reporting.
Inclusion of these additional data would allow for future design of the composite reference standard to pool data from multiple studies with the aim of improving its clinical application. Additionally, symptom onset and the time when the tests were performed must be included to allow direct comparison between studies and to gain a deeper understanding of the effect of the development of the disease and the change in the prevalence of these indicative characteristics over time.
Furthermore, as new serological testing for IgA and IgM is validated, an ELISA detection method has the potential to be used independently to confirm COVID-19 as part of a wider testing strategy in different settings. This technique could be particularly useful in the later stages of the disease when the detection rate of rRT-PCR falls.[18,19] The use of the IgG antibody as part of the reference standard needs to be carefully considered, especially if future studies confirm the presence of IgG antibodies for weeks or months after symptoms are resolved.
However, until such tests are independently validated for clinical use, the current CRS will focus solely on the existing testing methods, with the aim of incorporating serological assays for the winter season.
We proposed a roadmap for diagnostic accuracy studies for COVID-19 classification, including the definition of a composite reference standard for ongoing studies, indications for data collection and considerations around the possible introduction of antibody testing in the reference standard. Recommendations were based on literature review and elicitation of judgements from a panel of experts in diagnostic research across UK. Follow-up systematic literature reviews are being performed to update the CRS and clinical guidance going forward if new evidence emerges.
Disclaimer: the article has not been peer-reviewed; it should not replace individual clinical judgement and the sources cited should be checked. The views expressed in this commentary represent the views of the authors and not necessarily those of the host institution, the NHS, the NIHR, or the Department of Health and Social Care. The views are not a substitute for professional medical advice.
- Xiao AT, Tong YX, Zhang S. False-negative of RT-PCR and prolonged nucleic acid onversion in COVID-19: Rather than recurrence. J Med Virol 2020; doi:10.1002/jmv.25855
- Feng H, Liu Y, Lv M, Zhong J. A case report of COVID-19 with false negative RT-PCR test: necessity of chest CT. Jpn J Radiol. 2020;38(5):409‐ doi:10.1007/s11604-020-00967-9
- Green, K., Allen, A., Suklan, J et al.. What Is The Role Of Imaging And Biomarkers Within The Current Testing Strategy For The Diagnosis Of Covid-19? – CEBM [online] CEBM. Available at: <https://www.cebm.net/covid-19/what-is-the-role-of-imaging-and-biomarkers-within-the-current-testing-strategy-for-the-diagnosis-of-covid-19/> [Accessed 13 May 2020].
- Armitage A, Keeble-Ramsay D. The Rapid Structured Literature Review as a Research Strategy. Online Submission, 2009, 6(4), 27-38.
- Naaktgeboren Christiana A, Bertens Loes C M, Smeden Maarten van, Groot Joris A H de, Moons Karel G M, Reitsma Johannes B et al. Value of composite reference standards in diagnostic research BMJ 2013; 347:f5605
- bmj.com. Coronavirus Disease 2019 (COVID-19) – Symptoms, Diagnosis And Treatment | BMJ Best Practice. 2020 [online] Available at: <https://bestpractice.bmj.com/topics/en-gb/3000168> [Accessed 13 May 2020].
- Lechien, J.R., Chiesa-Estomba, C.M., De Siati, D.R. et al. Olfactory and gustatory dysfunctions as a clinical presentation of mild-to-moderate forms of the coronavirus disease (COVID-19): a multicenter European study. Eur Arch Otorhinolaryngol doi:10.1007/s00405-020-05965-1
- Wang D, Hu B, Hu C. Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus–Infected Pneumonia in Wuhan, China. JAMA. 2020;323(11):1061-1069. doi:10.1001/jama.2020.1585
- Huang C, Wang Y, Li X et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The Lancet, 2020, Volume 395, Issue 10223, 497 – 506 https://doi.org/10.1016/S0140-6736(20)30183-5
- Docherty A B, Harrison E M, Green C A et al. Features of 16,749 hospitalised UK patients with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol. MedRxiv,04.23.20076042. https://doi.org/10.1101/2020.04.23.20076042
- Rodriguez-Morales A, Cardona-Ospina J, Gutiérrez-Ocampo E et al. Clinical, laboratory and imaging features of COVID-19: A systematic review and meta-analysis. Travel Medicine and Infectious Disease 2020: https://doi.org/10.1016/j.tmaid.2020.101623
- PHE Guidelines: https://www.gov.uk/government/collections/coronavirus-covid-19-list-of-guidance [Accessed Date: April 2020]
- Chen N, Zhou M, Dong X et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study The Lancet, 2020 Volume 395, Issue 10223, 507 – 513 https://doi.org/10.1016/S0140-6736(20)30211-7
- Tostmann A, Bradley J, Bousema T et al. Strong associations and moderate predictive value of early symptoms for SARS-CoV-2 test positivity among healthcare workers, the Netherlands. Euro Surveill. 2020;25(16):pii=2000508. https://doi.org/10.2807/1560-7917.ES.2020.25.16.2000508
- Soraya GV, Ulhaq ZS. Crucial Laboratory Parameters in COVID-19 Diagnosis and Prognosis: An Updated Meta-Analysis. SSRN Electronic Journal. 2020 doi:10.2139/ssrn.3576912.
- Ai T, Yang Z, Hou H, Zhan C, et al. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology. 2020; 200642. 10.1148/radiol.2020200642
- International Severe Acute Respiratory and Emerging Infections Consortium: https://isaric.tghn.org/COVID-19-CRF/ [Accessed May 2020]
- Freeman B, Lester S, Mills L, Ata Ur Rasheed M et al. Validation of a SARS-CoV-2 spike protein ELISA for use in contact investigations and sero-surveillance bioRxiv04.24.057323; doi: https://doi.org/10.1101/2020.04.24.057323
- Liu, W., Liu, L., Kou, G., Zheng, Y., Ding, Y., Ni, W., Wang, Q., Tan, L., Wu, W., Tang, S., Xiong, Z. and Zheng, S. Evaluation of Nucleocapsid and Spike Protein-based ELISAs for detecting antibodies against SARS-CoV-2. Journal of Clinical Microbiology, 2020. DOI: 10.1128/JCM.00461-20
- Long, Q., Liu, B., Deng, H. et al. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat Med (2020). https://doi.org/10.1038/s41591-020-0897-1