What prognostic clinical risk prediction scores for COVID-19 are currently available for use in the community setting?

April 22, 2020

Samuel G. Urwin1,2, Gavinder Kandola3, Sara Graziadio1,2

On behalf of the Oxford COVID-19 Evidence Service Team
Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences
University of Oxford

1NIHR Newcastle In Vitro Diagnostics Co-operative
Newcastle upon Tyne NHS Hospitals Foundation Trust, Newcastle upon Tyne, NE1 4LP

2Translational and Clinical Research Institute
Newcastle University, Newcastle upon Tyne, NE1 4LP

3Cardiology and Respiratory Investigations Unit
County Durham and Darlington NHS Foundation Trust, DH1 5TW

Correspondence to sara.graziadio@newcastle.ac.uk


VERDICT
A number of clinical risk prediction scores have recently emerged that are currently available and methodologically suitable for use in the community following a potential infection with the SARS-CoV-2 virus causing COVID-19, or people with suspected or confirmed COVID-19. For some scores, there is evidence that testing has been performed. There are also claims that some of these scores have been validated, however there is limited evidence currently available in the public domain. There are also a number of existing scores that may be suitable that were not developed for, or in response to COVID-19. Further evidence needs to emerge around the validity of all of these scores in patients with COVID-19 before recommendations on their use can be appropriately made.

BACKGROUND
Having the ability to determine the risk of adverse outcomes for people in the community following a potential infection with the SARS-CoV-2 virus may be useful as part of an exit strategy out of the current government imposed restrictions in the United Kingdom (UK). Further, being able to determine the risk of adverse outcomes for people with suspected or confirmed COVID-19 in the community may be helpful in their clinical management, as well as the wider management of resources across healthcare organisations in their locality.

A recent systematic review by Wynants et al. (2020)1 presented information on prediction models for diagnosis and prognosis of COVID-19 up to the 24th of March 2020, concluding that the proposed models are poorly reported and at a high risk of bias. Our specific focus in this rapid review was to provide a quick reference summary of prognostic clinical risk prediction scores that are currently available in an accessible format and methodologically suitable for use in the community following a potential infection with the SARS-CoV-2 virus causing COVID-19, or people with suspected or confirmed COVID-19.

CURRENT EVIDENCE
A pragmatic, non-systematic search of the literature and online sources was performed on the 14th of April 2020. PubMed2 and the Google3 search engine were used.

Clinical risk prediction scores were selected based on meeting the following inclusion criteria:

  • Clinical risk prediction scores using data that could be collected by health care professionals (HCPs) in the community (i.e. demographics, comorbidities, vital signs etc. Parameters derived from blood or other substances were excluded due to the unavailability of relevant equipment/tests in the community);
  • Clinical risk prediction scores that were presented in an accessible format for use in clinical practice (i.e. a web application, equation etc).

Identified clinical risk prediction scores were stratified into two categories:

  • New scores developed specifically for, or in response to, COVID-19;
  • Existing scores that could be applied to COVID-19.

New scores developed for or in response to COVID-19

Table 1 presents a quick reference summary of the clinical risk prediction scores identified specifically for, or in response to COVID-19 (see the text for further details).

Table 1 – A quick reference summary of clinical risk prediction scores identified specifically for, or in response to, COVID-19

ScoreCOVID-19 data*PurposeSample size**Validated
Coronavirus Mortality Risk Calculator4NoMortality risk50,310No
COVID-19 Prognostic Tool5YesMortality riskChina: 44,672;

USA: 4,226

No
COVID-19 Vulnerability Index6NoComplications risk1,481,654No
Shi et al. (2020) Risk Score7YesSeverity risk487Yes
Surgisphere Mortality Risk Calculator8YesMortality risk4,296No
Surgisphere Severity Scoring Tool9YesSeverity  risk13,500No
Surgisphere Triage Decision Support Tool10YesTriage recommendation>10,000No

*Whether the score was developed using data from people with COVID-19
**The total sample size used in the development of the score, excluding testing or validation

Coronavirus Mortality Risk Calculator

Patient/near patient contact required: No

Developed: The ‘Coronavirus Mortality Risk Calculator4’ was developed by i5 Analytics. For high volume data processing, a representational state transfer (REST) application programming interface (API) is available for public health authorities and governments.

Purpose: Calculates mortality risk classification (output: categorical).

Variables:

  • Current signs and symptoms: N/A     
  • Comorbidities/medical history: Infectious diseases; blood or endocrine issues; heart problems; breathing or vision difficulties; mental and behavioural disorders; digestive or genitourinary issues; musculoskeletal and skin issues; current or previous cancers; any other symptoms; other health related issues
  • Demographics: Age

Manuscript: Braun et al. (2020)11 (not peer reviewed).

Summary: Braun et al. (2020)11 described that the ‘Coronavirus Mortality Risk Calculator’ was developed using Artificial Neural Networks (ANN) to calculate the mortality risk for patients infected with influenza or human coronavirus using data from April 2016 to March 2019. The ANN used 50,310 records (medium severity: 22,005; high severity: 27,053; died: 3,216) for training. It was tested on 5,630 records (medium severity: 2,510; high severity: 2,983; died: 327), and had a sensitivity of 80.1% across all three categories resulting in a misclassification error of 19.9%, and specificity of 78.2%. Braun et al. (2020)11 also described that it was validated on 8,423 records (medium severity: 3,755; high severity: 4,463; died: 489), however no results are presented.

COVID-19 Prognostic Tool

Patient/near patient contact required: No

Developed: The ‘COVID-19 Prognostic Tool5’ was contributed by Benjamin Mammon, and is hosted on the QxMD platform.

Purpose: Calculates mortality risk (output: percentage).

Variables:

  • Current signs and symptoms: N/A
  • Comorbidities/medical history: Cardiovascular disease; diabetes; chronic respiratory disease; hypertension; cancer; stroke; heart disease; chronic kidney disease
  • Demographics: Age

Publication: None.

Summary: The QxMD website5 described that the ‘COVID-19 Prognostic Tool’ was developed using both Chinese and American data, from the Centers for Disease Control and Prevention (CDC) Interim Clinical Guidance for Management of Patients with Confirmed Coronavirus Disease (COVID-19)12 to calculate mortality risk. The score does not output a single mortality risk that takes into account multiple variables, but rather presents mortality risk separately for the Chinese and American data, in addition to a cardiovascular disease and chronic respiratory disease specific mortality rate from Chinese Data.

The QxMD5 website described that the Chinese data was taken from a report by the Chinese Center for Disease Control and Prevention13. In this report, a total of 44,672 cases were confirmed as COVID-19 (through positive viral nucleic acid throat swab samples). The overall mortality rate was 2.3% (1023 deaths in 44,672 confirmed cases).

The QxMD5 website also described that the American data was taken from a CDC report entitled ‘Severe Outcomes Among Patients with Coronavirus Disease 2019 (COVID-19)14’. In this report, a total of 4,226 COVID-19 cases in the United States that occurred between the 12th of February and 16th of March 2020 were analysed by age group.

COVID-19 Vulnerability Index

Patient/near patient contact required: No

Developed: The ‘COVID-19 Vulnerability Index6’ was developed by Dave DeCaprio, Joseph Gartner, Thadeus Burgess, Sarthak Kothari, Shaayan Sayed, and Carol J. McCall.

Purpose: Calculates risk of vulnerability to severe complications (output: relative risk).

Variables:

  • Current signs and symptoms: Fever; shortness of breath; cough; fatigue; body aches; headache; diarrhoea; sore throat; decrease in ability to smell or taste; height; weight
  • Comorbidities/medical history: Chronic obstructive pulmonary disease (COPD), emphysema or chronic bronchitis; asthma; cystic fibrosis; hypertension; myocardial infarction; stroke; coronary atherosclerosis or other heart disease; congestive heart failure; rheumatic heart disease; chronic kidney disease; liver disease; cancer; neurocognitive conditions; sickle cell anaemia; human immunodeficiency virus (HIV) infection; transplant; haemodialysis;  pneumonia, acute bronchitis, influenza or other acute respiratory infection; number of admissions to hospital; number of times received treatment in an Emergency Department
  • Demographics: Age; sex; country; partial postcode

Publication: DeCaprio et al. (2020)15 (not peer reviewed).

Summary: DeCaprio et al. (2020)15 described that the ‘COVID-19 Vulnerability Index’ was developed using machine learning methods applied to an anonymised 5% sample of Medicare claims data from 2015 and 2016 to determine the risk of vulnerability to severe complications from COVID-19. The models used 1,481,654 records for training, and 369,865 for testing. A gradient boosted trees model produced an area under the receiver operating curve (AUROC) of 0.81, and a sensitivity of 0.231 and 0.327 at a 3% and 5% alert rate, respectively.

Shi et al. (2020) Risk Score

Patient/near patient contact required: No

Developed: The ‘Shi et al. (2020) Risk Score7’ was developed by Yu Shi, Xia Yu, Hong Zhao, Hao Wang, Ruihong Zhao, and Jifang Sheng

Purpose: Calculates severity risk (output: categorical).

Variables:

  • Current signs and symptoms: N/A
  • Comorbidities/medical history: Hypertension
  • Demographics: Age, sex

Publication: Shi et al. (2020)7

Summary:

Shi et al. (2020)7 describe that the ‘Risk Score’ was developed to establish a scoring system to identify people at risk of developing severe symptoms. A total of 487 patients with COVID-19 were included in analysis, with 438 mild (89.9%) and 49 (10.1%) severe cases at admission. In a multivariate analysis, elder age, male sex, and presence of hypertension were independently associated with severe disease at admission, and a risk score was developed using these variables which outputs a value from 0-3. The authors validated the risk score on 66 patients and found that 8.3% of people with a score of 0, 13.8% with a score of 1, 38.9% with a score of 2, and 42.9% with a score of 3 went on to develop severe symptoms.

Surgisphere Mortality Risk Tool

Patient/near patient contact required: No

Developed: The ‘Surgisphere Mortality Risk Tool8’ was developed by Surgisphere.

Purpose: Calculates mortality risk (output: percentage).

Variables:

  • Current signs and symptoms: N/A
  • Comorbidities/medical history: Hypertension; diabetes; current smoker; coronary artery disease or myocardial infarction; heart failure, COPD, chronic kidney disease, kidney failure or dialysis; cirrhosis or chronic liver disease; dementia; cancer; HIV or acquired immune deficiency syndrome (AIDS)
  • Demographics: Age

Publication: None (a paper has been submitted and is under review as of the 15th of April 2020).

Summary: The Surgisphere website16 described that the ‘Surgisphere Mortality Risk Tool’ was developed using data from Surgisphere’s real time global research network comprised of more than 1,200 healthcare organisations. A total of 6,103 patients with polymerase chain reaction (PCR) confirmed COVID-19 infection were evaluated, of which outcomes (survival vs. death) were known in 4,296 patients.

Surgisphere Severity Scoring Tool

Patient/near patient contact required: Yes

Developed: The ‘Surgisphere Severity Scoring Tool9’ was developed by Surgisphere in partnership with the African Federation for Emergency Medicine (AFEM).

Purpose: Calculates severity classification (output: categorical).

Variables:

  • Current signs and symptoms: Shortness of breath; temperature; heart rate; respiratory rate; oxygen saturation; mental status; auscultation
  • Comorbidities/medical history: Cardiovascular disease; hypertension; diabetes; COPD or asthma; current smoker; immunocompromised; tuberculosis; mobility or activity
  • Demographics: Age

Publication: None.

Summary: Wallis (2020)17 described that the ‘Surgisphere Severity Scoring Tool’ uses machine learning on hospital-based data from 13,500 COVID-19 patients. The score assigns patients into a severity category that aligns to the World Health Organisation’s (WHO’s) classification of critical, severe, or moderate/mild. By definition, critical patients require ventilation; severe patients require oxygen; moderate patients have pneumonia but no oxygen need, and mild patients only have upper respiratory tract disease. An early assessment suggests that the tool correctly classifies 93.6% of patients, overestimating 5.7% and underestimating 0.8% of patient severities (Wallis (2020)17).

Surgisphere Triage Decision Support Tool

Patient/near patient contact required: Yes

Developed: The ‘Surgisphere Triage Decision Support Tool10’ was developed by Surgisphere.

Purpose: Calculates triage recommendation (output: categorical).

Variables:

  • Current signs and symptoms: Dyspnea; mental status; temperature; heart rate; respiratory rate; auscultation
  • Comorbidities/medical history: Cardiovascular disease; hypertension; diabetes; COPD; current smoker; immunocompromised; mobility or activity
  • Demographics: Age

Publication: None (a paper has been submitted and is under review as of the 15th of April 2020).

Summary: The Surgisphere website18 described that the ‘Surgisphere Triage Decision Support Tool’ was developed using data from Surgisphere’s real time global research network comprised of more than 1,200 healthcare organisations. More than 10,000 patients with PCR-confirmed COVID-19 infection were evaluated. The score outputs a category of either ‘Critical’ (Immediate medical attention is required. There is a high risk of major morbidity and death), ‘Urgent’ (Evaluate promptly by a physician. There is a high likelihood that further treatment will be needed to avoid morbidity and possible mortality), or ‘Routine’ (Regularly monitor for signs of progression in a controlled environment. Additional medical attention may be needed). The score correctly classifies 95.5% of patients with regard to urgency of care. Patients with critical symptoms who required immediate medical attention were correctly classified 100% of the time (positive predictive value (PPV) and negative predictive value (NPV)). Patients in the urgent vs. routine categories were correctly classified approximately 94% of the time. The Surgisphere website18 described that the model has been prospectively validated, and that details will be made available along with a copy of the final manuscript once it is accepted for publication.

Existing scores

In addition to new scores developed specifically for, or in response to COVID-19, some existing scores may be useful for patients with suspected/confirmed COVID-19.

NICE guideline (NG 165) ‘COVID-19 rapid guideline: managing suspected or confirmed pneumonia in adults in the community19 suggests that the ‘CRB6520’ score could be used in patients with suspected COVID-19. NICE guideline (NG191) ‘Pneumonia in adults: diagnosis and management21 recommends the CRB65 score, although it has not been validated in patients with COVID-19.

NICE guideline (NG 165)19 also suggests that the use of the ‘NEWS222’ score in the community for predicting the risk of clinical deterioration may be useful, although it has not been validated in patients with COVID-19.

A further tool/model which may be useful is ‘qSOFA23’, which identifies high-risk patients for in-hospital mortality with suspected infection outside of the Intensive Care Unit, although it has not been validated in patients with COVID-19.

CONCLUSIONS

  • Seven clinical risk prediction scores were identified that are currently available and methodologically suitable for use in the community. Some of these scores have evidence that testing has been performed. Some of these scores claim that validation has been performed, however limited evidence was available in the public domain.
  • Three existing clinical risk prediction scores were identified that are currently available and methodologically suitable for use in the community. These were not developed for, or in response to COVID-19, and therefore require validation in people with COVID-19.
  • Further evidence needs to emerge around the validity of all of these scores in patients with COVID-19 before recommendations on their use can be appropriately made.

Disclaimer:  the article has not been peer-reviewed; it should not replace individual clinical judgement and the sources cited should be checked. The views expressed in this commentary represent the views of the authors and not necessarily those of the host institution, the NHS, the NIHR, or the Department of Health and Social Care. The views are not a substitute for professional medical advice.

SEARCH TERMS
A pragmatic, non-systematic search of the literature and online sources was performed on the 14th of April 2020. PubMed2 and the Google3 search engine were used.

The following terms were used in PubMed2 to search for relevant literature in the title and abstract: “(’risk’) AND ((‘tool’) OR (’model’) OR (‘score’)) AND (‘COVID-19’)”. A systematic review by Wynants et al. (2020)1 on prediction models for diagnosis and prognosis of COVID-19 searched up to the 24th of March 2020, so we restricted our search results between the 24th of March 2020 and 14th of April 2020, using the paper as a resource for literature and some online resources before the 24th of March 2020.

The following terms were used to search Google3 to form the basis of searching for relevant online resources: “COVID-19 clinical risk prediction model”, “COVID-19 clinical risk prediction tool”, “COVID-19 risk score”, “COVID-19 risk prediction”, “COVID-19 predictive tool” and “COVID-19 predictive model”. Links to other online resources from the main search results were followed where potentially relevant.

REFERENCES

  1. Wynants L, Van Calster B, Bonten MMJ, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328.
  2. National Center for Biotechnology Information. PubMed. https://pubmed.ncbi.nlm.nih.gov/. Accessed 14/04/2020.
  3. Google. Google Search. https://www.google.com/. Accessed 14/04/2020.
  4. i5 Analytics. Coronavirus Risk Calculator. https://www.coronavirusrisk.org/risk-calculator/. Accessed 14/04/2020.
  5. Mammon B. COVID-19 Prognostic Tool. https://qxmd.com/calculate/calculator_731/covid-19-prognostic-tool. Accessed 14/04/2020.
  6. DeCaprio D, Gartner J, Burgess T, Kothari S, Sayed S, McCall C. COVID-19 Vulnerability Index. https://closedloop.ai/c19index/. Accessed 14/04/2020.
  7. Shi Y, Yu X, Zhao H, Wang H, Zhao R, Sheng J. Host susceptibility to severe COVID-19 and  establishment of a host risk score: findings of 487 cases outside Wuhan. Critical Care. 2020;24:1-4.
  8. Surgisphere. COVID-19 Mortality Risk Calculator. https://surgisphere.com/research-tools/mortality.php. Accessed 14/04/2020.
  9. Surgisphere. COVID-19 Severity Scoring Tool. https://surgisphere.com/research-tools/severity.php. Accessed 14/04/2020.
  10. Surgisphere. Triage Decision Support Tool. https://surgisphere.com/research-tools/triage.php. Accessed 14/04/2020.
  11. Braun H, Patterson D, Molloy A, Davies K. Predicting Mortality Risk in Patients with Coronavirus or Influenza using Artificial Intelligence. 1-10.
  12. Centers for Disease Control and Prevention. Interim Clinical Guidance for Management of Patients with Confirmed Coronavirus Disease (COVID-19). https://www.cdc.gov/coronavirus/2019-ncov/hcp/clinical-guidance-management-patients.html. Accessed 14/04/2020.
  13. Wu Z, McGoogan JM. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72 314 Cases From the Chinese Center for Disease Control and Prevention. JAMA. 2020.
  14. Centers for Disease Control and Prevention COVID-19 Response Team. Severe Outcomes Among Patients with Coronavirus Disease 2019 (COVID-19). 2020.
  15. DeCaprio D, Gartner J, Burgess T, Kothari S, Sayed S, McCall C. Building a COVID-19 Vulnerability Index. arXiv.org, 2020:1-9.
  16. Surgisphere. COVID-19 Mortality Risk Calculator details. https://surgisphere.com/covid-19-mortality-risk-assessment-tool/. Accessed 14/04/2020.
  17. Wallis LA. COVID-19 Severity Scoring Tool for low resourced settings. Afr J Emerg Med. 2020.
  18. Surgisphere. Triage Decision Support Tool details. https://surgisphere.com/covid-19-triage-tool/. Accessed 14/04/2020.
  19. National Institute for Health and Care Excellence. COVID-19 rapid guideline: managing suspected or confirmed pneumonia in adults in the community. 2020.
  20. Lim W, van der Eerden M, M, Laing R, et al. CURB-65 Score for Pneumonia Severity. https://www.mdcalc.com/curb-65-score-pneumonia-severity. Accessed 14/04/2020.
  21. National Institute for Health and Care Excellence. Pneumonia in adults: diagnosis and management. 2019.
  22. Royal College of Physicians. National Early Warning Score (NEWS) 2. https://www.mdcalc.com/national-early-warning-score-news-2. Accessed 14/04/2020.
  23. Seymour C, W, Liu V, X, Iwashyna T, J, et al. qSOFA (Quick SOFA) Score for Sepsis. https://www.mdcalc.com/qsofa-quick-sofa-score-sepsis. Accessed 14/04/2020.