Question: Should the Roth score be used in the remote assessment of patients with possible COVID-19? Answer: No.

April 2, 2020

Professor Trisha Greenhalgh

On behalf of the Oxford COVID-19 Evidence Service Team
Nuffield Department of Primary Care Health Sciences
University of Oxford

Correspondence to trish.greenhalgh@phc.ox.ac.uk

VERDICT
Do not use the Roth score to assess breathlessness over the phone.

• There are no validated tests for assessing breathlessness over the phone in an acute primary care setting
• Measuring a patient’s respiratory rate over the phone using the Roth Score does not provide an accurate assessment of hypoxia and may lead to false reassurance
• Experts recommend an overall clinical assessment, including questions about the nature and rate of change of the breathlessness

BACKGROUND
The COVID crisis is requiring us to manage patients with as little in-person contact as possible. Remote assessment of a patient with respiratory problems is problematic, and clinicians understandably seek tools that make such assessment as objective and straightforward as possible. This is particularly important in unwell patients with COVID-19, since hypoxia is a serious warning sign for severe pneumonia,¹ and most patients do not have accurate oximeters at home. A previous rapid review by our team on assessing shortness of breath in remote consultations² found that most scores had been developed for the long-term monitoring of chronic respiratory illness in outpatient settings and were unsuitable for the acute setting. It also turned up a single study describing a test (the Roth score) for assessment of acute breathlessness.³

We originally stated that we did not recommend using the Roth score (because it was not validated in acute primary care settings, and because experts felt that it could cause harm), but that it would merit further evaluation. We also stated that assessment of the patient with shortness of breath should include a careful history with a series of questions (reproduced below) and preferably a video assessment.

Despite that very cautious recommendation, the Roth score was rapidly reproduced and circulated on social media, professional distribution lists and both local and national guidelines. The cautions associated with our original advice were not circulated. On the contrary, the Roth score was widely and quickly recommended as an “objective” test for hypoxia. It appears to have been used by some clinicians as a substitute for a full history and remote video examination.  We have received reports of significant events in which deteriorating patients were falsely reassured because of a “normal” Roth score.

In this paper, we reiterate our analysis of the Chorin paper, highlighting problems with both internal validity (can we trust the study?) and external validity (does it apply to the current clinical situation?).

WHAT IS THE ROTH SCORE?
The Roth Score is a tool for quantifying the level of breathlessness, which is assumed to correlate to the level of hypoxia. It combines maximal count reached (starting from 1 to 30 in one’s native language) during a single exhalation and the time taken to reach the maximum count (the second score is called the “counting time”).

The Roth score, named after the cardiologist who developed it, has been the subject of a single validation study against pulse oximetry in a small study from Israel, conducted on a coronary care unit.³ The patient group consisted of 93 people (53 men, 40 women) with a mean age of 76 +/- 13 years. Admission diagnoses were congestive heart failure exacerbation (25%), pneumonia (17%), and acute coronary syndrome (15%). The control group consisted of 103 healthy volunteers (64 men, 39 women) with mean age of 56 +/- 18 years.

The study showed a strong positive correlation between the pulse oximetry measurement on room air and both the maximal count achieved in 1 breath (r = 0.67; P < 0.001) and the counting time (r = 0.59; P < 0.001). All individuals in the control group counted to at least 15 in 1 breath, and 97 (94%) counted to at least 20. The scores plotted on a receiver operating characteristics (ROC) curve and the AUC was calculated. Counting time <8 seconds had a sensitivity of 78% and specificity of 71% for identifying a room-air pulse oximetry <95%.

Whilst these findings confirm that the Roth score is correlated with the level of hypoxia, they do not show that the test is either valid or safe. We critique the paper below.

CRITICAL APPRAISAL OF THE ROTH TEST PAPER BY CHORIN ET AL³
Was there an independent, blind comparison with a reference (gold) standard of diagnosis?
The Roth test was compared with pulse oximetry. This is not the gold standard (arterial blood gas would be better), but pulse oximetry correlates fairly well with ABG.

Was the diagnostic test evaluated in an appropriate spectrum of patients (like those a clinician would see in practice)?
Not in relation to the use case we are considering (assessment of ?COVID-19 patients over the phone). First, nobody in the sample had COVID-19. Second, nobody with a SpO₂ reading below 92% was included. Third, the patients in the test sample had already been admitted to hospital (i.e. the diagnostic decision to admit had already been made). Fourth, most were cardiac, not respiratory, patients. Fifth, they were assessed at the bedside, not over the phone. This validation sample compares poorly with COVID-19 patients (who may be profoundly hypoxic) assessed in a primary care setting.

Was the test validated in a second independent group of patients?
It was validated in healthy volunteers.

How accurate is the test?
Even under the conditions in which it was validated, the Roth score (as measured in terms of a counting time of <8 seconds – the so-called “8 second test”) will wrongly classify one hypoxic patient in five as normal. In addition, the Roth score is frequently abnormal in patients who are not hypoxic. In the context of a COVID pandemic, this would lead to such patients being sent to a ‘hot hub’ or a hospital unnecessarily, thereby exposing them to the risk of contagion.

CONCLUSIONS

• Even under the conditions in which it was tested, the Roth test is not sufficiently accurate to confidently classify people as hypoxic or not on the basis of an 8-second counting test
• The Roth test was validated under very different conditions from those in which it would be used in COVID-19 assessments
• We therefore don’t recommend it.

WHAT TEST SHOULD BE USED INSTEAD OF THE ROTH TEST?
Pending further research, the recommendations below are based on expert opinion. A rapid survey of 50 clinicians who regularly assess patients by phone (on 20.3.20) recommended not using the Roth score (though opinions were mixed) and gave the following advice:

1. Ask the patient to describe the problem with their breathing in their own words, and assess the ease and comfort of their speech. Ask open-ended questions and listen to whether the patient can complete their sentences.

“How is your breathing today?”

2. Align with NHS111 symptom checker, which asks three questions (developed through user testing but not evaluated in formal research):

“Are you so breathless that you are unable to speak more than a few words?”

“Are you breathing harder or faster than usual when doing nothing at all?”

“Are you so ill that you’ve stopped doing all of your usual daily activities?”

3. Focus on change. A clear story of deterioration is more important than whether the patient currently feels short of breath. Ask questions like

“Is your breathing faster, slower or the same as normal?”

“What could you do yesterday that you can’t do today?”

“What makes you breathless now that didn’t make you breathless yesterday?”

4. Interpret the breathlessness in the context of the wider history and physical signs. For example, a new, audible wheeze and a verbal report of blueness of the lips in a breathless patient are concerning.

In addition, as advised in the BMJ ‘Ten minute consultation’ on the remote assessment of a patient with possible COVID-19, we recommend that a video examination will add key detail such as whether the patient is blue, the extent of respiratory effort and the opportunity to count the respiratory rate.¹

Disclaimer:  the article has not been peer-reviewed; it should not replace individual clinical judgement and the sources cited should be checked. The views expressed in this commentary represent the views of the authors and not necessarily those of the host institution, the NHS, the NIHR, or the Department of Health and Social Care. The views are not a substitute for professional medical advice.

AUTHOR

Trish Greenhalgh is a Professor of Primary Care Health Sciences, co-Director of the Interdisciplinary Research In Health Sciences (IRIHS) unit, and joint module coordinator on the Knowledge Into Action (KIA) module of the MSc in Evidence Based Health Care.

REFERENCES

1. Greenhalgh T, Koh GCH, Car J. Covid-19: a remote assessment in primary care. Bmj 2020;368:m1182. doi: 10.1136/bmj.m1182 [published Online First: 2020/03/28]
2. Greenhalgh T, Kotze K, Van Der Westhuizen H-M. Are there any evidence-based ways of assessing dyspnoea (breathlessness) by telephone or video? : Oxford COVID-19 Evidence Service Rapid Review. 30th March 2020. Accessed 1.4.20 at https://www.cebm.net/covid-19/are-there-any-evidence-based-ways-of-assessing-dyspnoea-breathlessness-by-telephone-or-video/.
3. Chorin E, Padegimas A, Havakuk O, et al. Assessment of Respiratory Distress by the Roth Score. Clinical cardiology 2016;39(11):636-39.