Editor’s commentary: Rapid reviews of PPE – an update

April 14, 2020

The views expressed below are our own. They do not represent those of the Oxford Centre for Evidence-Based Medicine or our employers.

Professor Trisha Greenhalgh, University of Oxford
on behalf of the COVID-19 PPE rapid review team (comprising Sebastian Straube, Anil Adisesh, Kamlesh Khunti,  Xin Hui Chan, Chris Burton, Lawrence Ross, Elaine Toomey, Declan Devane, Mike Smalle, Tanya Jackson, Briana Coles, Marta Koch, Quentin Durand-Moreau and others). 

In early March 2020, as the COVID-19 crisis began to break in the UK, the Oxford Centre for Evidence-Based Medicine set up the COVID-19 Evidence Service. We invited questions from policymakers, healthcare practitioners and others on any aspect of the disease and its prevention and management.

One of our first questions, initially posed by the Royal College of General Practitioners (RCGP), was “what is the evidence that personal protective equipment (PPE) is effective in preventing infection with COVID-19 in those working at the clinical front line?”.  The RCGP’s particular interest was for those working in primary care. They pointed out that most of the published research appeared to have been conducted in hospital settings and were concerned about the effectiveness of the PPE they had been supplied with.

As often happens when reviewers begin to research a topic, we soon realised that the question needed to be broken down into several reviews.  As well as leading the first review, I took on an editorial role for the collection as a whole. Below, I offer some reflections on the reviews we’ve done so far. But first, let me explain some context.

We were aware of various ongoing systematic reviews, including an update of a 2019 Cochrane review by Jos Verbeek’s team: “Personal protective equipment for preventing highly infectious diseases due to contact with contaminated body fluids in health care staff” [due to be published in the second week of April].(1)  Verbeek et al were covering the full package of PPE equipment, which is generally used as a full-body ensemble (for example, if healthcare workers are wearing masks, they will usually be wearing gloves, gowns, and eye protection as well). Paul Hunter’s team from University of East Anglia were working on a review of face masks for both healthcare workers and the lay public (now available as a preprint (2)). Our aim was not to duplicate the work of those teams but to produce a rapid summary of evidence as quickly as possible for policymakers and clinicians preparing to deal with COVID-19, and focusing mainly but not exclusively on primary care.

It was clear from the outset that the reviews we had been asked to produce were not being undertaken in a political vacuum. The reason why the RCGP had asked the question about PPE was that front-line primary care staff were already voicing concerns that the protection they had been given was inadequate. I was sent an unsolicited photograph (below) by a GP whose partner (a nurse) had been supplied with these very items for seeing patients with possible COVID-19. Not unreasonably, the GP wanted to know the evidence for this combination of equipment.

PPE equipment provided to clinicians in UK primary health care

The first question we tackled was “What is the efficacy of standard face masks compared to respirator masks in preventing COVID-19 type respiratory illnesses in primary care staff?”.  You can read that review here. Unsurprisingly given that the disease had only just emerged, we found no head-to-head trials at all comparing the two kinds of masks in the prevention of COVID-19. We did find some trials which appeared to show that in non-aerosol-generating procedures (AGPs), the two kinds of mask appeared to be equally effective in preventing other respiratory illnesses such as influenza. For AGPs, it was a different matter: respirator masks were more effective at preventing healthcare worker infection than standard masks.

This first review in our series was initially misinterpreted by some policymakers as evidence that current UK policy on PPE is evidence-based (and, more specifically, that we had confirmed the safety of standard masks in a primary care setting). However, that was not what we concluded from our rapid review. We found no direct evidence one way or the other, and some indirect evidence of equivalence in a different disease. The latter evidence may or may not be relevant to COVID-19. Our first review also raised an important further question: which procedures – and indeed, which other kinds of interaction with patients (for example, being coughed on) – should be counted as an AGP?

This led to a further review in our series, provisionally entitled “What counts as an AGP, and how consistent and evidence-based is current guidance on AGPs?”. Our search for evidence to answer this question unearthed well over 100 different sets of guidance or position statements around the world, sometimes citing different underpinning evidence and sometimes interpreting the same evidence slightly differently. Without pre-judging the findings from that ongoing review, I can report that there is substantial variation across jurisdictions and official bodies about what is categorised as an AGP.

Furthermore, the study of AGPs requires an understanding of the basic science of droplets (particles for which the lower-level ‘contact and droplet’ precautions are recommended) and aerosols (a suspension of tiny particles in air, for which higher-level AGP precautions are recommended). A cough, by definition, is not an aerosol generating procedure (it’s not a medical procedure at all) – but it does generate aerosols.(3, 4) Hence, the distinction between AGPs and non-AGPs, even though useful to help identify exposure controls at a systems level, is not the only consideration when assessing risk. Our team is considering introducing a new term ‘Aerosol Generating Exposures’ (AGEs) to embrace these additional risks.

We hope to publish our review of AGPs, which is being led by occupational health physician Professor Sebastian Straube from the University of Alberta, Canada, in April.

The next review we took on addressed the question: “What is the efficacy of the other personal protective equipment (PPE) that is currently offered to UK primary care staff in the context of COVID-19?”. Particular concerns of front-line clinicians, which we addressed in separate reviews, included whether eye protection (goggles, face shields and so on) should be worn routinely, whether the thin plastic aprons supplied to primary care staff (see photo) were adequate or whether long-sleeved gowns were needed, and whether COVID-19 could be transmitted by splashes of body fluids onto clinicians’ shoes.

As with our first review on masks, we turned up similar findings in all these three linked reviews. First, that there are no randomised controlled trials at all for the preventive efficacy of the particular equipment actually supplied to UK primary care clinicians (surgical mask but no eye protection; thin plastic apron; everyday shoes) in any disease. Second, that there are, at the present time, no trials at all of any equipment in the specific context of COVID-19.  And third, that the (few) randomised controlled trials published are only part of the picture.  Also relevant are the basic and applied sciences – for example, taking swabs from floors, surfaces and footwear; using fluorescent fluids to identify the extent of self-contamination when donning (putting on) and doffing (taking off) PPE; and studying the dynamics of aerosolisation under laboratory conditions.

The limited scope, inconclusive results and questionable relevance of the randomised controlled trials in our reviews to date contrasts with the relatively consistent and fairly compelling (though arguably not definitive) evidence from other study designs. SARS-CoV-2 (the virus that causes COVID-19) appears to be transmitted in splashes and to survive for extended periods of time in the air and on surfaces.(5) Self-contamination while donning and doffing PPE appears to be a significant risk. Both droplets and aerosols are generated from human coughs. These different kinds of evidence all need to be considered as contributing to the overall body of evidence on the safety of PPE.

There is another kind of evidence that is emerging in the UK. Frontline healthcare workers are dying. The numbers (35 in UK last count, but rising daily) are more than statistics.  Their names and stories are more than anecdotes. These are real people who acquired COVID-19, most likely in the course of their duties, and who spent their last hours without the hand of a loved one to hold. They served as midwives, porters, cleaners, pharmacists, surgeons, nurses, physicians, healthcare assistants, receptionists and administrators in hospitals, care homes, GP surgeries, and laboratories. They leave real bereaved families and real unfinished lives.  And however uncomfortable we find it to see their faces highlighted daily in the media, these occupational deaths have become part of the evidence base for this suite of reviews. It is neither scientifically nor morally acceptable to ignore them.

I am reminded of Steve Macguire’s ethnographic study of an infectious scourge from an earlier era: HIV/AIDS.(6) In exploring the awkward symbiosis between research, policy and the devastating effects of the disease itself, Macguire challenged the assumption there could be a simple and apolitical relationship between researchers (what he called the “evidence-producing system”), policymakers (the “evidence-adopting system”) and people with the disease or risk state. His work demonstrated that progress towards the urgently-needed diagnostic and therapeutic solutions was characterised by an awkward and conflict-ridden interdependence between these three systems. People with HIV, for example, pushed back against the scientists’ and policymakers’ demand for “gold standard” evidence from randomised controlled trials with “hard” endpoints. Your hard endpoints, said people with HIV, are our dead bodies. Don’t measure the quality of your science with our lives – and don’t hold back a change in policy until more of us have died.

As COVID-19 deaths in the UK continue to escalate, the main story about PPE in the UK, and in many other countries, has become the lack of it.  Primary and secondary care are running low on various items of PPE.  Other key workers such as porters and cleaners are, allegedly, not always being supplied with it. Staff are, they claim, being told to make theirs last longer. Some National Health Service staff claim they’ve been told to buy their own. The media is buzzing with stories of visors being 3-D printed in garden sheds, masks stitched together on kitchen sewing-machines, and small construction companies donating boxes of masks originally intended for use on building sites. Keeping the NHS in PPE has become the 21^st century’s Dunkirk.

Two weeks after we began our series of rapid reviews on PPE, we added two additional reviews to our list: “What are the challenges associated with re-use and extended use of PPE equipment in the context of COVID-19?” and “What is the evidence for the efficacy and safety of masks repurposed from other industries in protecting healthcare staff from COVID-19 infection?”.

These two reviews are ongoing, and we are working with experts from other industries. There is some good science to be summarised (and some pseudoscience to be debunked). But as we sift the evidence, it is sobering to reflect that the reason these latest two reviews are needed is at best logistical (why are the PPE supplies in the wrong place?) and perhaps also political (why wasn’t procuring and distributing adequate supplies for all groups of staff prioritised?).

I have written in the past about the inherent tensions between evidence-based medicine and policymaking.(7-9) With respect to the wearing of masks by the lay public during the COVID-19 crisis, colleagues and I have argued that the search for perfect evidence – especially from the so-called “gold-standard” randomised controlled trial – may be the enemy of good policy.(10) We therefore have to rely on the totality of the current evidence, which is more observational than experimental.

Let me summarise the evidence we have found on PPE for healthcare workers to date.  First, there is almost no direct evidence on the efficacy of PPE from research studies on COVID-19. Second, there is a lot of indirect evidence from a variety of study designs and real-world data (randomised controlled trials, natural experiments, artificial laboratory studies and more). This evidence varies in quality, and its relevance to the current outbreak is contested. Third, PPE provided to healthcare workers is in short supply and it does not always meet the minimum standards recommended by national and international bodies. Fourth, healthcare workers at the front line are dying.

I am privileged to be working with an outstanding team of reviewers with expertise in systematic review, informatics, epidemiology, occupational health, infection control, primary health care, mask design and safety standards, and more. I am confident that we will be able to assess the scientific evidence and answer the questions highlighted in bold in this article. But science is only part of the picture. There are other questions that need to be asked, and wider debates to be had. I hope that our work in the COVID PPE rapid review team helps inform those wider debates.

Footnote: This rapid review project on PPE in COVID-19 is trying to keep abreast of frequent changes in policy. We have so far had to amend our published reviews twice to reflect changes in Public Health England, World Health Organisation or Centers for Disease Control recommendations.

1. Verbeek JH, Rajamaki B, Ijaz S, Tikka C, Ruotsalainen JH, Edmond MB, et al. Personal protective equipment for preventing highly infectious diseases due to exposure to contaminated body fluids in healthcare staff. Cochrane Database of Systematic Reviews. 2019(7).
2. Brainard JS, Jones N, Lake I, Hooper L, Hunter P. Facemasks and similar barriers to prevent respiratory illness such as COVID-19: A rapid systematic review. MedRxiv. 2020. Accessed 12.4.20 at https://www.medrxiv.org/content/10.1101/2020.04.01.20049528v1.abstract;Preprint published online 6th April 2020.
3. Huynh KN, Oliver BG, Stelzer S, Rawlinson WD, Tovey ER. A new method for sampling and detection of exhaled respiratory virus aerosols. Clinical Infectious Diseases. 2008;46(1):93-5.
4. Lindsley WG, Blachere FM, Thewlis RE, Vishnu A, Davis KA, Cao G, et al. Measurements of airborne influenza virus in aerosol particles from human coughs. PloS one. 2010;5(11).
5. van Doremalen N, Bushmaker T, Morris DH, Holbrook MG, Gamble A, Williamson BN, et al. Aerosol and surface stability of SARS-CoV-2 as compared with SARS-CoV-1. New England Journal of Medicine. 2020.
6. Maguire S. Discourse and adoption of innovations: A study of HIV/AIDS treatments. Health Care Management Review. 2002;27(3):74-88.
7. Greenhalgh T, Russell J. Evidence-based policymaking: a critique. Perspectives in biology and medicine. 2009;52(2):304-18.
8. Greenhalgh T, Russell J. Reframing evidence synthesis as rhetorical action in the policy making drama. Health Policy. 2006;1(2):34-42.
9. Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? Bmj. 2014;348:g3725.
10. Greenhalgh T, Schmid MB, Czypionka T, Bassler D, Gruer L. Face masks for the public during the covid-19 crisis. Bmj. 2020;369:m1435.