Listen to the Clinical Chemistry Podcast


Article

Mei San Tang, Karl G Hock, Nicole M Logsdon, Jennifer E Hayes, Ann M Gronowski, Neil W Anderson, Christopher W Farnsworth. Clinical Performance of Two SARS-CoV-2 Serologic Assays Clin Chem 2020; 66: 1055–1062.

Guests

Drs. Mei San Tang and Christopher Farnsworth of the Department of Pathology and Immunology at Washington University in St. Louis, Missouri.


Transcript

[Download pdf]

Bob Barrett:
This is a podcast from Clinical Chemistry, sponsored by the Department of Laboratory Medicine at Boston Children’s Hospital. I am Bob Barrett.

In December 2019, a cluster of atypical pneumonia patients was detected that were epidemiologically linked to a wholesale market in Wuhan, China. In short order, a novel betacoronavirus, now known as severe acute respiratory syndrome coronavirus 2, or SARS-CoV-2, has been responsible for a worldwide pandemic, disrupting nearly all facets of daily life. Testing for the presence of the virus itself has been an important tool in attempts to identify, treat, and isolate affected individuals. But testing for the virus itself may be only half of the story. Measuring antibodies formed due to viral infection may also be useful for detecting individuals who presumably have had the disease and recovered. Within weeks of the declaration of the COVID-19 pandemic, literally hundreds of test kits for detection of antibodies to the novel coronavirus have flooded the market and just as quickly, the accuracy of some of these procedures has been called into question.

A paper appearing in the August 2020 issue of Clinical Chemistry examined two such assays in an attempt to characterize assay performance and validate the manufacturers’ claims. We are pleased to have two authors of that paper as our guests for this podcast. Dr. Mei San Tang and Dr. Christopher Farnsworth are members of the Department of Pathology and Immunology at Washington University in St. Louis. And we’ll start with you, Dr. Tang. First of all, for some of our listeners who may not be laboratorians, can you tell us a little about the differences between testing for the novel coronavirus itself, the so-called diagnostic tests, and serological or antibody tests?

Mei San Tang:
Yes. So, diagnostic tests in general are those that should be used for acute diagnosis of SARS-CoV-2. As of right now, this is done by molecular techniques, primarily PCR assays to detect RNA from the virus. So, these are usually done on nasal swabs and sometimes saliva and this assay can tell if viral particles are present in your respiratory tract, but it cannot differentiate if the detected viral particles are infectious or not. In contrast, serological assays are blood tests that detect circulating antibodies to SARS-CoV-2 and as you mentioned, antibodies are formed following viral infection. And because of the memory component of our immune system, these antibodies can circulate for months or sometimes years, but again, levels may actually reduce over time. So, the presence of these antibodies indicates previous exposure to the virus, but one of the most important differences between molecular and antibody tests is that professional societies including the CDC and the FDA, the WHO, and the Infectious Diseases Society Of America do not advocate for the use of serological assay for an acute diagnosis of COVID-19.

Bob Barrett:
Dr. Farnsworth, there has been quite a bit written in both the lay press and in scientific journals on the accuracy of serological assays for severe acute respiratory syndrome coronavirus 2, or COVID-19.  Can you tell us how your study fits into this situation and with the other literature?

Chris Farnsworth:
Sure. So, we began planning our study prior to the FDA requiring emergency use authorization for serological assays for SARS coronavirus. Since these assays were not meant to be diagnostic, the FDA allowed companies to sell their assays without thorough vetting. The underlying assumption is that the tests would be run by high complexity laboratories mainly for seroprevalence studies. However, an unintended consequence of this ruling was that dozens of assays quickly became available, most of which with very small or no studies having been performed to assess their accuracy. At the time we were planning our study, there was also nothing in the published literature that compared various commercially available serological assays for accuracy. However, there was considerable coverage in the lay press questioning the accuracy of these early serological assays, particularly lateral flow-based methods from relatively unknown manufacturers.

Bob Barrett:
You studied three commercial assays for their antibodies. Dr. Tang, why were these three assays chosen and how did you choose your study cohort to evaluate their performance?

Mei San Tang:
So the three assays we chose to characterize could be implemented on the Roche and Abbott automated platforms as well as the EUROIMMUNE plate ELISA in a semi- (00:04:32) manner and all of these instruments are readily available in our clinical lab which influence this decision. One of the fundamental considerations when choosing a cohort for validation study in the clinical lab is to think about what the intended use of that assay is and for which patient groups. This was a bit tricky for us since there was no clear guidance on how to use serology assays and because we are a hospital-based lab, we felt that our positive specimens should be randomly chosen from the patient population we serve.

Our positive cohort ended up being mostly hospitalized patients with several immunocompromised patients. Since we are also a large testing center in the region, we thought that this cohort would better reflect our patient population when compared to specimens that are collected from convalescent plasma donors which have also been commonly used in some of the later studies from other institutions. And because we are in a region where COVID-19 prevalence is relatively low, we really needed an assay with very high specificity. So, we put in quite a bit of thought in choosing our negative control specimens. We not only included specimens that were pre-pandemic from 2015, but also specimens from patients with symptoms of a respiratory infection, but were negative for COVID and other respiratory pathogens by molecular testing. We thought that this latter group was particularly important because it would allow us to assess potential cross-reactivity with respiratory pathogens that are currently prevalent and this is something that we would not be able to assess if we had only relied on pre-pandemic negative specimens. And we also had some specimens from patients who were tested positive for other coronaviruses and influenza but negative for COVID PCR.

Bob Barrett:
Your study used an evaluation protocol from CLSI, a few questions about that. First, tell us what is CLSI and why you use their evaluation procedures and what’s involved in following this?

Mei San Tang:
Sure. So, CLSI stands for Clinical and Laboratory Standards Institute. This is an international nonprofit that brings together lab experts with the goal of fostering excellence in lab medicine. So, as a result, they have developed a series of expert consensus documents that address the development and implementation of medical lab testing standards. And importantly, the standards are also recognized by the FDA. So, as Chris mentioned earlier, the EUA was not mandatory for serology assays when we were planning on how to bring them in-house and some manufacturers were marketing assays with abbreviated validation. So, we felt that it was important for us to implement a validation study that adheres to these clinical guidelines as much as possible. The CLSI document that we primarily use for our study addresses topics that were specific to validating qualitative assays. So, I just told you about how we chose our study cohort. The other major consideration when planning a validation study is to decide on the number of specimens that will be used. We analyzed in total 103 PCR positive specimens and 153 negative control specimens, and since the CLSI guidelines recommended a minimum of 50 positive and 50 negative specimens, we felt that this was a good place to start analyzing our data. This is just one example of how we use the CLSI guideline, but the document also discusses how to assess diagnostic accuracy bias and imprecision of an assay. It can take up a lot of resources to adhere to these federal guidelines, but they are important studies especially if a clinical lab plans to offer a test that is not yet approved by the FDA.

Bob Barrett:
Okay. Well, let’s get down to it. Dr. Farnsworth, what do you think are the main take-home messages from your study?

Chris Farnsworth:
So first, we found that all three commercially available assays we compared have low sensitivity early after the onset of symptoms. In fact, with the three assays we assessed, the highest sensitivity between three to seven days of symptomatology was only 40%. This is important for two reasons. First, in our hospital, the median time of presentation for patients with COVID-19 is three days from symptom onset. Therefore, the diagnostic utility of serology is very low for symptomatic patients presenting to our hospital. Secondly, there’s some evidence in literature that key interventions for patients with COVID-19 are early during infections. Together, these indicate that serological testing is not adequately sensitive at early time points to be used as a diagnostic test for SARS coronavirus 2 infections. Our second main finding is that the sensitivity of each of the assays assessed was lower than that stated in the package insert claims by the manufacturers. This is most likely due to differences in patient populations. As Mei San indicated, our hospital treats mainly acutely ill patients including those with cancer, who are immunocompromised, or have other chronic diseases. Furthermore, the vast majority have acute infections. We intentionally selected this patient population due to the concern that this assay would be used to attempt to diagnose hospitalized patients with COVID-19 infections. Finally, we observed that these assays, despite differences in end genetic target, assay design, and method, performed largely similarly.

There were minor differences in sensitivity and specificity, most of which have now been replicated by other studies. These differences may be relatively negligible depending on the patient population being tested, but we would still urge all laboratorians to select a method with sufficient test characteristics that fit the need of their patient population and how their clinicians plan to use the assay.

Bob Barrett:
You submitted your paper in May of 2020 and here we are recording this in mid-August. Your hospital has presumably started testing patients for the presence of SARS-CoV-2 antibodies. Dr. Tang, in your experience, just how are the serological assays being used by clinicians?

Mei San Tang:
Yes, so, you’re absolutely right. It has been slightly more than three months since we have started using this assay in our hospital lab. Approximately 20% of these tests that we have performed so far had been done for seroprevalence studies, and of the non-study specimens, most of them, approximately 80%, were performed on outpatients. We think that these outpatients are typically patients who are generally well, but are curious about their antibody status. Serving these curious well-patients is somewhat of a new situation for us. Since we are a hospital lab, our efforts have always been directed towards generating lab values that can be used for clinical decision making. So for example, when we test a patient for antibodies against measles or hepatitis, that information is also being used to establish presumptive immunity in the workplace or to determine if other actions are necessary, such as administering additional booster vaccines. But we’re now seeing COVID serology being ordered for patients who simply wanted to know if they’ve been exposed to the virus. With the ongoing pandemic in the news, I think this is understandable for the general public, but it’s just not a role that we are used to as hospital-based laboratorians.

Bob Barrett:
Dr. Farnsworth, have you taken any steps to reduce inappropriate ordering of SARS-CoV-2 serology, for example in patients that have been symptomatic for only two days and may not have had diagnostic molecular testing performed.

Chris Farnsworth:
So, to reiterate Mei San’s statement, as a hospital lab, one of our primary goals is to ensure that appropriate testing is available for hospitalized patients. To this end, we really had two primary concerns with serological assays. The first was that providers would attempt to use them to diagnose acute infections. To help curb this, we created an ordering interface in which clinicians were provided data for specificity and sensitivity at zero to three days, three to seven days, eight to 13 days and 14 plus days post-symptom onset based on the study that we did. The goal is to deliver education in real time during test ordering to providers so that they understood the analytic connotations of the assay. The second major concern we have is the serological assays will be used to determine if a patient had acquired immunity to the SARS-CoV-2 or an immunity passport.

To help curb this, we provided considerable education for providers including newsletters, communication through our hospital’s incident command center, and links out of the electronic medical record to help providers understand other limitations for these assays. These have been largely successful in that we anticipated copious orders for COVID serology, however only received a small daily trickle. Furthermore, assessing physician responses to the electronic medical record prompts have revealed that the vast majority are ordered on patients with more than 14 days post symptom onset when the likelihood of antibodies against SARS-coronavirus 2 are more likely to have developed. In contrast, molecular diagnostic testing still remains the primary testing modality at our institution.

Bob Barrett:
Well finally, let’s look ahead, what do you think the role for antibody testing will be in a few months or even a year from now?

Chris Farnsworth:
So, this is a really great question and really the one-million-dollar question is, if the results from serological assays can be used to infer if a patient is protected from future infections with SARS-coronavirus 2. According to the FDA and consistent with the published literature, this is not currently known. However, if this can be proven and appropriate assays shown to be a proxy for protection, then there could certainly be a role for testing patients with previously known infections for antibody concentrations to the virus, but again, to be clear, this is not currently known and I’m not necessarily advocating for this role of serology at this time. Another potential utility of serological assays could be after the emergence of vaccines to SARS-coronavirus 2. Serological assays have been used for this purpose for other viruses to confirm vaccination history. For example, we use this now for measles. Once again, this will require an optimized assay for this purpose. Finally, one current utility for serology is possibly being underutilized in patients are those with signs and symptoms of COVID-19 that are persistently negative by PCR testing. This is especially true of patients who’ve had symptoms greater than 14 days; serology can actually be helpful in this subset of patients to distinguish COVID-19 infections from other potential causes of disease. So, that’s actually one use right now where serology could have a positive impact on patient treatment.

Bob Barrett:
That was Dr. Christopher Farnsworth. He was joined by Dr. Mei San Tang. Both are from the Department of Pathology and Immunology at Washington University in St. Louis. They are co-authors of the paper describing clinical performance of two SARS-coronavirus 2 serologic assays that appears in the August 2020 issue of Clinical Chemistry. I’m Bob Barrett. Thanks for listening.