Problem of inaccurate COVID tests — sensitivity, specificity and statistically based proofs

© 2020 Peter Free

 

21 April 2020

 

 

Handicapped by our own lollygagging?

 

One would have hoped that Stanford's medical school could have done better than to be forced to use an only questionably accurate Chinese antibody test to implement its study of COVID-19 infection data among Santa Clara County (California) residents.

 

That study is the below-cited one.

 

It concluded that COVID-19 is much more widespread than previously thought. And, therefore, less deadly.

 

The study is, as I said yesterday, a flawed but welcome effort to begin collecting information that is (objectively speaking) absolutely necessary to more wisely implementing American COVID-19 responses:

 

 

Eran Bendavid, Bianca Mulaney, Neeraj Sood, Soleil Shah, Emilia Ling, Rebecca Bromley-Dulfano, Cara Lai, Zoe Weissberg, Rodrigo Saavedra-Walker, Jim Tedrow, Dona Tversky, Andrew Bogan, Thomas Kupiec, Daniel Eichner, Ribhav Gupta, John P.A. Ioannidis, and Jay Bhattacharya, COVID-19 Antibody Seroprevalence in Santa Clara County, California, MedRXiv, doi: https://doi.org/10.1101/2020.04.14.20062463 (17 April 2020) (PDF here)

 

 

The questionable accuracy of the study's antibody test matters

 

Its false positive rate might account for most of the positives that the study found.

 

See:

 

 

Balaji S. Srinivasan, Peer Review of “COVID-19 Antibody Seroprevalence in Santa Clara County, California”, Medium (17 April 2020)

 

 

Dr. Srinivasan's statistics-based observation means that the study's reassuring conclusion might be mistaken.

 

David Cardinal, writing at ExtremeTech, came to same skeptical conclusion:

 

 

David Cardinal, How Deadly Is COVID-19? New Stanford Study Raises as Many Questions as It Answers, ExtremeTech (20 April 2020)

 

 

Test accuracy (and sample size) matters

 

I have noticed that relatively few people comment on the comparatively unreliable means that we have (at present) of detecting SARS-CoV-2 and antibodies to it.

 

There has been almost no Establishment emphasis on the importance of obtaining high sensitivity and specificity in these COVID-related tests.

 

Indeed at present, thanks to the relaxed standards that previous foot-dragging forced, US firms are manufacturing COVID tests that essentially are required to meet no supervised standards.

 

How that is going to turn out remains to be seen.

 

Let's delve into why arguable ambiguity like this matters.

 

 

Defining test 'sensitivity' and 'specificity'

 

Sensitivity measures the probability that a test will return as positive, when a specific disease is present.

 

Alternatively stated:

 

 

Sensitivity measures how proportionally many accurate (true) positives a test detects.

 

 

Specificity measures the probability that a patient is actually free of the specific disease being tested for.

 

Alternatively stated:

 

 

Specificity measures how proportionally many accurate (true) negatives a test detects.

 

 

See, for example:

 

 

© 2008 Rajul Parikh, Annie Mathai, Shefali Parikh, G Chandra Sekhar, and Ravi Thomas, Understanding and using sensitivity, specificity and predictive values, Indian Journal of Ophthalmology, 56(1): 45–50, doi: 10.4103/0301-4738.37595 (Jan-Feb 2008)

 

 

Necessary mathematics

 

Sensitivity = (true positives) divided by (true positives plus false negatives)

 

Specificity = (true negatives) divided by (true negatives plus false positives)

 

 

The hidden twist

 

We might think that a test that is 90-95 percent accurate is pretty darn good.

 

However, the combination of:

 

 

difficult to detect (meaning proportionally small) effects

 

and

 

comparatively small sample sizes

 

can cause even such a 'good' test to yield only indeterminate results, when trying to reach valid conclusions based upon it.

 

 

In other words, chance often can (or could) explain an individual study's entire result.

 

This is the mathematically subtle part of medical investigation that escapes most people, including most medical and science professionals.

 

It is the intellectual void that Big Pharma (for example) uses to repetitively claim to have proven favorable drug results — when it has, in actuality, not.

 

 

Using the Santa Clara County study to illustrate

 

If you are scientifically minded, first read Dr. Srinivasan's review (here) of the Santa Clara paper.

 

Afterward, read David Cardinal's plainer spoken analysis of the same study:

 

 

The researchers analyzed test results from the manufacturer and complemented them with additional testing on blood samples from Stanford.

 

Overall, they rated the sensitivity of the tests at 80.3 percent and the specificity at 99.5 percent.

 

Strikingly, though, the manufacturer’s test results for sensitivity (on 78 known positives) were well over 90 percent, while the Stanford blood samples yielded only 67 percent (on 37 known positives).

 

The study combined them for an overall value of 80.3 percent, but clearly, larger sample sizes would be helpful, and the massive divergence between the two numbers warrants further investigation.

 

This is particularly important as the difference between the two represents a massive difference in the final estimates of infection rate.

 

On sensitivity, the manufacturer’s results were 99.5 percent for one antibody and 99.2 percent for the other, on 371 samples. The tests for both antibodies performed perfectly on Stanford’s 30 negative samples.

 

Overall, Stanford estimated the test sensitivity at 99.5 percent. That’s important because if the sample population is dominated by negative results — as it is when testing the general public for COVID-19 — even a small percentage of false positives can throw things off.

 

There is some additional reason to be skeptical about the particular test used. In another pre-print, researchers from Hospitals and Universities in Denmark rated the Hangzhou-developed test last in accuracy of the nine they tested.

 

In particular, it had only an 87 percent specificity (it misidentified two of 15 negative samples as being positive). That is a far cry from the 99.5 percent calculated by Stanford . . . .

 

© 2020 David Cardinal, How Deadly Is COVID-19? New Stanford Study Raises as Many Questions as It Answers, ExtremeTech (20 April 2020)

 

 

Another Santa Clara County study methodological trait . . .

 

. . . is even more glaringly problematic.

 

The team used Facebook to obtain its population sample.

 

To see why this is a proof-tainting mistake, if that is not already obvious — and it will not be, unless you have a scientifically prone and probability-understanding mind — read Srinivasan and Cardinal's expressed skepticism.

 

 

The moral? — Test accuracy, sample sizes and investigative methodology all matter a lot

 

It is time for the United States to begin demonstrating its self-lauded competence in medicine and science.

 

With regard to COVID, I am not at all impressed with our (so far) stumbebum effort.

 

We are massively reacting to a disease that we still — apparently as a matter of intentional omission policy — know little about.

 

This ignorance occurs, in part, because we have abandoned an effort to manufacture the tools necessary to achieving nation-protecting self-sufficiency.

 

The COVID phenomenon indicates how utterly dependent Americans have allowed themselves to become on foreign nations' manufacturing and brainpower.

 

This is a ridiculous situation for a bunch of pride-filled braggarts (like us) to be in.