Three Ways Not To Analyze COVID-19 Statistics

The COVID-19 pandemic and lockdowns continue to cause unprecedented devastation of everyday life in the United States--approximately 100,000 deaths, tens of millions unemployed, and countless plans, activities, and goals put on an indefinite hold.

In this context, news outlets, politicians, and consumers are closely following the trends in the COVID-19 statistics, trying to answer the most pressing questions. Are things getting better or worse in the US? Have we succeeded in the flattening the curve? Are the reopened states seeing a surge in new cases that many have feared?

These are important questions. Unfortunately, much of the reporting on the COVID-19 data obfuscates the underlying reality. In most cases, the problem is not that the reporting is literally false. But it typically focuses on the wrong metrics and fails to account for the severe limitations in the underlying data. The end result is that readers--and perhaps policymakers--come away with a more optimistic or pessimistic understanding than is actually warranted.

With that in mind, here are three errors to watch out for in discussions on COVID-19 data.

1. Focusing on the number of newly reported positive cases

This problem has become more common, particularly since some states have started to reopen. Here are some examples of recent headlines that commit this error:

Virginia Reports Highest One Day Increase in Coronavirus Cases After Gov. Ralph Northam Criticized For Not Wearing Mask - Newsweek, 5/25/2020

Texas sees highest single-day hike in coronavirus deaths, cases - Texas Statesman, 5/14/2020

Intuitively, it seems like the number and trend of newly confirmed COVID-19 cases must be an important number. But by itself, it doesn't tell us much at all. To properly understand it, we also need to know the number and trend in total COVID-19 tests conducted over the same period.

As an illustrative example, let's consider two random days of test results from Virginia. All results that follow are originally sourced from The Atlantic's COVID Tracking Project:



With these facts alone, it would appear May 25 was a much worse day than April 13 for Virginia when it comes to the coronavirus. Over three times as many people were confirmed as positive. Surely, must mean the virus was spreading wider and was more out of control on May 25--after the reopening--than it was in mid-April during the lockdown, right?

Well, not quite. When we add the context of the number of tests performed and the positivity rate (the rate of positive tests out of total test results reported), a very different picture emerges. See below:



From this, we can see a more compelling explanation for why positive tests on April 13 were so much lower--namely, far fewer tests were conducted.

Based on these figures, there's very good reason to assume the virus situation was actually worse on April 13. The high rate of positives suggests that they were unable to test enough people. So if they had had enough resources to test all suspected individuals, it's likely that the number of positives would have been much higher.

But if you only focus on the positive cases, this reality gets completely turned on its head.

A similar version of this general error can be observed in many reports on record increases in daily cases. Confirmed cases are indeed continuing to rise throughout the US. But the good news is that in most places, the total number of tests is rising at an even faster clip.

2. Focusing on the percentage growth rate (or the doubling rate) of confirmed cases

A related analytical error gets made when media outlets report on the percentage growth rate. Examples of this error in the wild can be routinely found in Bloomberg Radio news updates. Last week, they were reporting around a 1.1% increase in cases, which varied slightly depending on the day.

For a print example, I offer this highly neutral take from Willamette Week in Oregon from May 21, "A Rise in COVID-19 Cases in Deschutes County Tests Whether the State Will Close Bars, Restaurants Again. (So Far? No.)":

The number of COVID-19 cases in Deschutes County has increased over the last seven days. On Wednesday, the county reported nine cases—more cases than it has on any other single day. 

Those increases raise the question of whether the state will order the county to shut down the bars, restaurants and hair salons that reopened just six days ago...

A 5 percent increase in COVID cases is the benchmark the state set for reviewing the status of a county and possibly shuttering it again. [Health Researcher Numi Lee] Griffith pointed to a 27 percent increase in cases in Deschutes County during the week ending May 20. (emphasis added)

This article is interesting for a couple reasons. First, we see that it actually starts out by committing error #1, reporting a record increase of nine cases without providing information about the number of tests.

(Later on, the article even notes that many of the new cases were actually identified proactively through contact-tracing rather than simple symptomatic testing. If anything, that's actually a positive indication about the county's preparedness to mitigate the virus, not a cause for alarm.)

But I digress. The key points in the Willamette Week article are that a) Oregon has actually built this metric into its reopening guidelines and b) Deschutes would have violated it with a 27% increase.

The reason people tend to focus on the growth rate (or in some cases, the days-to-doubling) is because we know that the virus naturally spreads at an exponential rate. One person gives it to three people who each give it to three more people and so on.

In theory, the growth rate is useful because it could offer a window into how quickly the virus is spreading currently, and whether the curve has been sufficiently flattened.

But here's the problem. One of the key features that makes COVID-19 harder to deal with is that many people who contract the virus, experience no symptoms at all. And while this is not entirely proven, it's generally believed that these asymptomatic individuals are still contagious and thus contribute to the exponential spread of the disease.

The challenge is that testing capacity has been so limited that states have not been able to conduct the kind of widespread random testing that would be needed to identify all of the asymptomatic cases. The other way to plausibly identify all or most asymptomatic cases is through a robust contact-tracing system like that of South Korea or Taiwan. But the US's capabilities here are still limited. Instead, COVID-19 testing around the country has been prioritized for people with symptoms and healthcare workers.

The upshot of all this is that the growth rate is not a useful proxy for the thing we're actually trying to measure. What we want to know is the true rate of spread for the virus, in real-time. But due to testing limitations, the growth rate mostly reflects a) the growth rate in testing capacity and b) the growth rate in symptomatic patients.

This error actually cuts in both directions. Early on in the COVID-19 crisis in February--when the CDC was hard at work developing a faulty test and the FDA was simultaneously preventing others from creating a better one--the nation was testing virtually no one. So most metrics looked good.

Then in mid-March as testing capacity finally got built out, the number of positive cases quickly exploded. Positive cases were doubling every two to three days, as this chart shows:



And then, starting in mid-April and continuing to the present, the growth rate and doubling rate slowed back down. Perhaps this can be partly explained by the voluntary precautions and the lockdowns. But clearly, the more important driver is this: While the virus may grow exponentially, US testing capacity does not.

At each point in the process, including today, these metrics have not been meaningful in the US. In March, they offered a belated confirmation that the virus was already spreading widely. And now, they suggest that virus is slowing down, in part because testing capacity can only grow so fast.

3. Citing the case fatality rate as a meaningful statistic

As its name implies, the case fatality rate (CFR) is calculated by taking the total number of deaths attributed to COVID-19 and dividing by the total number of confirmed cases. The calculation is straightforward, and but the result is worse than useless in the case of COVID-19, as we'll see.

The most high profile example of bad reporting on the CFR comes from the World Health Organization, whose director said this on March 3:

Globally, about 3.4% of reported COVID-19 cases have died. By comparison, seasonal flu generally kills far fewer than 1% of those infected.

This shockingly high 3.4% figure was used as one of the reasons to justify widespread lockdowns. And yet, the statement itself offers a clue about the problems with this metric.

In that quote, the WHO is comparing the then-calculated CFR of COVID-19 to the infection fatality rate (IFR) of seasonal influenza. These are not the same metric.

In effect, the CFR is what we can easily observe and calculate. The IFR is what we actually care about, but it's harder to determine. The difference between the two metrics is the denominator. The CFR divides by total confirmed cases, and the IFR divides by total infections.

Since confirmed cases are a subset of total infections, the CFR will always be higher this the IFR. This doesn't mean that COVID-19 is the same as the flu. But it does mean that comparing the CFR of one disease to the IFR of another is unlikely to provide useful information.

To be fair, it's conceivable that the gap between the CFR and IFR will not be significant for some diseases. If there was a well-known disease and widespread testing was available, it's likely that the number of confirmed cases would approximate the total number of infections and thus the CFR would be close to the IFR. However, this is not remotely true for COVID-19 now, and it was even less true at the beginning of March.

For COVID-19, there have been testing shortages all over the world, with a few exceptions. As a practical matter, this meant that tests were generally prioritized for people with severe symptoms and healthcare workers. This prioritization was necessary to try to treat patients more effectively and reduce spread in the hospital environment. But it also compromises the value of a CFR calculated off the resulting data.

The first problem is selection bias. If you're primarily testing patients that already had severe symptoms, then the population of confirmed cases is skewed towards those that are going to have worse health outcomes from the disease. In turn, this will systematically push up the CFR.

A related problem is that limited testing obviously means the number of confirmed cases will be far lower than the total number of infections. By contrast, the COVID-19 death count, while imperfect, should at least be less understated. The reason is that some jurisdictions, like the US, now include "probable" cases of COVID-19 in the death counts, even without a confirmed test. Thus, although limited testing will effectively cap the number of confirmed cases reported, it does not cap the number of deaths reported. This reality will also tend to systematically inflate the CFR.

The final problem with the CFR occurs simply because COVID-19 is a new disease, and there's a significant time lag between when someone contracts the disease and when they might actually pass away as a result. At any given time, some percentage of the total confirmed and active cases, relate to individuals who will eventually die from the disease. This will cause the CFR to be artificially lower than reality (though the effect is diluted as the disease progresses over time).

As we see, the errors in the CFR are considerable. And while they point in different directions, there's every reason to believe that on net, the CFR significantly overestimates the true lethality rate of COVID-19.

The problem is not that the CFR is literally false. The CFR for COVID-19 is being calculated correctly, it's just not a meaningful number.

"Follow the Data"

These days, it seems like we are constantly being told by pundits and politicians that we need to "follow the data" when it comes to COVID-19.

By itself, that's not bad advice. But too often, these people act as though the data provides a script. We just look at the data, put it in our model, and voila! enlightenment rains down upon us.

It would be nice if it worked that way. In reality, "The Data" doesn't tell us anything. People interpreting the data tell us about their conclusions, and they're not always right.

Media claims no state has met reopening guidelines for new cases, but they're doing the math wrong

As the debate over lifting the lockdowns in the US intensifies, key data points on testing and infections are routinely mischaracterized.

Consider this summary from Chris Wallace on Fox News Sunday on May 3:

As we said, about half the states -- more than half the states have started in some way, shape, or form, reopening. But we've crunched the numbers, Doctor, and not a single state has met the White House gating guidelines of two weeks of steady decline in new cases.

It's not just Fox News that is describing the data in this way. Here's how NBC News reported it on April 28:

As a handful of states begin to ease stay-at-home restrictions, no state that has opted to reopen has come close to the federally recommended decline in cases over a 14-day period...

Some states, such as Colorado and Kentucky, have reported fewer new cases in the past week. But no single state has had a two-week decline in case numbers.

The guidelines being referenced here are the White House's "Opening Up America Guidelines".  The document offers criteria in three different areas that states should meet before reopening, but the reports above are focused only on the standards related to new cases. These are the official criteria  on new infections:

Downward trajectory of documented cases within a 14-day period  

OR

Downward trajectory of positive tests as a percent of total tests within a 14-day period (flat or increasing volume of tests)

So what's going on here? Is it really true that no state in the US is on the down-slope of the COVID-19 crisis?

No, it's not.

In fact, a quick analysis of publicly available data shows that a full 40 states met one of the suggested criterion on new cases as of May 2. (Specifically, 22 meet the documented cases criterion and 37 states fit the alternative positive test percentage criterion.)

The disconnect arises because the White House guidelines don't directly spell out the calculation they have in mind when they ask for a "downward trajectory" within a 14-day period. Instead, different people apply their own calculation and reach different conclusions about the exact same data and exact same guidelines.

Worse still, the media outlets cited above are choosing to interpret the guidelines in a way that will be very difficult for any state to meet prior to outright eradication of COVID-19.

As Chris Wallace says in the quote, no state has had a steady decline in new cases over a two week period. If we assume a "steady decline" would require that a state report fewer (or flat) new cases every day compared to the prior day, then this is technically true. But it's also a really bad standard to use when dealing with data that is expected to have natural variation from day-to-day.

To see how this fails in practice, consider the state of Alaska. As of May 2, this is what Alaska's last 14 days of test results looked like, using data from the COVID Tracking Project:

In total, Alaska had 51 new cases over the two-week period that ended on May 2 out of nearly 12,000 people tested, for a positivity rate of under 0.5%. Even accounting for Alaska's smaller population, this data strongly suggests that Alaska has the virus reasonably under control. (For comparison, New Jersey's stats over this same period was around 42,000 new positive cases and a 42% positivity rate.)

That said, under the "steady decline" standard used by Fox (and NBC*), it would still fail. May 1 saw cases jump from 0 to 9, and broke a 2-day streak of declines. As it happens, the previous day with 0 positive cases also had 0 total tests, highlighting the extreme volatility in daily testing data at a state level. These types of wild fluctuations in daily test counts mean that any daily calculation is bound to be unreliable.

Alaska's data also shows us why this standard requires almost complete eradication before it can be met. During this period, Alaska had one day (April 25) with 0 positive test results in spite of many new tests being run. Thus, if they were to report even 1 new case in the next 13 days, it would still fail to meet the extreme "steady decline" standard. Clearly, this is not a reasonable requirement and is also not the intent the reopening guidelines.

The more appropriate way to do this calculation is to compare the total results from two sequential two-week periods against each other. This prevents volatility in any single day distorting the result, and provides a much better picture of the trend. Here's what the results look like for Alaska under those conditions:

In this view, we can see that, while Alaska never experienced much of a spike, its cases do appear to be on the decline. Despite tripling the number of new tests, they still reported fewer cases. The positivity rate shows a precipitous drop accordingly.

Stepping back from the details of Alaska, we can see this same trend play out across numerous other states. The table below shows the key data points for all states for the last two sequential two-week periods using the approach I discussed above. It also identifies which states would currently meet the case component of the White House reopening guidelines based on those results:

Of course, none of this means that the White House reopening guidelines are a good standard, nor that the lockdown policies are even desirable in the first place.

But, if journalists and public officials treat these guidelines as a reasonable standard for phasing out the lockdowns, it's important to get the math right.

*Note: The NBC article cited above includes a correction indicating that they misstated the guidelines as requiring daily declines. But even though they changed some of the wording in their article, they apparently didn't rerun the analysis and still concluded no states met the guidelines. That is not true now, and it was not true at the time of their article.

Chairman Kim to Leverage US-Granted Legitimacy in Upcoming One-Candidate Elections

 In the wake of the much-discussed Singapore Summit between North Korea and the US, Chairman Kim says he expects to use his newfound legitimacy to win the next round of one-candidate elections.

By Cancelling the Annual Practice Invasion of North Korea, Trump Betrayed Core American Values

 President Trump's summit with North Korea's leader was unprecedented, and for good reason. No recent president has been foolish enough to engage in such reckless diplomacy.

In case the photo ops and the compliments were not enough to make an American's stomach turn, Trump decided to add insult to patriotic injury at the press conference that followed. That's when he cancelled the annual practice invasion of North Korea.

This is a needless concession that sacrifices critical bargaining leverage. Worse than that, it's a total betrayal of American values--specifically, the quintessential American value of belligerence.*

Immigrant Rights Group Says It Will Support Deportation of Jeff Sessions, in Rare Exception to Traditional Stance

On Thursday, the prominent immigrant rights group NumerosUSA said it would make an exception to its traditional stance on deportation and support the "expedient and necessary deportation" of Attorney General Jeff Sessions.

EU Bureaucrats to Vote on Official Definition of Democracy

 
A series of dangerous "democratic" episodes have roiled members of the European Union, from Greece to Spain to Italy, threatening the stability of the overall project. Now, a group of intrepid EU bureaucrats are trying to get ahead of the next crisis by clarifying the definition of democracy.*

Gina Haspel’s Confirmation as CIA Director Rights Historic Injustice

In a major victory for feminism and equality, longtime CIA official Gina Haspel was confirmed to be the new CIA Director last week by the Senate. With the decision, Haspel becomes the first woman to lead the storied spy agency, showing millions of ambitious female sociopaths that they can indeed reach the pinnacle of their profession.

“This is a huge step toward gender equality in the nation’s military-industrial complex,” one feminist activist told The Face Palm.*

However, like many other Trump nominees, Haspel’s confirmation faced considerable pushback.

Most of the criticism focused on Haspel’s role in the CIA’s torture program. Haspel ran a black site in Thailand where the torture methods, including waterboarding, were used.

This context meant that Haspel’s confirmation was not only a question of whether the glass ceiling would be broken at the CIA; it was bigger than that. Her confirmation process was a test of whether powerful women who made horrible and indefensible choices in the past would still be able to advance to higher positions of power in the future.

For men, this question has never been in doubt.

Staten Island District Attorney Daniel Donovan botched the prosecution of the police officer that killed Eric Garner on video, and then promptly launched a successful campaign for Congress. President George W. Bush won reelection after launching the disastrous Iraq War. And of course, Donald Trump managed to become President of the United States after spending a lifetime being–well, Donald Trump.

But for women, it has always been different.

The most famous example is that of former Secretary of State Hillary Clinton. Due to her long tenure in the public eye, Clinton has made a number of demonstrably bad choices, including supporting her husband’s “tough on crime” policies in the 1990s, supporting the Iraq War in 2003, and lobbying for regime change in Libya in 2011, which sent that country into abject chaos.

These assorted disasters contributed to Clinton’s failed campaigns for the White House in 2008 and in 2016. Voters didn’t reject her just because she’d made terrible decisions; it was because she made terrible decisions and she was a woman.

Perhaps with Haspel’s confirmation, we can finally put this sordid chapter in our nation’s history behind us. Perhaps now, at long last, political officials of all genders will be able to continue their amoral ascent to power, uninhibited by the legendary failures of judgment left in their wake.

*This is a satirical post. The activist quoted in this piece is not real person.