Why does it look like the state is testing at only a quarter of its lab capacity?
Why does it look like Wisconsin’s positive test rate has exploded?
Is DHS making a simple math error?
Is DHS throwing away hundreds of thousands of negative test results to make its math work?
Is the Evers Administration making key decisions using bad math?
September 23, 2020
A fundamental flaw in how the Evers’ Administration calculates Wisconsin’s daily COVID-19 positive test rate has excluded hundreds of thousands of test results and led to a wildly distorted picture of the state’s progress in confronting the virus.
The positive test rate is important, because it is one of six gating criteria the Evers’ Administration uses to make public health policy decisions that affect the entire state. Specifically, the requirement is for the “Downward trajectory of positive tests as a percent of total tests within a 14-day period.”
Throughout August and September, Wisconsin’s positive test rate steadily increased from a weekly average of 6.2% to 19% (as of Sep. 21st).
The number of new people with a positive test result each day is less important, but gets more attention. That’s not surprising because the Department of Health Services (DHS) posts the number of new positive cases each day prominently at the top of its summary dashboard, while one must hunt for the actual test rate on the page.
As the test rate has steadily ticked up over the past two months, Wisconsin has had five record setting days for the number of new positive cases. Most of the increase has occurred among the 18-24 age group and is tied to college campuses. That’s where public officials are now focusing their efforts, but not every reporter has forgotten the importance of the rate.
“What’s behind the rise in percent positive rates?” Wispolitics’ Stephanie Hoff bluntly asked Dr. Ryan Westergaard, Wisconsin’s Chief Medical Officer on Sep. 15th.
This systematic error means DHS is tossing hundreds of thousands of negative test results when calculating the positive test rate. The real rate could be half of what DHS claims.
“Well, it’s a ratio so it has two components: the numerator – the top number of course is the number of positive cases,” he replied. “The bottom number of the ratio is the denominator. That’s the number of people that have been tested overall.”
This was an incredible admission. If the goal is to calculate the daily positive test rate, then DHS is using the wrong numerator and denominator. What they’re actually calculating is the daily percentage of new COVID-19 cases among those who have never been tested before – a fairly meaningless statistic. It is not the positive test rate.
On that same day, Sep. 15th, DHS recorded the rate as 11%. That was calculated by comparing 1,352 new cases to 12,266 new people tested. However, there were almost 20,000 tests collected that day. If DHS compared that day’s number of positive tests to the total number of test results, the rate would have been much lower – possibly as low as 6.7%. They’ve been making this same mistake since the spring.
This systematic error means DHS is tossing hundreds of thousands of negative test results when calculating the positive test rate. The real rate could be half of what DHS claims.
Origin of the Error
When it first started collecting COVID-19 data, DHS recognized that people were getting tested multiple times. Some people tested positive for the virus one week, and then still had it when they were retested a week later. If the department wasn’t careful, it might be accidentally counting single cases of COVID-19 over and over, and include them in the official tally. That would inflate Wisconsin’s total COVID-19 cases – a mistake that surely would be caught.
The department decided on a data collection method that would pick and choose which test results to use. Dr. Westergaard explained the reasoning during a June 11th press conference.
“Our data reports individuals tested. So, if an individual is tested more than once because they were being followed to see if they cleared the infection or if they were tested a couple times weeks apart, they would be considered a single case and not multiple cases in our data,” Westergaard explained.
DHS acknowledges this method on its data summary dashboard. The fine print plainly states “Multiple tests per person are not included in these summary statistics.”
And so, every day DHS compares the number of new COVID-19 cases to the number of people who were tested for the first time to determine the positive test rate. Under this method, it throws away the results for everyone that has ever been tested before. That means every day thousands of negative test results are not included in the rate calculation.
DHS has discarded hundreds of thousands of results since it began reporting COVID-19 data. Most people who are tested for COVID-19 receive negative test results.
As the pool of people who have never been tested before continues to shrink, the number of negative results getting tossed will naturally increase. Subsequently the positive test rate will continue to artificially rise.
Scope of the Error
Each day’s positive test rate needs to be calculated using all the test results from that day. Only DHS can correct this error, because only it has access to all the test results.
Although DHS’ website is filled with all kinds of COVID-19 related data – it does not post the total number of test results it receives each day. However, it does report the number of total tests collected by health care professionals across the state each day. That figure cannot be used to calculate the daily rate, because there is a lag from the time a test is collected and when the results come back.
Each day’s positive test rate needs to be calculated using all the test results from that day.
However, we can still get a good look at the problem using weekly averages.
Altogether, there have been approximately 2.1 million tests, 1.4 million people tested, and 101,321 positive cases (as of Sep. 20th). Using DHS’ flawed methodology, officials have tossed around 600,000 test results in calculating the daily positive test rate. That results in an overall rate of 7%, while the real rate could be as low as 4.8%.
That might not seem like a big difference overall. However, public officials aren’t looking at the big picture. They’re looking at this rate on a day-to-day basis, and only 14 days back. The inherent flaws in DHS’ formula means the difference between the official rate and the real rate consistently grows.
When DHS first began collecting COVID-19 data, no one had been tested before, and few results were discarded. At that time, the rate calculated by DHS would have been fairly accurate.
However, as time goes on, more and more people are re-tested. That means more and more results are thrown out. Inevitably the rate will become more and more inaccurate unless DHS corrects its calculations.
Yesterday, 12,537 people in Wisconsin, who had never been tested before, got their #COVID19 test results. Those were the only results @DHSWI used to calculate the daily positive test rate. It’s been using this method from the beginning. https://t.co/8qx2wIe0lh #WIright pic.twitter.com/tYNOpDDVuW
— MacIver Institute (@MacIverWisc) September 23, 2020
Testing Capacity
The cracks in DHS’ methodology are beginning to show.
The Evers Administration is increasingly confronted with questions it cannot answer (even when the answers are right on DHS’ website).
For example, some reporters have noticed that the state’s lab capacity is much higher than – what they believe to be – the number of new test results each day.
In August, lab capacity was around 26,000 tests a day, and an average of 15,600 test were administered each day. That meant labs were running at about 60% capacity. In September, lab capacity increased to 38,000 tests a day, and an average of 20,685 tests were administered each day. That meant labs were running at about 54% capacity. However, the way DHS presents its data could lead some to believe there are only around 10,000 tests being administered each day – or about 26 – 38%.
“Why do we continue to be at a quarter the rate of testing capacity? Is it a shortage of supplies or people just not getting tested?” a reporter asked Palm at a press conference on Sep. 16th.
“This is certainly a complicated question and therefor a complicated answer and some of it is just speculation,” Palm replied.
Clearly, the reporter was confused or did not realize that the number of new people being tested is different from the number of tests being collected. Palm had the opportunity to correct this confusion, but she either didn’t listen to the question or doesn’t understand the issue herself.
Even Dr. Westergaard, who has repeatedly demonstrated his understanding of what data is available and how the rate is calculated, has yet to point out the complete truth of these fundamental issues.
That reporter is not the first one to ask why test numbers always appear to be well below the state’s capacity. Once reporters begin to understand the difference between individuals tested and the number of tests actually collected, they will be able to identify the bad math DHS uses to calculate Wisconsin’s rate. They will then be able to ask why the state is making public policy decisions based on this bad math.
Whether or not the Evers Administration or the Capitol press corps figure this out on their own eventually won’t matter. As the number of new individuals getting tested continues to drop, the day will come when there are more positive test results than new individuals being tested. Once the daily positive test rate rises above 100%, no one will be able to ignore the math any longer.
Solving the Problem
DHS was right to take steps to ensure its total case count was accurate, but the positive test rate is just as important.
DHS should also calculate the positive test rate every day using all the results from that day. Since no one is tested twice in the same day, this will ensure that each day’s rate is accurate. That will ensure the gating requirement is reflecting the correct data.
This is something DHS could start doing today. It needs to start doing this today, because it’s current method is producing inaccurate results – which the Evers Administration continues to use in its gating criteria to make pivotal public policy decisions that affect everyone in Wisconsin.
MacIver reached out to Dr. Westergaard on Tuesday morning, but has yet to receive a response.