This is a repost from 2009. It talks about breast cancer, but applies equally well to Covid testing, given the high percentage of false positives (not to mention the more worrisome false negatives). We simply are not wired well for probability…
Yudkowsky poses the following canonical problem:
1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies.
A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?
The frightening thing is that only 15% of doctors get this right. And they’re off by a lot. That is, the average answer is in the range of 80% while the correct answer is 7.8%. Apparently, there is something about the way we think about the problem that makes 7.8% hard to accept, and Yudkowsky does a great job of walking you through the logic in painfully small steps.
Let’s try something similar here…
What Do We Know & What Does It Imply?
We have three pieces of information:
1% of sample are TRUE (that is, they have cancer)
80% of sample who are TRUE will test TRUE
9.6% of sample who are FALSE will test TRUE.
On the face of it, we should guess that the percentage of women who test TRUE who actually are TRUE (test positive and actually have cancer) is pretty small based on two facts provided: the actual percentage of women from the sample who are TRUE (regardless of testing) is only 1%, and the test has a false positive for 9.6% of those tested.
So, to solve this:
1) Assume we have a sample of 1000 women (I use 1000 to reduce the amount I have to talk about fractional people, but I don’t use 10,000 as I get lost in the zeros).
2) We know that the reality is that of the 1,000 women, 10 will have cancer (1%).
990 = no cancer
10 = cancer
3) Of the 10 who have cancer, 8 will test positive
8 out 1000 women tested will test True and are True
4) Of the 990 with no cancer 9.6% will also test positive = 990 * .096 = 95.04.
95.04 women out of 1,000 will test True but are False.
5) The total number testing true is 8 + 95.04 = 103.04.
Of these, 8 actually have Cancer.
6) So the value for tests positive (103.04) versus is positive (8) is 8/103.4 or 0.773 or 7.8%
