By Tajammal MaqboolFounder & Developer
Identify the most dangerous statistical fallacies that lead to wrongful convictions, failed policies, wasted research funding, and medical harm. These advanced scenarios test whether you can spot subtle errors involving Simpson's paradox, the prosecutor's fallacy, multiple comparisons, selection bias, and expected value traps that regularly fool judges, journalists, scientists, and executives.
Statistical fallacies are the ways valid-looking analyses reach wrong conclusions. They turn up in published research, courtrooms, and business dashboards, often stated confidently by people who can recite the formulas but missed the structural problem underneath. This exercise drills the most consequential advanced patterns behind much of the replication crisis: Simpson's paradox, regression to the mean, survivorship bias, multiple comparisons, and p-hacking.
Statistical errors have produced wrongful convictions, failed policies, and medical harm. This exercise trains you to identify the most consequential ones, namely base-rate neglect, the prosecutor's fallacy, Simpson's paradox, and multiple comparisons, in the contexts where they actually cause damage rather than as abstract definitions.
Simpson's paradox is the strangest: a pattern that holds in every subgroup can flip when you combine them. In a famous 1973 admissions case, a university looked biased against women overall, yet nearly every individual department admitted women at a higher rate. Women had simply applied more to the most competitive departments. Whether to trust the combined or the split view depends on which one answers your actual question.
Survivorship bias is the other big one. In World War II, the military wanted to armor the bullet-riddled areas of returning bombers. A statistician pointed out those were the survivable hits. The armor belonged where returning planes had no holes, because those planes never came back. Studies of successful companies and traders make the same mistake by ignoring the failures. See Scientific Thinking and Cognitive Biases: Memory & Self.
Question 1
A kidney stone treatment study finds: Treatment A is more effective than Treatment B for large stones (93% vs. 87%), AND Treatment A is more effective for small stones (87% vs. 83%). But when all patients are combined, Treatment B appears more effective overall (83% vs. 78%). The hospital board, looking only at the combined data, chooses Treatment B for all patients. What went wrong?
Question 2
In a criminal trial, a forensic expert testifies: 'The probability of a random person matching this DNA profile is 1 in 10 million. Therefore, the probability that the defendant is innocent is 1 in 10 million.' The defense attorney objects. What error has the expert committed?
Question 3
A gambler explains his Martingale strategy: 'I bet $10. If I lose, I double to $20, then $40, and so on. When I eventually win, I recover all losses plus $10 profit. I've made $2,000 over three months, so it's mathematically guaranteed.' He is about to bet $5,120 after nine consecutive losses. What is fundamentally wrong?
Question 4
After a mass shooting, a data analyst maps all mass shootings over a decade and discovers a geographic 'cluster' in the Midwest. A news outlet reports: 'Data reveals Midwest mass shooting hotspot. What is causing this regional pattern?' A statistician says the cluster is likely meaningless. How can a visible pattern in real data be meaningless?
Question 5
Researchers test 20 common food additives for links to childhood hyperactivity. They find Additive #14 shows a 'statistically significant' result (p = 0.03). They publish: 'Additive #14 linked to hyperactivity (p < 0.05).' The paper does not mention the other 19 additives tested. Why should this be treated with extreme skepticism?
Question 6
A hedge fund advertises: 'Our fund has beaten the market for 8 consecutive years.' Investigation reveals the parent company launched 256 funds simultaneously with different strategies 8 years ago. How many funds would you expect to beat the market all 8 years purely by chance, assuming a 50% probability each year?
Where to go after this exercise.
