Survivorship Bias

The year was 1942 and World War 2 was at its peak. Researchers at the Center for Naval Analysis were faced with a challenging problem. Bombers were routinely getting shot down during their runs over Germany and the ones that came back were riddled with bullet holes. The researchers were tasked with improving the odds of a plane surviving a bombing run.

The simplest solution would be to add more armor to the plane. But armor would make the planes heavier and less maneuverable making them even more vulnerable. Using too much or too little would both be an issue. Since it’s an optimization problem, they turned to data and analyzed the bullet holes and damage that the bombers suffered after each run and a clear picture started to emerge.

Now the solution was simple - Add armor to the areas that had the maximum chance of getting hit so that you minimize damage to the aircraft. Can you see the flaw in the plan?

Before the modifications were made, famous statistician Abraham Wald reviewed the data and found a critical flaw in the analysis - They were only looking at planes that came back. The areas with the minimum bullet holes were actually the most important since any hit there downed the plane. Wald’s recommendation was quickly put into effect and was so successful that it’s still being used today.