What is survivorship bias?

How a statistician’s observations of damage to WWII bombers established the concept of survivorship bias, helping us understand our blindness to random processes.

Have you ever wondered why stories of big Lottery Jackpots are always accompanied by images of ecstatic winners gripping giant oversized cheques? The salient image encourages us to focus on the outcome – the smiling winner – but ignore the process – millions of losing tickets.  This is known as survivorship bias, which works because humans are suckers for stories that stir our imagination and are prone to screen out the less salient details simultaneously. Understanding survivorship can help you make better decisions, and there’s no better place to start learning than the Mountain of Hell Downhill Bike Race.

Every June, hundreds of mountain-biking lunatics head for the Alps for one of the world’s most challenging and dangerous races.

The course starts at an altitude of 3,000m and is a straight race to the bottom, covering every terrain from snow to dirt.

Without knowing anything about the race, you could be forgiven for looking at the podium photo and attributing the success of the first three over the finish line to their superior skill and physical ability. That’s how our brains create narratives. 

Now, look at footage of the start of the race as the contestants hurtle down the snowy mountainside with little to no control over where they’re going. This is survivorship bias in action.

Many of the riders wipe out, but their sheer number means that some will have the right combination of skill and luck to avoid a terminal collision and make it safely through the toughest part of the race.

By looking at just the winners, you’re screening out the silent history of the wipeouts and ignoring the role that chance played in the outcome.

Hatchlings on a beach

Nature provides countless examples of literal survivorship bias. One of the most enduring images is the tiny hatchling turtle scrambling down a beach, navigating a life-threatening obstacle course to reach the water.

• Hatchlings face numerous predators, including ghost crabs, birds, raccoons, dogs, and fish.
• Hatchlings depend on the natural light horizon and the white foam of the surf to know which direction the sea is in. Manmade lights – such as streetlights and cars – can send the hatchlings in the wrong direction.
• Hatchlings rely on the temperature of the sand as a guide to when to emerge, so climate change poses a huge risk finally.

Source: See Turtles

Given these dangers, for every turtle that survives to adulthood, 999 won’t make it.

We can think of that surviving turtle as the equivalent of the lottery winner, though in reality, the odds of winning the UK National lottery are 1 in 43 million, so about 43,000 times less likely.

Derren Brown’s survivorship system

Derren Brown is one of the UK’s most celebrated illusionists, famed for elaborate stunts and stage shows using a broad range of techniques, including misdirection, hypnosis and cognitive psychology. Brown’s act is hugely entertaining but is also designed to expose the tricks used by mediums, faith healers and grifters. 

In 2008, his Channel 4 TV special ‘The System’ used survivorship bias to brilliantly expose how betting tipsters can convince people that they have a 100% winning system. 

The show focused on one member of the public, Khadisha, to whom Brown had sent five consecutive winning horse-racing predictions.

Though that feat is possible from luck alone, it is highly improbable, but his success created a convincing enough narrative for Khadisha to wager £4,000 of their own money on the sixth and final race.

Khadesha didn’t know that ‘The System’ had started by sending tips to 7,776 people split into six equal groups of 1,296. Brown then chose a six-horse race and gave each of the six groups one of the selections as a tip. 

After race one, 6,480 people would have been tipped losers, eliminated and forgotten; 1,296 would have been tipped a winner but would have been oblivious to the losing cohort.

The winning group would then be split again into six groups of 216, and the process repeated. By the sixth race, Khadesha was the sole survivor of the process and became the show’s subject, oblivious to the process and the 7,775 losers, the silent history. She was convinced that The System was foolproof.

• 7,776 start
• 1,296 after race 1
• 216 after race 2
• 36 after race 3
• Six after race 4
• One remains after race 5

In the event, Brown gave Kadisha a losing tipping but, to add an extra twist, changed his mind at the last minute and actually backed the winning horse. Well, that was his explanation; he likely backed every horse just in case.

Abraham Wald – the father of survivorship bias

The concept of the survivorship bias was established by a Jewish-Hungarian mathematician called Abraham Wald while working for the Statistical Research Group. 

The SRG was established in the wake of the Pearl Harbour Attack on the United States in 1941 to use statistical analysis to improve the survivability of fighter aircraft based on the physical evidence obtained from returning planes.

Those surviving craft showed a distribution pattern of anti-aircraft fire, e.g. wings, tail, fuselage and fuel lines. The operational command wanted to know what tactics could be used to minimise damage and which parts of the airframe could be reinforced to increase the chances of surviving damage.

Wald’s analysis was groundbreaking in its counter-intuitive conclusion. Rather than reinforcing the areas that statistically showed the most intense damage, Wald’s recommendation was to do the opposite. Protect the areas of the plane that showed the least damage on the surviving planes. 

The crucial insight was that the damage distribution on the surviving planes was the reason they made it back; those that didn’t return, the silent history, sustained terminal damage elsewhere, crucially the engines and cockpit, which became the focus of the structural improvements.

So how do you judge decision-making success?

Not only did Wald’s work with the SRG establish the concept of survivorship bias, along with another statistician, Jacob Wolfowitz, he also created a test for detecting it. The Wald-Wolfowitz test can establish whether a binary data sequence arises from a random process.

Ruling out the role of luck is the only way to judge genuine talent in making effective decisions under uncertainty, and the Good Judgement Project are the best in the business.

• Read our article on the ice cream truck driver who exposed a TV Gameshow’s lack of a random process to win a fortune

The GJP uses a panel of Superforecasters to make predictions about critical issues of public and commercial interest. Everything from the US Election to whether Russia will engage in lethal force with NATO. 

The GJP sells their services, but unlike betting tipsters, it uses a specific mathematical process to rank its forecasters called the Brier Score, originally used to predict the weather.

The Brier Score uses a binary result system, 0 – an event doesn’t occur – or 1 when it does, but forecasters indicate a degree of confidence by making a prediction somewhere in that range.

Success isn’t about being right or wrong but establishing how far away you are from the right answer.

Your prediction is subtracted from the outcome, and the difference is squared. So if you were to predict that tomorrow it will rain with a confidence of 0.6, and it does rain, your Brier Score would be:

(1 – 0.6)squared + (0 – 0.4)squared = 0.16 + 0.16 = 0.32

The worst Brier score is two because you either show absolute certainty in an event that doesn’t occur or predict that an event that does occur has no chance.

The best Score is 0, which might seem counterintuitive but reverses that logic.

If that seems a bit complex, there is a slightly simpler version for judging the success of a betting tipster taking the square root of the total number of selections and adding that number to one-half of the total plays made:

√(No. Selections) + ½ (Total Plays Made)

For example, if a tipster makes 400 tips, the square root would be 20, adding to one-half of 400, giving a total of 220 theoretical wins. The expectation would be that for a random event, the average tipster would make 200 correct predictions with a 1 in 40 chance of getting 220 correct predictions.

Survivorship Bias & Social Media

The principle of survivorship bias can be dumbed right down to the phrase ‘it’s a numbers game’. Throw enough darts at the wall, and one of them will hit the bullseye or put enough monkeys in front of typewriters, and one of them will eventually produce the script for Planet of the Apes.

Of course, we no longer live in a world of typewriters, but smartphones, constantly at our fingertips, plugging us into social media platforms, the modern equivalent of books and newspapers.

With millions promoting themselves as star stock traders, crypto gurus and betting tipsters to eager audiences, a few will inevitably produce strings of winners, just like Derren Brown’s experiment.

So what Twitter, TikTok and Instagram enable is survivorship bias on a massive scale. But rather than being skilled predictors, these self-declared social media savants are just the turtle that made it to the sea.

---