Tag: Survivorship Bias

What Sharks Can Teach Us About Survivorship Bias

Survivorship bias refers to the idea that we get a false representation of reality when we base our understanding only on the experiences of those who live to tell their story. Taking a look at how we misrepresent shark attacks highlights how survivorship bias distorts reality in other situations.

When asked what the deadliest shark is to humans, most people will say the great white. The lasting influence of the movie Jaws, reinforced by dozens of pop culture references and news reports, keeps that species of shark at the top of the mind when one considers the world’s most fearsome predators. While it is true that great white sharks do attack humans (rarely), they also leave a lot of survivors. And they’re not after humans in particular. They usually just mistake us for seals, one of their key food sources.

We must be careful to not let a volume of survivors in one area blind us to the stories of a small number of survivors elsewhere. Most importantly, we need to ask ourselves what stories are not being told because no one is around to tell them. The experiences of the dead are necessary if we want an accurate understanding of the world.

***

Before we drill down into some interesting statistics, it’s important to understand that great whites are one member of a class of sharks with many common characteristics. Great whites are closely related to tiger and bull sharks. They all have similar habitats, physiology, and instincts. They are also all large, with an average size over ten feet long.

Tiger and bull sharks rarely attack humans, and to someone being bit by one of these huge creatures, there isn’t all that much difference between them. The Florida Museum’s International Shark Attack file explains that “positive identification of attacking sharks is very difficult since victims rarely make adequate observations of the attacker during the ‘heat’ of the interaction. Tooth remains are seldom found in wounds and diagnostic characters for many requiem sharks [of which the great white is one] are difficult to discern even by trained professionals.”

The fatality rate in known attacks is 21.5% for the bull shark, 16% for the great white, and 26% for the tiger shark. But in sheer volume, attacks attributed to great whites outnumber the other two species three to one. So there are three times as many survivors to tell the story of their great white attack.

***

When it comes to our picture of reality of the most dangerous shark, there are other blind spots. Not all sharks have the same behaviors as those three, such as swimming close to shore and being around enough prey to develop a preference for fat seals versus bony humans. Pelagic sharks live in the water desert that is the open ocean and have to eat pretty much whatever they can find. The oceanic white tip is a pelagic shark that is probably far more dangerous to humans—we just don’t come into contact with them as often.

There are only fifteen documented attacks by an oceanic white tip, with three of those being fatal. But since most attacks occur in the open ocean in more isolated situations (e.g., a couple of people on a boat versus five hundred people swimming at a beach), we really have no idea how dangerous oceanic white tips are. There could be hundreds of undocumented attacks that left behind no survivors to tell the tale.

One famous survivor story gives us a glimpse of how dangerous oceanic white tips might be. In 1945, a Japanese submarine shot down the USS Indianapolis. For a multitude of reasons, partly due to the fact that the Indianapolis was on a top secret mission and partly due to tragic incompetence, a rescue ship was not sent for four days. Those who survived the ship’s sinking had to then try to survive in the open ocean with little gear until rescue arrived. The water was full of sharks.

In Indianapolis: The True Story of the Worst Sea Disaster in US Naval History and the Fifty-Year Fight to Exonerate an Innocent Man, Lynn Vincent and Sara Vladic quote Boatswain’s Mate Second Class Eugene Morgan as he described part of his experience: “All the time, the sharks never let up. We had a cargo net that had Styrofoam things attached to keep it afloat. There were about fifteen sailors on this, and suddenly, ten sharks hit it and there was nothing left. This went on and on.” These sharks are believed to have been oceanic white tips. It’s unknown how many men died from shark attacks. Many also perished due to exposure, dehydration, injury, and exhaustion. Of the 1,195 crewmen originally aboard the ship, only 316 survived. It represents the single biggest loss of life from a single ship in US naval history.

Because humans are rarely in the open ocean in large numbers, not only are attacks by this shark less common, there are also fewer survivor stories. The story of the USS Indianapolis is a rare, brutal case that provides a unique picture.

***

Our estimation of the shark that could do us the most harm is often formed by survivorship bias. We develop an inaccurate picture based on the stories of those who live to tell the tale of their shark attack. We don’t ask ourselves who didn’t survive, and so we miss out on the information we need to build an accurate picture of reality.

The point is not to shift our fear to oceanic white tips, which are, in fact, critically endangered. Our fear of sharks seems to make us indifferent to what happens to them, even though they are an essential part of the ocean ecosystem. We are also much more of a danger to sharks than they are to us. We kill them by the millions every year. Neither should we shift our fear to other, more lethal animals, which will likely result in the same indifference to their role in the ecosystem.

The point is rather to consider how well you make decisions when you only factor in the stories of the survivors. For instance, if you were to try to reduce instances of shark attacks or try to limit their severity, you will not likely get the results you are after if you only pay attention to the survivor stories. You need to ask who didn’t make it and try to figure out their stories as well. If you try to implement measures aimed only at great whites near beaches, your measures might not be effective against other predatory sharks. And if you conclude that swimmers are better off in the open ocean because sharks seem to only attack near beaches, you’d be completely wrong.

***

Survivorship bias crops up all over our lives and impedes us from accurately assessing danger. Replace “dangerous sharks” with “dangerous cities” or “dangerous vacation spots” and you can easily see how your picture of a certain location might be skewed based on the experiences of survivors. We can’t be afraid of a tale if no one lives to tell it. More survivors can make something seem more dangerous rather than less dangerous because the volume of stories makes them more memorable.

If fewer people survived shark attacks we wouldn’t have survivor stories influencing our perception about how dangerous sharks are. In all likelihood we would attribute some of the ocean deaths to other causes, like drowning, because it wouldn’t occur to us that sharks could be responsible.

Understanding survivorship bias prompts us to look for the stories of those who weren’t successful. A lack of visible survivors with memorable stories might mean we view other fields as far safer and easier than they are.

For example, a field of business where people who experience failures go on to do other things might seem riskier than one where people who fail are too ashamed to talk about it. The failure of tech start-ups sometimes feels like daily news. We don’t often, however, hear about the real estate agent who has trouble making sales or who keeps getting outbid on offers. Nor do we hear much about architects who design terrible houses or construction companies who don’t complete projects.

Survivorship bias prompts us to associate more risk with industries that exhibit more public failures. But the failures from industries or businesses that aren’t shared are equally important. If we focus only on the survivor stories, we might think that being a real estate agent or an architect is safer than starting a technology company. It might be, but we can’t only base our understanding on which career option is the best bet on the widely shared stories of failure.

If we don’t factor survivorship bias into our thinking we end up in a classic map is not the territory problem. The survivor stories become a poor navigational tool for the terrain.

Most of us know that we shouldn’t become a writer based on the results achieved by J.K Rowling and John Grisham. But even if we go out and talk to other writers, or learn about their careers, or attend writing seminars given by published authors, we are still only talking to the survivors.

Yes, it’s super inspiring to know Stephen King got so many rejections early in his career that the stack of them was enough to pull a nail out of the wall. But what about the writers who got just as many rejections and never published anything? Not only can we learn a lot from them about the publishing industry, we need to consider their experiences if we want to anticipate and understand the challenges involved in being a writer.

***

Not recognizing survivorship bias can lead to faulty decision making. We don’t see the big picture and end up optimizing for a small slice of reality. We can’t completely overcome survivorship bias. The best we can do is acknowledge it, and when the stakes are high or the result important, stop and look for the stories of those who were unsuccessful. They have just as much, if not more, to teach us.

The next time you’re assessing risk, ask yourself: am I paying too much attention to the great white sharks and not enough to the oceanic white tips?

Survivorship Bias: The Tale of Forgotten Failures

Survivorship bias is a common logical error that distorts our understanding of the world. It happens when we assume that success tells the whole story and when we don’t adequately consider past failures.

There are thousands, even tens of thousands of failures for every big success in the world. But stories of failure are not as sexy as stories of triumph, so they rarely get covered and shared. As we consume one story of success after another, we forget the base rates and overestimate the odds of real success.

“See,” says he, “you who deny a providence, how many have been saved by their prayers to the Gods.”

“Ay,” says Diagoras, “I see those who were saved, but where are those painted who were shipwrecked?”

— Cicero

The Basics

A college dropout becomes a billionaire. Batuli Lamichhane, a chain-smoker, lives to the age of 118. Four young men are rejected by record labels and told “guitar groups are on the way out,” then go on to become the most successful band in history.

Bill Gates, Batuli Lamichhane, and the Beatles are oft-cited examples of people who broke the rules without the expected consequences. We like to focus on people like them—the result of a cognitive shortcut known as survivorship bias.

When we only pay attention to those who survive, we fail to account for base rates and end up misunderstanding how selection processes actually work. The base rate is the probability of a given result we can expect from a sample, expressed as a percentage. If you play roulette, for example, you can be expected to win one out of 38 games, or 2.63%, which is the base rate. The problem arises when we mistake the winners for the rule and not the exception. People like Gates, Lamichhane, and the Beatles are anomalies at one end of a distribution curve. While there is much to learn from them, it would be a mistake to expect the same results from doing the same things.

A stupid decision that works out well becomes a brilliant decision in hindsight.

— Daniel Kahneman

Cause and Effect

Can we achieve anything if we try hard enough? Not necessarily. Survivorship bias leads to an erroneous understanding of cause and effect. People see correlation in mere coincidence. We all love to hear stories of those who beat the odds and became successful, holding them up as proof that the impossible is possible. We ignore failures in pursuit of a coherent narrative about success.

Few would think to write the biography of a business person who goes bankrupt and spends their entire life in debt. Or a musician who tried again and again to get signed and was ignored by record labels. Or of someone who dreams of becoming an actor, moves to LA, and ends up returning a year later, defeated and broke. After all, who wants to hear that? We want the encouragement survivorship bias provides, and the subsequent belief in our own capabilities. The result is an inflated idea of how many people become successful.

The discouraging fact is that success is never guaranteed. Most businesses fail. Most people do not become rich or famous. Most leaps of faith go wrong. It does not mean we should not try, just that we should be realistic with our understanding of reality.

Beware of advice from the successful.

— Barnaby James

Survivorship Bias in Business

Survivorship bias is particularly common in the world of business. Companies which fail early on are ignored, while the rare successes are lauded for decades. Studies of market performance often exclude companies which collapse. This can distort statistics and make success seem more probable than it truly is. Just as history is written by the winners, so is much of our knowledge about business. Those who end up broke and chastened lack a real voice. They may be blamed for their failures by those who ignore the role coincidence plays in the upward trajectories of the successful.

Nassim Taleb writes of our tendency to ignore the failures: “We favor the visible, the embedded, the personal, the narrated, and the tangible; we scorn the abstract.” Business books laud the rule-breakers who ignore conventional advice and still create profitable enterprises. For most entrepreneurs, taking excessive risks and eschewing all norms is an ill-advised gamble. Many of the misfit billionaires who are widely celebrated succeeded in spite of their unusual choices, not because of them. We also ignore the role of timing, luck, connections and socio-economic background. A person from a prosperous family, with valuable connections, who founds a business at a lucrative time has a greater chance of survival, even if they drop out of college or do something unconventional. Someone with a different background, acting at an inopportune time, will have less of a chance.

In No Startup Hipsters: Build Scalable Technology Companies, Samir Rath and Teodora Georgieva write:

Almost every single generic presentation for startups starts with “Ninety Five percent of all startups fail”, but very rarely do we pause for a moment and think “what does this really mean?” We nod our heads in somber acknowledgement and with great enthusiasm turn to the heroes who “made it” — Zuckerberg, Gates, etc. to absorb pearls of wisdom and find the Holy Grail of building successful companies. Learning from the successful is a much deeper problem and can reduce the probability of success more than we might imagine.

Examining the lives of successful entrepreneurs teaches us very little. We would do far better to analyze the causes of failure, then act accordingly. Even better would be learning from both failures and successes.

Focusing on successful outliers does not account for base rates. As Rath and Georgieva go on to write:

After any process that picks winners, the non-survivors are often destroyed or hidden or removed from public view. The huge failure rate for start-ups is a classic example; if failures become invisible, not only do we fail to recognise that missing instances hold important information, but we may also fail to acknowledge that there is any missing information at all.

They describe how this leads us to base our choices on inaccurate assumptions:

Often, as we revel in stories of start-up founders who struggled their way through on cups of ramen before the tide finally turned on viral product launches, high team performance or strategic partnerships, we forget how many other founders did the same thing, in the same industry and perished…The problem we mention is compounded by biographical or autobiographical narratives. The human brain is obsessed with building a cause and effect narrative. The problem arises when this cognitive machinery misfires and finds patterns where there are none.

These success narratives are created both by those within successful companies and those outside. Looking back on their ramen days, founders may believe they had a plan all along. They always knew everything would work out. In truth, they may lack an idea of the cause and effect relationships underlying their progress. When external observers hear their stories, they may, in a quasi-superstitious manner, spot “signs” of the success to come. As Daniel Kahneman has written, the only true similarity is luck.

Consider What You Don’t See

When we read about survivorship bias, we usually come across the archetypical story of Abraham Wald, a statistician studying World War II airplanes. His research group at Columbia University was asked to figure out how to better protect airplanes from damage. The initial approach to the problem was to look at the planes coming back, seeing where they were hit the worst, then reinforcing that area.

However, Wald realized there was a missing, yet valuable, source of evidence: Planes that were hit that did not make it back. Planes that went down, that weren’t surviving, had much better information to provide on areas that were most important to reinforce. Wald’s approach is an example of how to overcome survivorship bias. Don’t look just at what you can see. Consider all the things that started on the same path but didn’t make it. Try to figure out their story, as there is as much, if not more, to be learned from failure.

Considering survivorship bias when presented with examples of success is difficult. It is not instinctive to pause, reflect, and think through what the base rate odds of success are and whether you’re looking at an outlier or the expected outcome. And yet if you don’t know the real odds, if you don’t know if what you’re looking at is an example of survivorship bias, then you’ve got a blind spot.

Whenever you read about a success story in the media, think of all the people who tried to do what that person did and failed. Of course, understanding survivorship bias isn’t an excuse for not taking action, but rather an essential tool to help you cut through the noise and understand the world. If you’re going to do something, do it fully informed.

To learn more, consider reading Fooled By Randomness, or The Art of Thinking Clearly.

The Art of Thinking Clearly

Rolf Dobelli’s book, The Art of Thinking Clearly, is a compendium of systematic errors in decision making. While the list of fallacies is not complete, it’s a great launching pad into the best of what others have already figured out.

To avoid frivolous gambles with the wealth I had accumulated over the course of my literary career, I began to put together a list of … systematic cognitive errors, complete with notes and personal anecdotes — with no intention of ever publishing them. The list was originally designed to be used by me alone. Some of these thinking errors have been known for centuries; others have been discovered in the last few years. Some came with two or three names attached to them. … Soon I realized that such a compilation of pitfalls was not only useful for making investing decisions but also for business and personal matters. Once I had prepared the list, I felt calmer and more levelheaded. I began to recognize my own errors sooner and was able to change course before any lasting damage was done. And, for the first time in my life, I was able to recognize when others might be in the thrall of these very same systematic errors. Armed with my list, I could now resist their pull — and perhaps even gain an upper hand in my dealings.

Dobelli’s goal is to learn to recognize and evade the biggest errors in thinking. In so doing, he believes we might “experience a leap in prosperity. We need no extra cunning, no new ideas, no unnecessary gadgets, no frantic hyperactivity—all we need is less irrationality.”

Let’s take a look at some of the content.

Guarding Against Survivorship Bias

People systematically overestimate their chances of success. Guard against it by frequently visiting the graves of once-promising projects, investments, and careers. It is a sad walk but one that should clear your mind.

Pattern Recognition

When it comes to pattern recognition, we are oversensitive. Regain your scepticism. If you think you have discovered a pattern, first consider it pure chance. If it seems too good to be true, find a mathematician and have the data tested statistically.

Fighting Against Confirmation Bias

[T]ry writing down your beliefs — whether in terms of worldview, investments, marriage, health care, diet, career strategies — and set out to find disconfirming evidence. Axing beliefs that feel like old friends is hard work but imperative.

Dating Advice and Contrast

If you are seeking a partner, never go out in the company of your supermodel friends. People will find you less attractive than you really are. Go alone or, better yet, take two ugly friends.

Think Different

Fend it off (availability bias) by spending time with people who think different than you do—people whose experiences and expertise are different from yours.

Guard Against Chauffeur Knowledge

Be on the lookout for chauffeur knowledge. Do not confuse the company spokesperson, the ringmaster, the newscaster, the schmoozer, the verbiage vendor, or the cliche generator with those who possess true knowledge. How do you recognize the difference? There is a clear indicator: True experts recognize the limits of what they know and what they do not know.

The Swimmer’s Body Illusion

Professional swimmers don’t have perfect bodies because they train extensively. Rather, they are good swimmers because of their physiques. How their bodies are designed is a factor for selection and not the result of their activities. … Whenever we confuse selection factors with results, we fall prey to what Taleb calls the swimmer’s body illusion. Without this illusion, half of advertising campaigns would not work. But this bias has to do with more than just the pursuit of chiseled cheekbones and chests. For example, Harvard has the reputation of being a top university. Many highly successful people have studied there. Does this mean that Harvard is a good school? We don’t know. Perhaps the school is terrible, and it simply recruits the brightest students around.

Peer Pressure

A simple experiment, carried out in the 1950s by legendary psychologist Solomon Asch, shows how peer pressure can warp common sense. A subject is shown a line drawn on paper, and next to it three lines—numbered 1, 2, and 3—one shorter, one longer, and one the same length as the original one. He or she must indicate which of the three lines corresponds to the original one. If the person is alone in the room, he gives correct answers because the task is really quite simple. Now five other people enter the room; they are all actors, which the subject does not know. One after another, they give wrong answers, saying “number 1,” although it’s very clear that number 3 is the correct answer. Then it is the subject’s turn again. In one-third of cases, he will answer incorrectly to match the other people’s responses

Rational Decision Making and The Sunk Cost Fallacy

The sunk cost fallacy is most dangerous when we have invested a lot of time, money, energy, or love in something. This investment becomes a reason to carry on, even if we are dealing with a lost cause. The more we invest, the greater the sunk costs are, and the greater the urge to continue becomes. … Rational decision making requires you to forget about the costs incurred to date. No matter how much you have already invested, only your assessment of the future costs and benefits counts.

Avoid Negative Black Swans

But even if you feel compelled to continue as such, avoid surroundings where negative Black Swans thrive. This means: Stay out of debt, invest your savings as conservatively as possible, and get used to a modest standard of living—no matter whether your big breakthrough comes or not

Disconfirming Evidence

Munger Destroy ideas

The confirmation bias is alive and well in the business world. One example: An executive team decides on a new strategy. The team enthusiastically celebrates any sign that the strategy is a success. Everywhere the executives look, they see plenty of confirming evidence, while indications to the contrary remain unseen or are quickly dismissed as “exceptions” or “special cases.” They have become blind to disconfirming evidence.

Still curious?

Read The Art of Thinking Clearly.