Survivorship Bias Starts With the Mornings You Don't Remember

W. Allen Wallis directed research at the Statistical Research Group at Columbia University during the Second World War, and he later described it as “what surely must be the most extraordinary group of statisticians ever organized.” Milton Friedman, Harold Hotelling and Leonard Savage belonged to it. So did Abraham Wald, the Hungarian-born mathematician who became the patron saint of survivorship bias.

Survivorship bias is the error of studying only the cases that made it through some filter and treating them as the whole population. Wald’s contribution, in memoranda written in 1943, was a method for estimating how vulnerable each part of an aircraft was using only the damage on planes that came back, with the planes that didn’t come back accounted for by arithmetic since nobody could inspect them. The same error runs through success advice, and it runs through your memory of your own mornings.

What is the bullet-hole story, and how much of it holds up?

The popular version goes like this. Analysts count the bullet holes on bombers returning from missions and find them clustered on the wings and fuselage. The brass wants armor there. Wald points out that the holes mark places a plane can be hit and still fly home, and that the missing planes are the ones hit somewhere else. Armor the clean spots.

Bill Casselman, writing for the American Mathematical Society in 2016, put the status of that story carefully: there is “a solid germ of truth in it,” but “very little evidence for the best bits.” What survives scrutiny is thinner and more interesting. Wald wrote eight memoranda in 1943; Casselman counts five of them as devoted to estimating a plane’s probability of surviving once it has been hit. The Center for Naval Analyses reissued them in 1980, and Mangel and Samaniego’s 1984 paper in the Journal of the American Statistical Association says the method was used in the Second World War and again in Korea and Vietnam.

The retelling also flattens an assumption. Wallis wrote that Wald “assumed, on good evidence, that hits in combat were uniformly distributed over the planes.” If enemy fire had been concentrated on the engines, the same pattern of holes would mean something different. The conclusion rests on a premise about where the bullets fell, and the premise was supported by evidence rather than assumed for convenience.

Details that circulate with the story, such as which branch asked the question, which parts were armored, and the exact diagram, do not trace back to anything in Wald’s own papers, and Casselman notes that those writings are the only reliable account of his work. Wallis’s memoir mentions the vulnerability work only briefly. Wald himself was officially an enemy alien for part of this period and, in Wikipedia’s summary of Hotelling’s account, was barred from restricted material until a court naturalized him. On December 13, 1950, at 48, he and his wife died when an Air India DC-3 crashed in the Nilgiri Mountains during a lecture tour of India.

Why is success advice a hangar full of returning planes?

Two thousand years before Wald, Cicero told the story of Diagoras, shown a wall of votive paintings left by sailors who had prayed in a storm and lived. Diagoras asked where the paintings were of those who had prayed and drowned. Francis Bacon retold it in the Novum Organum in 1620. The pictures prove only that prayer is followed, sometimes, by survival.

A book about 5 a.m. routines is a wall of votive paintings. The people who rose at five and built a company are visible, and the people who rose at five and burned out in a year don’t get keynotes. The same goes for the college dropout who became a billionaire, whose story is told as evidence about dropping out when the number you would need, how many people left and what happened to them, is never printed. David McRaney has called the advice business “a monopoly run by survivors.”

The sample is also filtered by who feels entitled to a lesson. A successful person looks back for causes and finds the habits she still has, which are the ones that suited her. Starting from everyone instead of from the famous can shrink a claim, as in the 2014 pooled analysis of deliberate practice. Even a charming rule can be a survivor’s artifact: the Seinfeld calendar chain is a method attached to one famous name through a story its supposed source later disowned, with no count of how many other people tried it and quit.

Where are the bullet holes in your own mornings?

Run Wald’s question on yourself. If you ask how your mornings have gone lately, you answer from the planes that landed: the day you ran at six, the week things clicked. The mornings you missed left no hole to photograph. You remember the intention, and the intention was always fine.

The gaps are not random either. Wald’s whole point was that fatal hits cluster in particular places, and so do missed mornings: Sunday night, the third bad sleep in a row, the trip, the week of the deadline. Those are the vulnerable spots in your plan, and because they produce no memorable record, your private estimate of yourself (“basically an early riser, a few slips”) is computed from survivors. Feynman’s story of rats that learned to hear the floor describes the same trap from the other side: a self-report draws on whatever is cheapest to recall.

The fix is the one Wald used, adapted to a smaller problem. Don’t ask what the landed days show. Ask what the missing ones would have shown, and then make them show something. A miss has to be written down the same day, with the weekday and the night before, or it evaporates. It also helps if somebody other than you holds the record, since a private tally is edited by its owner.

What does a log look like when the failures are visible?

Having friends hold the record is the reason I think a tool like DontSnooze matters at all for this problem. Its rule is that if you haven’t filmed yourself awake within 30 seconds of the alarm, a random photo from your camera roll goes to friends you chose. Leave the motivational claims aside; the data effect is that a miss becomes an event somebody else can see, where before it was an absence only you could notice.

It is one instrument, with limits. It tells you that you missed, not why, and it records only the mornings that run through the app. Friends are a partial witness too: in one personal 47-morning tally, 19 proof videos drew any visible reaction. A group of five people will not see everything, and a person determined to hide a bad stretch can still go quiet. What changes is the starting point: a morning log that includes the failures at least has holes in it where the failures happened.

Wald never saw the planes that went down. He put a number on them anyway, and a morning record with the misses in it does a small version of the same thing.