Base Rates for the Impatient

Kahneman and Tversky ran the following experiment in 1973, and variants have been run for fifty years with consistent results. Subjects read a short description of an individual: thoughtful, tidy, introverted, writes well, neat handwriting. They are told this description was drawn from a sample of seventy lawyers and thirty engineers. Which is the individual more likely to be?

Most subjects answer engineer. The description is a better fit for the stereotype of an engineer than a lawyer. The answer that follows from the numbers is the opposite. Seventy lawyers and thirty engineers means any given individual is more than twice as likely to be a lawyer, regardless of their tidiness, unless the description is so extreme that it overcomes the 7-to-3 base rate. "Thoughtful and neat" is nowhere near extreme enough.

This is the base rate fallacy, and it is so robust across subject populations, including populations of statisticians and professional forecasters, that Kahneman spent decades of his career returning to it. The finding is not that people ignore base rates when asked. The finding is that people neglect base rates almost entirely unless specifically prompted, and that the prompting has to be recent. Within a week of reading about the fallacy, subjects still make the error.

Why this matters to operators

Operators forecast continuously. Will the deal close. Will the hire work out. Will the launch hit the number. Will the competitor move first. In practice, almost none of these forecasts explicitly reference a base rate, and almost all of them are dominated by case-specific detail: the founder seems sharp, the pipeline looks strong, the market feels ready.

The base rate the forecaster should have consulted is boring. What percentage of deals at this stage close? What percentage of hires at this level make it past year two? What percentage of launches hit the original number? For most operators, the honest answers are uncomfortable: roughly 20-30% of pipeline deals close, roughly 40-50% of senior hires work out, almost no launches hit the original number. These numbers are available, approximately, from any operator's own history. They are consulted rarely.

When they are not consulted, the forecast is dominated by the feeling of the specific case. The feeling is almost always more optimistic than the historical rate, because the specific case is vivid and the historical rate is aggregate.

The superforecaster finding

Philip Tetlock's 2015 book Superforecasters reports on the Good Judgment Project, a multi-year competition to forecast geopolitical events. The best forecasters outperformed subject-matter experts by 60% or more. Tetlock and colleagues published the inside of the practice: the single most important procedural move the best forecasters made, more important than information access or intelligence, was starting with a base rate and adjusting, rather than starting with the specific case and tweaking.

This is directional. Most forecasters, professional and amateur, start with the specific case ("this one looks like it'll close because the buyer is decisive and the price point is right"). The superforecaster starts with the reference class ("what's the base rate for B2B SaaS deals of this size at this stage?") and then considers what about this specific case, if anything, should move the forecast off the base rate by how much.

The second move, sensibly handled, can only adjust the first. It cannot replace it. The error pattern is that the specific case replaces the reference class, and the reference class never gets consulted.

The five-minute practice

An operator who wants the Kahneman-Tetlock correction can adopt a specific habit that takes about five minutes per forecast and applies to most forecasts worth writing down.

Step one: name the reference class. What is the class of events this forecast belongs to? "B2B SaaS deals at $200K ACV, post-demo, with CFO in the room" is a reference class. "This specific deal" is not a reference class; it is the thing being forecast.

Step two: estimate the base rate. What percentage of events in this reference class result in the outcome being forecast? The estimate can be approximate. An operator's own memory is often sufficient; a rough external datapoint is better when available. For most forecasts, writing down the base rate on a sheet of paper, without further analysis, produces a number in the 15-40% range for outcomes the operator was intuitively estimating at 60-80%.

Step three: name up to three specific-case adjustments. What is true about this specific case that should move the forecast meaningfully off the base rate? No more than three. Each must include a specific claim about how much it moves the probability and why. "Strong champion (+10%)." "Competitor in procurement (-10%)." "Q4 timing (+5%)."

Step four: report the adjusted base rate, not the case feeling. The forecast is the base rate plus or minus the adjustments. If the base rate is 25% and the adjustments net to +10%, the forecast is 35%. The forecast is almost always lower than the case feeling said it would be. The gap between the forecast and the case feeling is the calibration tax the base rate is charging the operator, which was already there but was not being paid explicitly.

The compound effect

An operator who runs this practice on fifty forecasts over six months will notice a specific thing. Their calibration improves, not because they became more accurate at any given forecast, but because they stopped systematically overstating confidence in the specific-case direction. Brier scores, if they keep them (see /library/two-kinds-of-confidence), improve by 10-20% over this period. The improvement is not from getting cleverer. It is from ceasing to ignore a piece of information the mind is actively wired to skip, which is that this specific case is not the only case of its kind, and that the class this case belongs to has a history.

The practice is boring. That is partly why it works. The question "what's the base rate" is a deliberately unglamorous question. It interrupts the vividness of the specific case long enough for the reference class to get consulted. Kahneman's fifty-year program converges on exactly this one corrective, applied reliably, over time. There is no more sophisticated version.