Window and Scoreboard Metaphor to Understand Statistics

There are two kinds of people in life: those who like crude dualities and those who do not. And now that I’ve unmasked myself as the first, please allow me to introduce a statistical distinction I find helpful: windows vs. scoreboards.

A “window” is a number that offers a glimpse of reality. It does not feed into any incentive scheme. It cannot earn plaudits or incur punishments. It is a rough, partial, imperfect thing—yet still useful to the curious observer. Think of a psychologist asking a subject to rate his happiness on a scale from 1 to 10. This figure is just a crude simplification; only the most hopeless “1” would believe that the number is happiness.

Or imagine you’re a global health researcher. It’s not possible to quantify the physical and mental well-being of every human in a country. Instead, you look to summary statistics: life expectancy, childhood poverty, Pop-Tarts per capita. They’re not the whole reality, but they’re valuable windows into it.

The second kind of metric is a “scoreboard.” It reports a definite, final outcome. It is not a detached observation, but a summary judgment, an incentive scheme, carrying consequences.

Think of the score in a basketball game. Sure, bad teams sometimes beat good ones. But call the score a “flawed metric for team quality,” and people will shoot you the side-eye. You don’t score points to prove your team’s quality; you improve your team’s quality to score more points. The scoreboard isn’t a rough measure, but the desired result itself.

Or consider a salesperson’s total revenue. The higher this number, the better you’ve done your job. End of story.

A single statistic may serve as window or scoreboard, depending on who’s looking. As a teacher, I consider test scores to be windows. They gesture at the truth but can never capture the full scope of mathematical skills (flexibility, ingenuity, affection for “sine” puns, etc.). For students, however, tests are scoreboards. They’re not a noisy indicator of a nebulous long-term outcome. They are the outcome.

Many statistics make valuable windows but dysfunctional scoreboards. For example, heed the tale of British ambulances. In the late 1990s, the UK government instituted a clear metric: the percentage of “immediately life-threatening” calls that paramedics reached within eight minutes. The target: 75%.

Nice window. Awful scoreboard.

First, there was data fudging. Records showed loads of calls answered in seven minutes and 59 seconds; almost none in eight minutes and one second. Worse, it incentivized bizarre behavior. Some crews abandoned their ambulances altogether, riding bicycles through city traffic to meet the eight-minute target. I’d argue that a special-built patient-transporting truck in nine minutes is more useful than a bicycle in eight, but the scoreboard disagreed.

Replication of Previous Research

If a finding is true, then rerunning the experiment should generally yield the same outcome. If it’s false, then the result will vanish like a mirage.

**

Replication is slow, unglamorous work. It takes time and money while producing nothing new or innovative. But psychology knows the stakes and is beginning to face its demons. One high-profile project, published in 2015, performed careful replications of 100 psychological studies. The findings made headlines: 61 of the 100 failed to replicate.

In that grim news, I see progress. The research community is taking a sobering look in the mirror and owning up to the truth, as ugly as it may be. Now social psychologists hope that other fields, such as medicine, will follow their lead.

Science has never been defined by infallibility or superhuman perfection. It has always been about healthy skepticism, about putting every hypothesis to the test. In this struggle, is the field of statistics an essential ally. Yes, it has played a part in bringing science to the brink, but just as surely, it will play a part in bringing science back.

Not all 0.04s are created equal!

In 1925, a statistician named R. A. Fisher published a book called Statistical Methods for Research Workers. In it, he proposed a line in the sand: 0.05. In other words, let’s filter out 19 of every 20 flukes.

Why let through the other one in 20? Well, you can set the threshold lower than 5% if you like. Fisher himself was happy to consider 2% or 1%. But this drive to avoid false positives incurs a new risk: false negatives. The more flukes you weed out, the more true results get caught in the filter as well.

Suppose you’re studying whether men are taller than women. Hint: they are. But what if your sample is a little fluky? What if you happen to pick taller-than-typical women and shorter-than-typical men, yielding an average difference of just 1 or 2 inches? Then a strict p-value threshold may reject the result as a fluke, even though it’s quite genuine.

The number 0.05 represents a compromise, a middle ground between incarcerating the innocent and letting the guilty walk free.

For his part, Fisher never meant 0.05 as an ironclad rule. In his own career, he showed an impressive flexibility. Once, in a single paper, he smiled on a p-value of 0.089 (“some reason to suspect that the distribution… is not wholly fortuitous”) yet waved off one of 0.093 (“such association, if it exists, is not strong enough to show up significantly”).

To me, this makes sense. A foolish consistency is the hobgoblin of little statisticians. If you tell me that after-dinner mints cure bad breath (p = 0.04), I’m inclined to believe you. If you tell me that after-dinner mints cure osteoporosis (p = 0.04), I’m less persuaded. I admit that 4% is a low probability. But I judge it even less likely that science has, for decades, overlooked a powerful connection between skeletal health and Tic Tacs.

All new evidence must be weighed against existing knowledge. Not all 0.04s are created equal.

Experimental Results

Scientists want true positives. They are known as “discoveries” and can win you things like Nobel Prizes, smooches from your romantic partner, and continued funding.

True negatives are less fun. They’re like thinking you’d tidied the house and done the laundry, only to realize that, nope, that was just in your head. You’d rather know the truth, but you wish it were otherwise.

By contrast, false negatives are haunting. They’re like looking for your lost keys in the right place but somehow not seeing them. You’ll never know how close you were.

Last is the scariest category of all: false positives. They are, in a word, “flukes,” falsehoods that, on a good hair day, pass for truths. They wreak havoc on science, sitting undetected in the research literature for years and spawning waste-of-time follow-ups. In science’s never-ending quest for truth, it’s impossible to avoid false positives altogether—but it’s crucial to keep them to a minimum.

That’s where the p-value comes in. Its whole purpose is to filter out flukes.

Statistics

A statistic is an imperfect witness. It tells the truth, but never the whole truth.

Restrictions Birth Creativity

Human laws are flexible, subject to repeal and renegotiation. The drinking age is 21 in the US, 18 in Britain, 16 in Cuba, “never” in Afghanistan, and “step right up” in Cambodia. Any country can stiffen or relax its drinking laws at a whim (and of course, the stiffer your drink and the laxer your laws, the more whim-prone you become). Geometry’s laws are not like that. There’s no wiggle room: no president to issue pardons, no jury to acquit, no officer to let you off with a warning. Math’s rules are self-enforcing, unbreakable by their nature.

Yet as we’ve seen, and shall see many times again, that’s not a bad thing. Restrictions birth creativity. The laws on what shapes can’t do come packaged with case studies illuminating what they can. In design projects ranging from sturdy buildings to useful paper to planet-destroying space stations, geometry inspires even as it constrains.

Why Babies Need Blankies?

Although the cutting edge of parenting advice evolves minute by minute, some principles never change: cuddles are good; head trauma is bad; and swaddling your little one is a must. We’ve been wrapping our young since the Paleolithic, and thousands of years from now, I’m sure that the survivors of the zombie apocalypse will still be swaddling their traumatized infants.

Babies need blankets because—please forgive the technical jargon—babies are small.

Once again, ignore the details: the tiny toothless mouth, the eensy wiggling toes, the small bald head that smells so amazing. Think of a baby the way you’d think of any organism: as a homogenous bundle of chemical reactions. Every activity of the body is built on such reactions; in some sense, the reactions are the creature. That’s why animals are so temperature-sensitive: Too cold, and the reactions slow to a halt; too hot, and some chemicals deform, making key reactions impossible. You’ve got to keep a close eye on the thermostat.

Heat is created by reactions in each cell (i.e., in the interior). And heat is lost through the skin (i.e., at the surface). This creates a familiar tug-of-war: interior vs. surface.

Bigger animals, being more interior-heavy, will have an easy time keeping warm. Smaller ones, being surface-heavy, will struggle. That’s why you’re most vulnerable to cold in your surface-heavy extremities: fingers, toes, and ears. This also explains why cold climates support only big mammals: polar bears, seals, yaks, moose, mooses, meeses, and (depending on your zoology professor) the Sasquatch. A surface-heavy mouse wouldn’t stand a chance in the Arctic. Even at moderate latitudes, mice cope with heat loss by eating a quarter of their body weight in food each day.

A baby isn’t a mouse, but it’s definitely not a yak. Its tiny body expends heat like governments spend money. And to stifle that heat loss, there’s no cuddlier option than a blankie.

Creativity vs Rules... and Math

You can make the case that all creative endeavors are about pushing against constraints. In the words of physicist Richard Feynman, “Creativity is imagination in a straitjacket.” Take the sonnet, whose tight formal restrictions—Follow this rhythm! Adhere to this length! Make sure these words rhyme! Okay… now express your love, lil’ Shakespeare!—don’t undercut the artistry but heighten it. Or look at sports. Humans strain to achieve goals (kick the ball in the net) while obeying rigid limitations (don’t use your hands). In the process, they create bicycle kicks and diving headers. If you ditch the rulebook, you lose the grace. Even the wacky, avant-garde, convention-defying arts—experimental film, expressionist painting, professional wrestling—draw their power from playing against the limitations of the chosen medium.

Creativity is what happens when a mind encounters an obstacle. It’s the human process of finding a way through, over, around, or beneath. No obstacle, no creativity.

But mathematics takes this concept one step further. In math, we don’t just follow rules. We invent them. We tweak them. We propose a possible constraint, play out its logical consequences, and then, if that way leads to oblivion—or worse, to boredom—we seek a new and more fruitful path.

How To Think Like A Mathematician

To be honest, mathematicians don’t do much. They drink coffee, frown at chalkboards. Drink tea, frown at students’ exams. Drink beer, frown at proofs they wrote last year and can’t for the life of them understand anymore.

It’s a life of drinking, frowning, and, most of all, thinking.

Rather, the verbs of the mathematician all boil down to actions of thought. When we calculate, we turn one abstraction into another. When we give proofs, we build logical bridges between related ideas. When we write algorithms or computer programs, we enlist an electronic brain to think the thoughts that our meat brains are too slow or too busy to think for themselves.

Every year that I spend in the company of mathematics, I learn new styles of thought, new ways to use that nifty all-purpose tool inside the skull: How to master a game by fussing with its rules. How to save thoughts for later, by recording them in loopy Greek symbols. How to learn from my errors as if they were trusted professors. And how to stay resilient when the dragon of confusion comes nibbling at my toes.

In all these ways, mathematics is an action of the mind.

Math With Bad Drawings

One day that September, I found myself leading an awkward impromptu discussion of why we study geometry. Did grown-ups write two-column proofs? Did engineers work in “no calculator” environments? Did personal finance demand heavy use of the rhombus? None of the standard justifications rang true. In the end, my 9th graders settled on “We study math to prove to colleges and employers that we are smart and hardworking.” In this formulation, the math itself didn’t matter. Doing math was a weightlifting stunt, a pointless show of intellectual strength, a protracted exercise in résumé building. This depressed me, but it satisfied them, which depressed me even more.

The students weren’t wrong. Education has a competitive zero-sum aspect, in which math functions as a sorting mechanism. What they were missing—what I was failing to show them—was math’s deeper function.

Why does mathematics underlie everything in life? How does it manage to link disconnected realms—coins and genes, dice and stocks, books and baseball? The reason is that mathematics is a system of thinking, and every problem in the world benefits from thinking.