Why does every rating end up gamed?

Picture a car up on rollers inside a government lab, its tailpipe capped to a hose, engineers watching a screen read the exhaust in real time. Most cars cannot tell a test bay from a real road. This one could. Buried in its engine software sat code built to notice the exact signature of a laboratory test cycle: the fixed speeds, the steering wheel held dead straight, a drive that never once turns a corner. Only then did the software switch on the car’s full emissions controls. Drive the same car home from the lab and the controls quietly back off. The mileage improves. The engine breathes easier. And the exhaust carries far more of exactly what the test exists to catch.

On September 18, 2015, the Environmental Protection Agency issued Volkswagen a formal Notice of Violation of the Clean Air Act, naming the software for exactly what it was: any device that bypasses, defeats, or renders inoperative a required element of the vehicle’s emission control system. That notice covered the cars the EPA itself had tested on American roads, something like half a million of them. Four days later, on September 22, Volkswagen went further than any regulator had and admitted, on its own, that the same software rode inside roughly eleven million vehicles worldwide, sold across model years 2009 through 2015.

Every one of those cars passed its test, every single time regulators ran it in the room built for exactly that purpose. That is what makes this the cleanest case there is. The measurement never failed. The car had simply learned the measurement, rather than the road.

Give the law its real shape before you reach for it. In 1975, an economist named Charles Goodhart stood up at a conference in Sydney and made a narrow point, about central banks alone: any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. He meant interest rates and the money supply, nothing wider. The crisp, quotable line everyone actually repeats, when a measure becomes a target, it ceases to be a good measure, belongs to somebody else: the anthropologist Marilyn Strathern, writing in 1997 about grade inflation inside British universities. Strathern herself hands the name onward again, crediting a scholar named Hoskin with calling the pattern Goodhart’s law in the first place. Three names, two decades apart, two entirely different fields. None of them were writing about ratings or reviews. All three had found the same mechanism, walking in through a different door.

A parallel discovery came from a field with no connection to either one. Donald Campbell worked in program evaluation, watching how governments measured social programs, and reached the identical conclusion entirely on his own: the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. His own worked example still reads like this morning’s news: But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. Three thinkers, three fields, two decades, one shape underneath all of it: reward the number, and people move the number, rarely the thing it was standing in for.

Every gauge, every score, every KPI runs the same trade: a cheap, visible stand-in for something expensive to see directly. The stand-in stays honest only as long as nobody is graded on it. The moment a raise, a bonus, a rating, or a fine depends on the number moving, the number stops being a mirror and starts being a target, and targets get hit by whatever means turns out cheapest, intended or otherwise.

Here is the trade, drawn as an instrument. The dial reads quality, and its needle settles high, right where a good score is supposed to sit. Drag the slider and raise the effort spent gaming that score. Watch what happens underneath it.

The needle never moves again once it settles. That is the whole argument, made physical: a score can hold perfectly still while the thing underneath it keeps sliding, because nothing forces the gauge and the ground truth to agree once the room’s effort points at the gauge instead of the ground. A dial has no way to catch its own bad reading. That takes a person willing to look underneath it.

Walk onto a different floor entirely: a Wells Fargo branch, sometime in the years leading up to 2016. Every employee carried a number, a cross-sell quota, sold up to the bank’s own board as proof the branches were thriving. Miss the number for long enough and a manager heard about it. Some employees found the fastest way to hit it anyway: a savings account bolted onto a checking account, a credit card mailed to an address nobody in the room had confirmed, a signature the actual customer never gave. The board watched a beautiful, climbing chart. The chart was the gauge. A large share of the accounts underneath it belonged to customers who had never asked for them.

On September 8, 2016, the Consumer Financial Protection Bureau announced the reckoning: a $100 million fine, alongside $35 million to the Office of the Comptroller of the Currency and $50 million to the City and County of Los Angeles, $185 million in total. Wells Fargo’s own analysis, filed with that same announcement, put the number of unauthorized accounts at more than two million. That was the figure the fine was actually priced against.

A separate, outside review landed eight months later, in May 2017, and counted deeper than the bank’s first pass had gone. The revised figure came in near 3,500,000 accounts, a different count from a different moment, and never the number the original fine was priced against. Two disclosures, eight months apart, both true, easy to blur into one convenient headline. Keep them separate and the shape gets sharper rather than smaller: the fine came first, sized to what the bank admitted early. The fuller damage surfaced only once somebody outside the bank went looking harder.

Run the needle back over both cases and it lands in the same place twice. Volkswagen’s engines told the truth about the test, every time, precisely because the software had learned to recognize the test itself, and the test was the only gauge mounted on the dashboard. Wells Fargo’s board watched a chart of accounts climb for years while the number that chart was supposed to stand for, willing customers, quietly came apart from it underneath. Different industries, different decades, the same dial holding steady while the ground it was reading fell away.