Measurement Error
Your metrics lie. Not maliciously—but systematically. Understanding how they lie prevents optimizing the wrong things.
Goodhart’s Law
“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart (Goodhart, 1984)
The moment you optimize for a metric, people (including you) game it. The metric stops measuring what you cared about. Donald Campbell observed the same phenomenon: “The more any quantitative social indicator is used for decision-making, the more subject it will be to corruption pressures” (Campbell, 1979).
| Original Goal | Metric | Gaming Behavior | Actual Result |
|---|---|---|---|
| Good health | Weight | Crash diets, dehydration | Metabolic damage |
| Learning | Test scores | Teaching to the test | Surface knowledge |
| Productivity | Hours worked | Presence without output | Burnout |
| Connection | Messages sent | Shallow check-ins | No deep relationships |
Types of Measurement Error
1. Proxy Distance
Metrics are proxies for what you actually care about. Some proxies are closer than others.
| Goal | Close Proxy | Distant Proxy |
|---|---|---|
| Health | Functional fitness, energy | Scale weight |
| Wealth | Net worth, savings rate | Income |
| Relationships | Quality of connection | Number of contacts |
| Meaning | Engagement, satisfaction | Achievements |
Rule: Measure the closest proxy available. Acknowledge the gap.
2. Lagging vs. Leading Indicators
| Type | Definition | Example | Problem |
|---|---|---|---|
| Lagging | Outcome after the fact | Weight, net worth, test results | Too late to course-correct |
| Leading | Predictor of outcome | Workouts done, savings rate, study hours | Doesn’t guarantee outcome |
Rule: Track both. Lead indicators for daily action. Lag indicators for periodic validation.
3. Survivorship Bias
You see what survived. You don’t see what failed. Abraham Wald famously identified this during WWII: the military wanted to armor the bullet holes on returning planes, but Wald realized they should armor where there were no holes—those planes didn’t make it back (Wald, 1943).
- Successful entrepreneurs → “Take risks!” (ignoring failed risk-takers)
- Healthy 90-year-olds → “Drink wine daily!” (ignoring those who died at 70)
Rule: Ask “What am I not seeing?“
4. Precision vs. Accuracy
- Precision: Consistent measurements (always says 150 lbs)
- Accuracy: True measurements (actually weighs 155 lbs)
A scale that’s always 5 lbs off is precise but inaccurate. Trends still work. Absolute values don’t.
Cross-Domain Measurement Errors
| Domain | Common Metric | What It Misses |
|---|---|---|
| Health | Weight | Body composition, energy, function |
| Wealth | Income | Expenses, savings rate, satisfaction |
| Social | Followers/friends count | Depth, reciprocity, support |
| Meaning | Accomplishments | Engagement, alignment, satisfaction |
| Meta | Tasks completed | Impact, priority, sustainability |
Defense Strategies
1. Multiple Metrics
Don’t rely on one number. Triangulate with several.
2. Qualitative Checks
Regularly ask: “Does the metric match my experience?“
3. Rotate Metrics
Change what you measure periodically to prevent gaming.
4. Measure Inputs AND Outputs
Track both what you do (leading) and what happens (lagging).
5. Question Outliers
Sudden changes often indicate measurement problems, not real changes.
Related
- Feedback Loops — Metrics enable feedback
- Weekly Review — Interpret metrics in context
- Incentives — Metrics create incentives
The map is not the territory. The metric is not the goal. Measure carefully. Interpret skeptically.