In healthcare, we understand that metrics have limits. So when we measure diabetes control, we know we're not seeing all of healthcare quality. It's only part of the picture. And we know that each number has a story behind it. For example, diabetes control in a disadvantaged population means something different than in a wealthy population. There are multiple factors underlying the number, and to understand it, we need to ask questions and have conversations.
I like to say that "metrics are good, but thinking is indispensable."
Following are some interesting thoughts on metrics from Dr. David Shaywitz's December blog post at Forbes, and other stuff he points to.
In Nobember 1984, when spreadsheets were permeating the business world, Steven Levy wrote a thoughtful essay in Harper's on how spreadsheets were transforming business thought. (The article's title was "A Spreadsheet Way of Knowledge," apparently an allusion to Carlos Castenada's book, The Teachings of Don Juan, a Yaqui Way of Knowledge, which is a highly mystical story of a Native American shaman):
- Measuring something makes it easy to ignore what is not measured. "The aspects that get emphasized are the ones easily embodied by numbers. ... Even a hostile takeover seems cut and dried, perfectly logical, in the world of spreadsheets. The spreadsheet user has no way of quantifying a corporate tradition or the misery of stockholders or whether the headaches of a drawn out takeover bid will ultimately harm the corporate climates of the firms involved."
- Spreadsheets and mathematical models are based on assumptions with highly variable reasonableness. "The accuracy of a spreadsheet model is dependent on the accuracy of the formulas that govern the relationships between various figures. These formulas are based on assumptions made by the model maker. An assumption might be an educated guess about a complicated cause-and-effect relationship. It might also be a wild guess, or a dishonestly optimistic view. ... People tend to forget that even the most elegantly crafted spreadsheet is a house of cards, ready to collapse at the first erroneous assumption "
- Spreadsheets are "a metaphor," and there are other kinds of metaphors apart from math.
From Frank Pasquale of Yale Law School:
- Don't let metrics be the only thing that tells you what is good: "There is an almost evangelical quality to [metric-driven] work, a passionate belief that older, intuition-driven decisions are a sinful relic of a fallen world…. [Metrics] colonize our own understandings of merit.”
From The Black Swan author Nassim Taleb:
- Measurement "increases overconfidence": "Risk measurement and prediction —any prediction — has side effects of increasing risk-taking, even by those who know that they are not reliable. We have ample evidence of so called ‘anchoring’ in the calibration of decisions."
Please let me know what you think!