Metrics are based on humanness

In healthcare, we understand that metrics have limits. So when we measure diabetes control, we know we're not seeing all of healthcare quality. It's only part of the picture. And we know that each number has a story behind it. For example, diabetes control in a disadvantaged population means something different than in a wealthy population. There are multiple factors underlying the number, and to understand it, we need to ask questions and have conversations. 

I like to say that "metrics are good, but thinking is indispensable."

Following are some interesting thoughts on metrics from Dr. David Shaywitz's December blog post at Forbes, and other stuff he points to. 

In Nobember 1984, when spreadsheets were permeating the business world, Steven Levy wrote a thoughtful essay in Harper's on how spreadsheets were transforming business thought. (The article's title was "A Spreadsheet Way of Knowledge," apparently an allusion to Carlos Castenada's book, The Teachings of Don Juan, a Yaqui Way of Knowledge, which is a highly mystical story of a Native American shaman):

• Measuring something makes it easy to ignore what is not measured. "The aspects that get emphasized are the ones easily embodied by numbers. ... Even a hostile takeover seems cut and dried, perfectly logical, in the world of spreadsheets. The spreadsheet user has no way of quantifying a corporate tradition or the misery of stockholders or whether the headaches of a drawn out takeover bid will ultimately harm the corporate climates of the firms involved."
• Spreadsheets and mathematical models are based on assumptions with highly variable reasonableness. "The accuracy of a spreadsheet model is dependent on the accuracy of the formulas that govern the relationships between various figures. These formulas are based on assumptions made by the model maker. An assumption might be an educated guess about a complicated cause-and-effect relationship. It might also be a wild guess, or a dishonestly optimistic view. ... People tend to forget that even the most elegantly crafted spreadsheet is a house of cards, ready to collapse at the first erroneous assumption "
• Spreadsheets are "a metaphor," and there are other kinds of metaphors apart from math.  

From Frank Pasquale of Yale Law School: 

• Don't let metrics be the only thing that tells you what is good: "There is an almost evangelical quality to [metric-driven] work, a passionate belief that older, intuition-driven decisions are a sinful relic of a fallen world…. [Metrics] colonize our own understandings of merit.”

From The Black Swan author Nassim Taleb: 

• Measurement "increases overconfidence": "Risk measurement and prediction —any prediction — has side effects of increasing risk-taking, even by those who know that they are not reliable. We have ample evidence of so called ‘anchoring’ in the calibration of decisions."

Please let me know what you think!

Comments

[Daniel Metrick]

See also: Goodheart's Law- by measuring and rewarding one thing, you cause neglect of another thing to free up resources for the thing being measured.

[Amin S]

I enjoy reading your articles.
True, from a mathematical perspective not every subset is measurable. Also even if a subset is measurable, understanding the measure can be extremely complex. We can easily understand and feel one dimensional measures, but beyond 1 dimension, things become a lot harder to grasp and make sense of what the measure is telling us...even worse, a 2 dimentional space or higher is not a well order space, for example: let’s assume that diabetes is a 2 dimentional problem (only 2 factors to consider) and we have 2 treatments that resulted in (2,1) and (1,2). Which treatment is better is the same as asking: which is bigger (1,2) or (2,1)? We can not compare!
So is not having a measure at all better than developing a non perfect measure? ( like reducing many dimensions of the problem to 1 dimension or 1 number) I don’t think so. But as you said thinking is indispensable.

[Bob Ludwig]

Another idea to remember is that the phrase, "If you can't measure it, you can't manage it," is when stated baldly, false.

Anytime the term 'measure' springs up, pause. Frequently what you want to be able to do is, 'assess'. Using 'assess' implies examining the situation rather than assuming you've got something unequivocally 'the same' and 'discrete' - over time.