Jerry Muller’s book on The Tyranny of Metrics argues that metrics are overused. (I have discussed other frustrations with the book here). Muller, however, very neatly illustrates the problem with his approach. Anecdotal evidence, by this here I mean stories not rigorously detailed, is problematic in justifying an argument. Basically he throws some numbers into his story to give it an appearance of using evidence. Yet his anti-numbers attitude leads him to use the numbers so casually that he claims the numbers support his argument when they really don’t. The strange things is I basically agree with the thrust of his point in this specific case. If metrics are in control of his life, as his claim of tyranny seems to imply, I can only recommend that he should get better at understanding them.
Metrics Get Us Past The Fact That Anecdotal Evidence Is Problematic
Throughout Muller’s book I agreed with Muller that problems exist and he definitely points to some challenges. But his lax attitude to evidence leads to confusing diagnoses. Are the stories he presents about the use of the wrong metric or the use of a metric badly rather than the idea of any use of metrics being wrong in that situation?
Yes, many metrics are misused. Many are used for purposes for which they are not useful. That seems to me an argument to get better acquainted with metrics so you can see the flaws in their use. If you just throw your hands up and scream ‘tyranny’ whenever you see a number I don’t see how we can make progress.
I Disagree With Muller While Agreeing That What He Criticizes Is Shocking
The challenge of Muller’s failure to engage with metrics becomes apparent when he casually uses numbers to support his position. It shows a person who has a limited understanding of metrics. His knowledge of business seems equally poor. He claims evidence shows a problem for the firm. I agree there is a problem, certainly for society and likely for the firm, but the evidence he presents doesn’t really show this.
I will solely use the evidence that Muller presents. To be clear, I believe the evidence exists to make the case that he wants to make. My objection is not to his argument, per se, but the way he presents his argument.
The Mylan Case
The case he discusses is that of Mylan, who sell the Epipen. The company bought the rights to sell the Epipen. Without meaningful competition, they were able to greatly increase the price to vulnerable people who had no option but to buy it. I am not an expert on this but it looks like a frightening story of greed and lack of meaningful regulation.
Ripping Off Consumers To Benefit Shareholders
To my mind, there is a clear villain in this piece. Pay for performance metrics are certainly part of the story but they only incentivize people to do what the firm sets as the targets. The real villain is the simplistic idea that firms are there only to maximize shareholder value. The company seemed to incentivize the executives to try to maximize shareholder value by ripping off vulnerable customers. It is pretty despicable but it is a logical consequence of Milton Friedman’s argument that this is all that firms are for. Friedman did put in a caveat into his argument about ethics but I see this as an ethical joker, see Friedman’s Ethical Joker. In Friedman’s theory what should not be done for reasons of ethics is unspecified and so seems just like a get-out-of-jail-free card if the political pressure becomes too hot.
The point is that, to my mind, it wasn’t the use of metrics per se that caused a problem. Instead, it was the ideology that said preying on vulnerable customers was the fiduciary responsibility of the executives. The executives did what the firm wanted them to do. In that sense, the performance metrics did their job as part of an incentive system. It was just the definition of performance the company used that was a problem. (Rather than the metrics that measured performance). Still what actually happened isn’t really my point in this post. Instead, I want to discuss Muller’s use of evidence.
Muller’s Use Of Anecdotal Evidence Is Problematic
Muller wants to argue that performance metrics caused the problem so he tells us that:
When Bresch took charge as CEO [Jan 2012], the stock price stood at $22
Muller, 2018, page 141
He gives us this as a baseline. Now if the firm only cares about share price and metrics have caused a problem for the firm, as he seems to imply, the share price really needs to go down for his example to work. (This is a simplification but Muller doesn’t mention any other major changes — like corporate restructurings — and doesn’t mention inflation or a rising stock market. He hates numbers so I’m pretty sure he isn’t making a sophisticated argument about other potential uses of capital and positive but sub-optimal returns).
Price Rises And Share Price
He discusses the obscene price increases that occurred after Mylan bought the rights to the Epipen in 2007. Some of the increases were before Heather Bresch got the job, but many were after. Clearly, the price was rising before Bresch took over. The exact causal process he is alleging is unclear. Still, it seems reasonable to me to conclude, as Muller does, that the executives were getting rich on the backs of their vulnerable customers.
Muller says, correctly, that these price rises caused an outcry but it still isn’t clear in his argument how exactly the firm was hurt. (Charges swirled and Bresch apologized but she stayed in her post). The cautionary tale needs to show that it hurt the firm. The share price did fall from the giddy heights it reached when the stock market originally heard about the price rises and initially thought these were a wonderful idea. (Yep). Yet Muller ends by giving a number that is supposed to show how it went wrong for the firm which undermines his entire argument. Almost as though he thinks metrics are so tyrannical he won’t do simple math. Just because a price rose and then fell doesn’t mean it fell over the entire period.
…outcry against the company… led the price to drop to $36 in October 2016. The top executives’ single-minded focus on hitting outsized profit metrics had led to a collapse of the company’s reputation.
Muller, 2018, page 142
But The Share Price Went Up In His Evidence
When he ends his story the share price has actually gone up under Bresch’s tenure. (More has happened since but I’m only interested in the data he uses to support his story).
The evidence he showed tells us that the company almost certainly increased shareholder value over the period. The share price went up from $22 to $36 in only four years. His evidence doesn’t support his case that this was a disaster for the firm. The firm’s use of metrics could not have really hurt the firm because they weren’t hurt according to the evidence he showed. Of course, customers were badly hurt. (And that is the real scandal). Still, he argued the firm was hurt but didn’t show evidence of this.
Using Evidence
Again I want to repeat that Muller was right to criticize Mylan. The story is shocking but you need to be clear on what happened. It did not seem to be metrics that caused any problem. It was the firm’s narrow shareholder-focused philosophy and probably lack of regulation. Still, when you want to argue that a problem was caused for the firm, rather than the customers, make sure your numbers show there was a problem from the firm’s perspective. Anecdotal evidence is problematic. Maybe a metric related to social responsibility or corporate reputation would have helped to illustrate the point he wanted to make. If only he thought deeply about metrics.
Read: Jerry Z. Muller (2018) The Tyranny of Metrics, Princeton University Press
