Can You Really Manage What You Measure?

Originally published February 5, 2009

Measurement is at the core of performance management, but is it really possible to use measurement – or metrics in the current parlance – to drive an organization? There are two points of view, one widely accepted and current, the other opposing and more abstract.  

The conventional wisdom is that our technology is perfectly capable of providing detailed, current and relevant performance information to stakeholders in an enterprise, including executives, managers, functional people, customers, vendors and regulators. Because we are blessed with abundant computing resources, connectivity, bandwidth and even standards, it is possible to present this information in cognitively effective ways (dashboards and visualization, for example). Recipients are able to receive the information in the manner in which they choose, and the whole process pays dividends by supporting the notion that "If you can't measure it, you can't manage it." It is hard to imagine how anyone could manage a large undertaking without measurement, isn't it? And most presentations I've heard quickly stress that measurement is only part of the solution.

The first step is knowing what to measure; then measuring it accurately; then finding a way to disseminate the information for maximum impact (figuring out how to keep it current and relevant); and then being able to actually do something about the results. A different way of saying this is that technology is never a solution to social problems, and interactions between human beings are inherently social. This is why performance management is a very complex discipline, not just the implementation of dashboard or scorecard technology. Luckily, the business community seems to be plugged into this concept in a way they never were in the old context of business intelligence. In this new context, organizations understand that measurement tools only imply remediation and that business intelligence is most often applied merely to inform people, not to catalyze change. In practice, such undertakings almost always lack a change management methodology or portfolio. 

But there is an argument against measurement too. Unlike machines or chemical reactions in a beaker, human beings are aware that they are being measured. In the realm of physics, Heisenberg's Uncertainty Principle demonstrates that the act of measurement itself can very often distort the phenomena one is attempting to measure. When it comes to sub-atomic particles, we can pretty much assume it is a physical law that underlies this behavior. With people, the unseen subtext is clearly conscious. People find the most ingenious ways to distort measurement systems to generate the numbers that are desired, not only NOT providing the desired behaviors, but often becoming more dysfunctional through the effort. There are excellent, documented examples of this in Measuring and Managing Performance in Organizations by Robert D. Austin. The author’s contention is that measurement of people always introduces distortion and often brings dysfunction because measurement is never more than a proxy or an approximation of the real phenomena.

In a particularly colorful analogy, Austin writes:

“Kaplan and Norton’s cockpit analogy would be accurate if it included a multitude of tiny gremlins controlling wing flaps, fuel flow, and so on of a plane being buffeted by winds and generally struggling against nature, but with the gremlins always controlling information flow back to the cockpit instruments, for fear that the pilot might find gremlin replacements. It would not be surprising if airplanes guided this way occasionally flew into mountains when they seemed to be progressing smoothly toward their destinations.”

We all know that incomplete proxies are too easy to exploit in the same way that inadequate software with programming gaps beckons unscrupulous hackers. However, one doesn't have to be malicious to subvert a measurement system. After all, voluntary compliance to the tax code encourages a national obsession with "loopholes," and what salesman hasn't "sandbagged" a few deals for next quarter after she's met her quota for the current one? 

The solution is not to discard measurement but rather to be conscious of this tendency and to be vigilant and thorough in the design of measurement systems. We all have a tendency toward simplifying things; but in some cases, it appears better to not measure at all than to produce something inadequate. Performance management, to achieve its goals, has to be applied effectively, which is to say, with superior execution of technology, implementation and management. It has to be designed to be responsive to both incremental and unpredicted changes in the organization and the environment. There are no road maps for this. This is truly the first time that analytical and measurement technology can be embedded in day-to-day, instantaneous decision making and tracking; and the industry is sorely lacking in skills and experience to pull it off. Those organizations that have been successful so far have relied on existing methodologies (activity-based costing or balanced scorecard, for example) to guide them through the more uncertain steps of metric formulation and change management to close the loop.

The question of whether you can ever adequately measure an organization is still open. To the extent that there are statutory and regulatory requirements, such as taxation, SEC or specific industry regulations, the answer is clearly yes. But those measurements are dictated. To measure performance after the fact, at aggregated levels, is only useful to a point. The closer and closer a measurement system gets to the actual events and actions that drive the higher level numbers, the less reliable the cause-effect relationship becomes, just like Heisenberg found so long ago. There are many examples in the management literature of everyone "doing the right thing" while the wheels are coming off the organization.

Recommended Reading:

Robert D. Austin, Measuring and Managing Performance in Organizations (New York: Dorset House Publishing, 1996)

Shoshana Zuboff, In the Age of the Smart Machine: The Future of Work and Power (New York, Basic Books, 1988) 

Peter Drucker, "The New Productivity Challenge," Harvard Business Review, (November-December 1991): 70.

Peter Drucker, "The Information Executives Truly Need," Harvard Business Review, (January-February 1995): 54-62        

SOURCE: Can You Really Manage What You Measure?

  • Neil RadenNeil Raden

    Neil Raden is an "industry influencer" – followed by technology providers, consultants and even industry analysts. His skill at devising information assets and decision services from mountains of data is the result of thirty years of intensive work. He is the founder of Hired Brains, a provider of consulting and implementation services in business intelligence and analytics to many Global 2000 companies. He began his career as a casualty actuary with AIG in New York before moving into predictive modeling services, software engineering and consulting, with experience in delivering environments for decision making in fields as diverse as health care to nuclear waste management to cosmetics marketing and many others in between. He is the co-author of the book Smart (Enough) Systems and is widely published in magazines and online media. He can be reached at nraden@hiredbrains.com.

 

Comments

Want to post a comment? Login or become a member today!

Posted February 16, 2009 by Gareth Horton

Good article Neil,

Choosing what to measure and doing it well is critical.

Poorly thought out and costed incentives to individuals that are not correctly aligned with business objectives and the long term profitability and health of the business itself are common.

I have sometimes seen the scenario (vastly simplified) where salespeople are effectively in the situation of being able to sell £20 notes for £18.  Plus, with many revenue based incentives, the more they sell, the richer they become and the poorer the organization becomes.

This situation is usually in the cases of complex, long-term projects or, as we know, with long-term returns from financial intruments of dubious value.  The complexity tends to mask the problem, but incentives have to be much simpler and short-term, leading to this disconnect.

As you say, where there is an opportunity to game the system for personal financial gain, it will be taken.  This is especially true when they are really just playing by the rules within the structure laid down by those higher up, even when they can see that it will be to the detriment of that organisation.

This gets even more acute when the environment consists of multiple discrete organizations, such as the mortgage issue.  The broker's job is to sell a mortgage and be duly compensated that month/quarter, not to worry about what happens to the lender 5 or 10 years later. This disconnect continues on and on downstream to the eventual bagholder who then wishes they had more control of the process (which in turn, often means control of incentives!).

Admittedly, the "products" in this disaster were just bad products, but with the level of incentives that were floating around, who would want to rock the boat? Maybe it was greedy short-termism and maybe just Groupthink, but there were many, including the Chief Economist at HSBC that called this situation accurately many years ago.  This is a different league than armchair pundits like myself calling it based on the old "gut feel", common sense and a few pertinent facts, this is a pro in a top position. 

Judging by the number of £150,000+ plus cars in my town and new build houses priced in the multi-millions (popular with City of London types with families) that gravy train was a pretty nice place to be, and why not ride it until the eventual derailment?

Gareth

Is this comment inappropriate? Click here to flag this comment.

Posted February 6, 2009 by Peter Thomas

Interesting thoughts and a a lot of truth in what is said here. 

Is this comment inappropriate? Click here to flag this comment.