Oct 28 2011

The staying power of bad metrics

A speaker I once heard on manufacturing metrics started with a quote from football coach Vince Lombardi: “If you’re not keeping score, you’re only practicing.” In a sport, your score or your rank is, by definition, the correct measure of success, and we assume too easily that this kind of thinking crosses over to every human endeavor, from national economies to plant performance or education. In this process, we begin using highly aggregated metrics as if they were physical measurements like mass or speed, and avert our eyes from how these sausages are made.

Following are a few of the egregious examples:

  • GDP. Gross Domestic Product (GDP), for example, is in the news every day. If you pollute and spend money to clean up your toxic waste, you contribute more to the GDP than if you produce cleanly. Because of this kind of absurdity, GDP as a metric has been criticized by many economists, including Joseph Stiglitz. In 2009, he even convinced French president Nicolas Sarkozy to seek alternatives. Yet, two years later, the same president is pushing to include in the country’s constitution a “Golden Rule” that caps budget deficits at a percentage of the same flawed GDP!
  • IQ. In the US, IQ  is still widely treated as a measure of intelligence. On its face, the notion that human intelligence is reducible to a number is an insult to its subject. In fact, all an IQ measures is the ability to take an IQ test. Psychologists recognize this, but many school teachers and the public at large don’t. (See Steven Jay Gould’s The Mismeasure of Man.)
  • Food calories. Calories are the most commonly used metric in nutrition. What this number actually represents is the heat generated by drying and burning a food item. But is digestion the same as combustion? Obviously not, for example, for fibers, which cross the human body unchanged. The absurdity of assigning calories to fibers has not escaped one dieter, who questioned it on a Calorie Count forum, and received, among other replies, the following:

Fiber calories are included in nutrition information, but only in come countries. In the US, it is legal to not put in fiber calories because they are not digestible. Therefore, they do not “count” as such. however, if you, like most people, tend to underestimate cals sightly, there is nothing wrong with including them to create a “buffer zone.”

In other words, it makes no sense but you should pretend it does.

Do we behave the same way in the manufacturing world? Yes. For example, many companies measure productivity in terms of Sales/Employee. There is an easy way to boost this metric: outsource all production, close all plants and become a trading company. It is not easy to find metrics for quality, cost, delivery, safety, and morale that are meaningful and cannot be gamed, but it can be done. For overall company productivity, for example, you can use Value added/Employee, where

Value-added = Sales – (Materials + Energy + Outsourced Services)

This is what Peter Drucker called Contributed Value. Value added/Employee is not a perfect metric, but at least it does not provide a perverse incentive to outsource, and the US census bureau publishes statistics on value-added and employment by industry, that are helpful for benchmarking.

Following are a few conditions that a good metric must meet:

  1. A good metric is immediately understandable. No training or even explanation is required to figure out what it means, and the number directly maps to reality, free of any manipulation. One type of common manipulation is to assume that one particular ratio cannot possibly be over 85%, and redefine 85% for this ratio as “100% performance.” While this makes performance look better, it also makes the number misleading and difficult to interpret.
  2. People see how they can affect the outcome. With a good metric, it is also easy to understand what kind of actions can affect the value of the measurement. A shop floor metric, for example, should not a be a function of the price of oil in the world market, because there is nothing the operators can do to affect it. Their actions, on the other hand, can affect the number of labor-hours required per unit, or the rework rate.
  3. A better value for the metric always means better business performance for the company. One of the most difficult characteristics to guarantee is that a better value of a metric always translates to better business performance for the company. Equipment efficiency measures are notorious for failing in this area because maximizing them often leads to overproduction and WIP accumulation.
  4. The input data of the metric should be easy to collect. Lead time statistics, for example, require entry and exit timestamps by unit of production. The difference between these times then only gives you the lead time is calendar time, not in work time. The get lead times in work time, you then have to match the timestamps against the plant’s work calendar. Lead time information, however, can be inferred from WIP and WIP age data, which can be collected by direct observation of WIP on the shop floor. Metrics of
    WIP, therefore, contain essentially the same information but are easier to  calculate. (See Little’s Law.)
  5. All metrics should have the appropriate sensitivity. If daily fluctuations are not what is of interest, then they need to be filtered out. A common method for doing this is to plot 5-day moving averages instead of individual values — that is, the point plotted today is the average of the values observed in the last five days. Daily fluctuations are smoothed away, but weekly trends stand out.

Peter Drucker sold corporate America on the idea that you can’t manage what you can’t measure, and this has led many managers to believe that employees would do whatever it takes to maximize their scores. Given flawed metrics, it if fortunate for the companies that these managers were wrong. If they had been right, all the companies that measure productivity in terms of  Sales/Employee would actually have outsourced all production. They didn’t, because metrics are only one of many factors influencing behavior. Most employees, at all level, will not maximize their metrics through actions they feel violate common sense or are inconsistent with their personal ethics.

By • Metrics • 12 • Tags: ,