The staying power of bad metrics

A speaker I once heard on manufacturing metrics started with a quote from football coach Vince Lombardi: “If you’re not keeping score, you’re only practicing.” In a sport, your score or your rank is, by definition, the correct measure of success, and we assume too easily that this kind of thinking crosses over to every human endeavor, from national economies to plant performance or education. In this process, we begin using highly aggregated metrics as if they were physical measurements like mass or speed, and avert our eyes from how these sausages are made.

Following are a few of the egregious examples:

  • GDP. Gross Domestic Product (GDP), for example, is in the news everyday. If you pollute and spend money to clean up your toxic waste, you contribute more to the GDP than if you produce cleanly. Because of this kind of absurdity, GDP as a metric has been criticized by many economists, including Joseph Stiglitz. In 2009, he even convinced French president Nicolas Sarkozy to seek alternatives. Yet, two years later, the same president is pushing to include in the country’s constitution a “Golden Rule” that caps budget deficits at a percentage of the same flawed GDP!
  • IQ. In the US, IQ  is still widely treated as a measure of intelligence. On its face, the notion that human intelligence is reducible to a number is an insult to its subject. In fact, all an IQ measures is the ability to take an IQ test. Psychologists recognize this, but many school teachers and the public at large don’t. (See Steven Jay Gould’s The Mismeasure of Man.)
  • Food calories. Calories are the most commonly used metric in nutrition. What this number actually represents is the heat generated by drying and burning a food item. But is digestion the same as combustion? Obviously not, for example, for fibers, which cross the human body unchanged. The absurdity of assigning calories to fibers has not escaped one dieter, who questioned it on a Calorie Count forum, and received, among other replies, the following:

Fiber calories are included in nutrition information, but only in come countries. In the US, it is legal to not put in fiber calories because they are not digestible. Therefore, they do not “count” as such. however, if you, like most people, tend to underestimate cals sightly, there is nothing wrong with including them to create a “buffer zone.”

In other words, it makes no sense but you should pretend it does.

Do we behave the same way in the manufacturing world? Yes. For example, many companies measure productivity in terms of Sales/Employee. There is an easy way to boost this metric: outsource all production, close all plants and become a trading company. It is not easy to find metrics for quality, cost, delivery, safety and morale that are meaningful and cannot be gamed, but it can be done. For overall company productivity, for example, you can use Value added/Employee, where

Value added = Sales – (Materials + Energy + Outsourced Services)

This is what Peter Drucker called Contributed Value. Value added/Employee is not a perfect metric, but at least it does not provide a perverse incentive to outsource, and the US census bureau publishes statistics on value added and employment by industry, that are helpful for benchmarking.

Following are a few conditions that a good metric must meet:

  1. A good metric is immediately understandable. No training or even explanation is required to figure out what it means, and the number directly maps to reality, free of any manipulation. One type of common manipulation is to assume that one particular ratio cannot possibly be over 85%, and redefine 85% for this ratio as “100% performance.” While this makes performance look better, it also makes the number misleading and difficult to interpret.
  2. People see how they can affect the outcome. With a good metric, it is also easy to understand what kind of actions can affect the value of the measurement. A shop floor metric, for example, should not a be function of the price of oil in the world market, because there is nothing the operators can do to affect it. Their actions, on the other hand, can affect the number of labor-hours required per unit, or the rework rate.
  3. A better value for the metric always means better business performance for the company. One of the most difficult characteristics to guarantee is that a better value of a metric always translates to better business performance for the company. Equipment efficiency measures are notorious for failing in this area, because maximizing them often leads to overproduction and WIP accumulation.
  4. The input data of the metric should be easy to collect. Lead time statistics, for example, require entry and exit timestamps by unit of production. The difference between these times then only gives you the lead time is calendar time, not in work time. The get lead times in work time, you then have to match the timestamps against the plant’s work calendar. Lead time information, however, can be inferred from WIP and WIP age data, which can be collected by direct observation of WIP on the shop floor. Metrics of
    WIP, therefore, contain essentially the same information but are easier to  calculate. (See Little’s Law.)
  5. All metrics should have the appropriate sensitivity. If daily fluctuations are not what is of interest, then they need to be filtered out. A common method for doing this is to plot 5-day moving averages instead of individual values — that is, the point plotted today is the average of the values observed in the last five days. Daily fluctuations are smoothed away, but weekly trends stand out.

Peter Drucker sold corporate America on the idea that you can’t manage what you can’t measure, and this has led many managers to believe that employees would do whatever it takes to maximize their scores. Given  flawed metrics, it if fortunate for the companies that these managers were wrong. If they had been right, all the  companies that measure productivity in terms of  Sales/Employee would actually have outsourced all production. They didn’t, because metrics are only one of many factors influencing behavior. Most employees, at all level, will not maximize their metrics through actions they feel violate common sense or are inconsistent with their personal ethics.

12 comments on “The staying power of bad metrics

  1. Contributed value is quite interesting. Have you used it in your business? What level of acceptance did it receive from the executive team? I’m inspired to track it “off the radar” for awhile to see what it tells me.

  2. I prefer to call it value added rather than contributed value because that is the term used by the US census bureau and by the tax collectors of countries that have a Value-Added Tax (VAT).

    I have found it useful as a quantification of what is called Value Added in much of the literature on Lean. There, an activity is said to be “value added” if the customer is willing to pay for it; it is an attribute that the activity has or doesn’t have. The US census bureau concept of value added is the amount customers are actually paying for the services of a company in addition to the external inputs — materials, energy and outsourced services — that they would buy themselves if they chose to bring the work in-house.

    If you take the ratio of value added to external inputs, you find that it varies between industries, and that the differences go a long way towards explaining organizational behavior. To make profits with a low value added, you have to turn over your inventory quickly, and this drives both production line designs and operating policies. A high value-added, on the other hand, hides many sins and is often frittered away in lax manufacturing practices that shock visitors from the more disciplined low value-added industries.

    Some industries also strategically seek ways to increase their value added by customizing their products or making them easier to use. It happens when an auto parts company, for example, sets up a finishing shop next to a customer plant or when a food company packages vegetable in microwave-ready plastic bags.

    At an industry level, value added per employee reflects the level of investment in equipment and automation. If is often interesting to compare this parameter for an individual company to the industry average. If it is five times lower, it raises questions about the ways the company uses people.

  3. Thanks Michel. Its easy to “see” in a physical manufacturing environment. Have you applied it in a transactional environment for example data transformation, where the value add is taking many types of data from many sources, scrubbing, transforming and standardizing it into a sellable product?

  4. Comment in the Schlumberger discussion group on LinkedIn:

    I would be curious to know whether “bad metrics” is intentional or not. There are 2 relevant quotes in this regard. The first one, commonly states that “numbers don’t lie”, or “that numbers talk”. The second one, if from an ex-French Minister, who said “les statistiques, sont comme un string … ca montre tout sauf l’essentiel”. Translated, this says “talking statistics, is like wearing a string … it shows everything, but the essential”. I, too, believe that bad metrics, is an intentional and conscious form of manipulation, and a weapon of “mass seduction”.

  5. Comment in the Schlumberger discussion group on LinkedIn:

    A method used in very competitive markets to give the impressions of success and obtaining a standard of performance goal. Narrowing the definitions for market to mirror what you do not what the industry is doing.

  6. Comment in the Schlumberger discussion group on LinkedIn:

    It is vital for an organization to know what the key metrics that they need to drive at the highest level that will improve the health of the business as opposed to multiple metrics that sub optimize performance overall.

  7. Comment in the Supply Chain Movement discussion group on LinkedIn:

    Have you ever heard…”Measure me and I’ll tell you how I’ll perform”. The key to leadership is to hold managers accountable to core measurements that truly lead the organization to longterm success and profitability. Equally as important is the have enough measurements to balance the positive and negative aspects of each; if you only measure customer service your inventory levels will sky rocket.

    Leadership can’t only measure, they must train and educate all levels of the organization on how key measurements are interwoven. As team members learn and understand measurement interactions, and are held accountable for results they will move the organization toward world class performance.

    Lastly, as you train and educate, the team members will challenge certain measurements. This is when a decision needs to be made. Is the measurement required, can it be eliminated, is the goal attainable or is the team member a “C” or “B” player and not capable of being stretched to peak performance.

  8. Pingback: Lean Lies | Wiegand’s Watch | Michel Baudin's Blog

  9. Pingback: Cars per employee and Productivity at Volkswagen versus Toyota | Michel Baudin's Blog

Leave a Reply

Your email address will not be published. Required fields are marked *