Metrics in Lean – Part 4 – Gaming and how to prevent it

As massively practiced today, Management-by-Objectives (MBO) boils down to management imposing  numerical targets on a few half-baked metrics, cascading this approach down the organization and giving individuals a strong incentive to spin their numbers. It is a caricature of the process Peter Drucker recommended almost 60 years ago, and he deserves no more of the blame for it than Toyota does for what passes as Lean in most companies that claim to implement it.

A non-manufacturing example of decadent MBO is the French police under former president Sarkozy, which was tasked by the government to decrease the crime rate by 3%/year while increasing the proportion of solved cases. According to the French press, this was achieved by gaming the numbers. The journalists first latched on to a reported yearly decrease in identity theft, which seemed unlikely. They found that police stations routinely refused to register complaints about identity theft on the grounds that the victims were the banks and not the individuals whose identities were stolen. A retired officer also explained how crimes were systematically downgraded with, for example, an attempted break-in recorded as the less severe “vandalism.”

The fastest way the police had found to boost the rate of case solutions was to focus on violations detected through their own actions, such as undocumented aliens found through identity checks. The solution rate for such crimes is 100%, because they are simultaneously discovered and solved. The challenge is to generate just enough of such cases to boost the solution rate without increasing the overall crime rate… To achieve this result, packs of police officers stalked train stations in search of offenders, as reported both by cops who felt this was not what they had joined up to do, and innocent citizens who complained about being harassed for their ethnicity.

In organizations affected by this kind of gaming, members work to make numbers look good rather than fulfill their missions. It is a widely held belief that you get what you measure and that people will always work to improve their performance metrics, but this is not a simplistic view of human nature. This behavior does not come naturally. On their own, schoolteachers focus on educating children, not boosting test scores, and production operators on making parts they can take pride in. It takes heavy-handed management to turn conscientious professionals into metrics-obsessed gamers, in the form, for example, of daily meetings focused entirely on the numbers, backed up by matching human resource policies on retention, promotion, raises and bonuses.

But enough about police work. Let us return to Manufacturing, and list a few of the most common ways of gaming metrics in our environment:

1. Taking advantage of bad metrics. As discussed in The Staying Power of Bad Metrics, many metrics commonly used in manufacturing are poorly defined, providing gaming opportunities, such as outsourcing in order to increase sales per employee.
2. Stealing from the future. In sports, nothing is more dramatic than the game won by points scored in the last seconds of a game. The bell rings right after the ball spirals into the basket and the Cinderella team wins the championship. In business, the end of an accounting period is the end of a game, and, as it approaches, sales scrambles to close last-minute deals and manufacturing to ship a few more orders. This is what Eli Goldratt called the “hockey stick effect.” Of course, this is done by moving up activities that would otherwise have taken place a few days later, during the beginning of the next accounting period. As a consequence, the beginning of the period is almost quiescent. Not much is going on, but it will be made up at the end…
3. Redefining 100%. Many ratios, by definition, top out at 100%. A machine cannot run 25 hours/day, and a manufacturing process cannot produce more good parts than the total it makes. This is why ratios like equipment uptime and first-pass yield top out at 100%. Any result under 100%, however, invites questions on how it could be improved. A common way to fob off the questioners is to decree, for example, that a particular machine could not possibly be up more than 85% of the time, and redefine the scale so that 85% uptime is 100% performance. For production rates in manual operations, the ratio of an operator’s output to a work standard is often used instead of process times or piece rates. Such ratios have the advantage of being comparable across operations, and are not bounded in either direction. But their relevance depends on a work standard, and, when everybody in a shop performs at 140% of standard, chances are that the standards are engineered for this purpose.
4. Leveraging ambiguity. Terms like availability, cycle time, or value added are used with different meanings in different organizations, creating many opportunities to game the metrics. If the product’s market share in the first quarter went for 1% to 2%, it doubled, but, if it went back to 1% in the second quarter, it went down by 1%.
5. …

Why do people who, in other parts of their lives, may be model citizens, engage in such behaviors, ranging from spinning to cheating? One answer is that, with what MBO has degenerated into in many companies, management is co-opting metrics gamers into its ranks. It is not that gaming is human nature, but instead that you are actively weeding out those who don’t engage in it. Changing such habits in an organization is obviously not easy.

Assume, for example, that your goal is to be competitive by having a skilled work force, and that your analysis shows that it requires employees to stay for entire careers so that what they learn at the company stays in the company. You then apply a number of different methods to make it happen:

• Communications. You make sure that all employees know what you are doing.
• Career planning. You have human resources develop a plan with all employees so that each one knows what he or she can aspire to by staying with the company.
• Organized professional development. You organize formal training, on-the-job training, and continuous improvement to provide opportunities for employees to develop the skills they need to execute their plans.
• Job enrichment. You redesign the jobs themselves to make more effective use of each employee’s talents.

If employees appreciate their jobs and have long-term career perspectives within the company, few of them should quit or make excuses not to come to work today, and the results should be visible in lower employee turnover rates and absenteeism.

The metrics are there to validate the approaches taken to reach the goal, but the goal is not to improve the metrics. It is a subtle difference. If you have the flu, you have a fever, but your goal is to heal, not just to bring down the fever. Once you are healed, you fever will be gone, and the decrease in your temperature is therefore a relevant indicator of your healing process, but it is not the healing process. If bringing down the fever were the goal, you could “game” your temperature and bring it down without healing. This distinction existed in Drucker’s original writings about MBO, but got lost in implementation.

So, what can you do to prevent metrics gaming? Let us examine three strategies:

1. Review the metrics themselves. Use the requirements listed in my first post on this subject. You may not be able to completely game-proof your metrics, but at least you could make sure that they make sense for your business and are not trivially easy to game.
2. Decouple the metrics from immediate rewards. Piece rates used to be the most common form of payment for production work, but have almost entirely vanished in advanced manufacturing economies, and been replaced by hourly wages. Performance expectations are attached, but there is no direct link to the amount produced in a given hour of a given day. There are many reasons for this evolution:
   • The pace of work is often set by machines or by a moving line, rather than by the individual.
   • The best performance for the plant is not necessarily achieved by every individual maximizing output at all times.
   • More is expected of all individuals than just putting out work pieces, including training or participating in improvement activities.

   One consequence of this decoupling is that time studies are easier and more accurate than in a piece rate environment. The same logic applies in management: the more direct the link between metrics and individual evaluations, the more intense the gaming. Don’t make the metrics the key to promotions or to prizes representing a substantial part of a manager’s compensation. Use them only as indicators to inform discussions on plans and strategies.

3. Increase the measurement frequency. The longer the reporting period, the more opportunities it offers for gaming the metrics by stealing from the future, and the more pronounced the hockey stick effect. Conversely, you can reduce it by measuring more often, and eliminate it by monitoring continuously, as is done, for example by the electronic production monitors that keep a running tally of planned versus actual production in a line during a shift. Periods exist in accounting because of the limitations of data processing technology at the time the accounting methods were developed. In the days of J.P. Morgan, closing the books was a major effort that a company could only be do every so often. In 2012, there is no technical impediment to the “anytime close,” but the publication of periodic reports continues by force of habit. Metrics in the language of things as well as the language of money can be monitored continuously.
4. Have third parties calculate the metrics. In principle, counting chips should be done more accurately by agents with no stake in where they may fall. In practice, it is not only expensive but does not always produce the desired result. It is the approach used in Management Accounting. A plant’s accounting manager, or comptroller, is not chosen by the plant manager, he or she reports directly to corporate finance, and has no motivation to humor the plant manager. This is a double-edged sword because, with neutrality, comes a distance from the object of the measurement that may cause misunderstandings, and Management Accounting leaders like Robert Kaplan, Orrie Fiume, or Brian Maskell  have been struggling with the challenge of providing relevant, actionable information to managers for the past 30 years. Outside of Accounting, for metrics in the language of things, the closest you can come to having a 3rd party produce the measurements is to have a computer system do it, based on automatic data acquisition. There is no opportunity for gaming, but the issues of relevance are as acute as in Management Accounting.