Benchmarking

Jul 30 2014

“Studies show…” or do they?

Various organization put out studies that, for example, purport to “identify performances and practices in place among U.S. manufacturers.”  The reports contain tables and charts, with narratives about “significant gaps” — without stating any level of significance — or “exponential growth” — as if there were no other kind. They borrow the vocabulary of statistics or data science, but don’t actually use the science; they just use the words to support sweeping statements about what manufacturers should do for the future.

At the bottom of the reports, there usually is a paragraph about the study methodology, explaining that the data was collected as answers to questionnaires mailed to manufacturers and made available on line, with the incentive for recipients to participate  being a free copy of the report. The participants are asked, for example, to rate “the importance of process improvement to their organization’s success over the next five years” on a scale of 1 to 5.

The results are a compilation of subjective answers from a self-selected sample. In marketing, this kind of surveys makes sense. You throw out a questionnaire about a product or a service. The sheer proportion of respondents gives you information about the level of interest in what you are offering, and the responses may further tell you about popular features and shortcomings.

But it is not an effective approach to gauge the state of an industry. For this purpose, you need objective data, either on all companies involved or on a representative sample that you select. Government bodies like the Census Bureau or the Bureau of Labor Statistics collect useful global statistics like value-added per employee or the ratio indirect to direct labor by industry, but they are just a starting point.

Going beyond is so difficult that I don’t know of any successful case. Any serious assessment of a company or factory requires visiting it, interviewing its leaders in person, and reviewing its data. It takes time, money, know-how, and a willing target. It means that the sample has to be small, but there is a clash between the objective of having a representative sample and the constraint of having a sample of the willing.

For these reasons, benchmarking is a more realistic approach, and I know of at least two successful benchmarking studies in manufacturing, both of which, I believe, were funded by the Sloan Foundation:

  • The first was the International Assembly Plant Study, conducted in the late 1980s about the car industry, whose findings were summarized in The Machine That Changed The World in 1990. The goal was not to identify the distribution of manufacturing practices worldwide but to compare the approaches followed in specific plants of specific companies, for the purpose of learning. Among other things, the use of the term “Lean” came out of this study.
  • The second is the Competitive Semiconductor Manufacturing Program, which started in the early 1990s with a benchmarking study of wafer fabrication facilities worldwide. It did not have the public impact of the car assembly plant study, but it did provide valuable information to industry participants.

The car study was conducted out of MIT; the semiconductor study, out of UC Berkeley. Leadership from prestigious academic organizations helped in convincing companies to participate and provided students to collect and analyze the data. Consulting firms might have had better expertise, but could not have been perceived as neutral with respect to the approaches used by the different participants.

The bottom line is that studies based on subjective answers from a self-selected sample are not worth the disk space you can download them onto.

By • Management • 4 • Tags: , , , , ,

Lean implementation survey

Nov 9 2011

More Flak on Lean Based on the Same Survey

Managing Automation published another response to the same study that claims to show that Lean does not work: Lean Manufacturing and Operation Excellence: Not Worth Their Weight? 

As described in the press and their own press release, the AlixPartners study commingles Lean with “Six Sigma and other productivity programs,” which raises the following issues:

  1. Lean Manufacturing is based on the Toyota Production System, which includes neither Six Sigma nor Operation Excellence nor  “other productivity programs,” whatever those may be.
  2. Lean Manufacturing is not a “productivity program,” but the pursuit of concurrent improvement in quality, productivity, delivery, safety, and morale. I  know I am repeating myself, but it needs to be said until the leaders of manufacturing companies hear it.

If these press accounts are correct, the survey confuses Lean with other approaches in an open-ended list, misstates its purpose, and considers exclusively metrics of cost reduction.

The effectiveness of Lean is not an easy subject to study. Should we survey all the companies that claim to be Lean, have a Lean program in place, have been certified Lean by some external authority, or are top performers in their industry? Once we agree on this, we still need yardsticks to quantify both  the effort they put into Lean and the rewards from it.

I took a stab at it a few years ago, and did my own analysis, the results of which were published as a Viewpoint in Manufacturing Engineering in 2006. I chose 40 winners of the Shingo Prize and searched Hoovers Online, for comparative performance data with their 400 top competitors. On the average, the data did not show that the Shingo Prize predicted any advantage in profitability, market share or employment growth. The AlixPartner press release says roughly the same thing, but I see it as reflecting on the Shingo Prize itself, not Lean.

The Shingo Prize is supposed to be the “Nobel Prize for Manufacturing,” but what are the criteria used to award it? You can download the Shingo Prize Guidelines and see for yourself. A team of Shingo Prize auditors visits the plants and awards points  to measure “the degree to which the behaviors in an organization are aligned with the principles of operational excellence.” In other words, the plants are measured on process compliance. They score points for practices they have in place. It is like measuring chess players on the number of pawns they move, and is correlated to victory like the Shingo Prize to business performance.

Toyota did not grow based on a compliance checklist. When I visit a plant, based on what I see and what people tell me, I can form an opinion as to whether they are among the few that have the spirit of Lean or the many that are going through the motions. But I don’t know how to generate a checklist that could be systematically applied to arrive at such a conclusion, and, desirable though it may be, I don’t believe a real survey is feasible.

Jamie Flinchbaugh doesn’t like sports metaphors, but I can’t resist one here. Usain Bolt  is the fastest man alive. Let us assume somebody publishes a book entitled “The Running Secrets of Usain Bolt.” How Usain Bolt actually trains is probably not trivial and certainly involves sustained effort and ferocious discipline. The author of the book, however, is concerned that a stern, eat-your-vegetables message would hurt sales, and focuses instead on easier topics, like shoes. As a result, kids flock to shoe stores thinking that wearing these shoes will make them fast, but the real ones are too expensive, so they buy cheap imitations instead. Six months later, based on their responses, a survey concludes that Usain Bolt’s methods don’t work.

Most Lean programs today are to serious implementations as cheap imitation shoes are to the training of Usain Bolt. Where they may succeed is in ruining the reputation of Lean. It is bound to happen sooner or later. As a brand, Lean has had a 22-year run so far, already longer than I expected.

By • Management, Press clippings • 10 • Tags: , , ,