When you hear anyone say “Studies show…”, you want to know who studied what and how, so that you can use what Kaiser Fung calls your number sense to decide what, if anything, can be learned.

Since 2008, “Next Generation Manufacturing Studies” have been conducted in the US by the following:

• NIST, the US government’s National Institute of Standards and Technology.
• The American Small Manufacturers Coalition, a group of consulting firms subsidized by NIST.
• The Manufacturing Performance Institute, a company that produces studies.

Results are available on line up to 2011, and the 2013 study is underway. It is intended to evaluate “awareness, best practices, and achievements” in the following six areas:

1. Customer-focused innnovation.
2. Engaged people/human-capital acquisition, development, and retention.
3. Superior process/improvement focus.
4. Supply-chain management and collaboration.
5. Sustainability.
6. Global engagement.

By what methods are these studies conducted? The following is p.27 from the 2011 National Executive Summary:

This study is therefore entirely based on questionnaires filled out by a small, self-selected sample of companies rating themselves on a scale of 1 to 5 on issues like “the importance of customer-focused innovation.” It involves no site visit or personal  interviews. The responses were only “cleansed to ensure answers were plausible,” a statement that leaves much to the imagination.

This raises the following questions:

• What can we learn from such a study?
• Is this the best we can do with 21st-century data mining technology?