The fallacy of maturity assessments | Chris Hohmann

“Maturity assessments are a kind of qualitative audit during which the current ‘maturity’ of an organization is compared to a maturity reference model and ranked accordingly to its score.[…] The maturity assessment is usually quite simple: a questionnaire guides the assessment, each maturity level being characterized by a set of requirements. It is close to an audit.

The outcome of such an assessment is usually a graphic summary displaying the maturity profile or a radar chart, comments about the weak points / poor scores and maybe some recommendation for improvement. […]

Maturity assessments are not a bad thing per se, but their practicality and simplicity are often misused to assess more than just maturity (or awareness). This is most often misleading because of the false underlying assumptions and promoting wrong behaviors and practices.”

Sourced through Chris Hohmann’s blog

Michel Baudin‘s comments:

I agree with Chris’s analysis, but my conclusions are blunter. Scoring an organization in terms of compliance with a set of practices is like judging a chess player by the number of pawns moved per game. It’s doable but irrelevant, and a distraction from the real work of improvement. The record of this approach is that you have organizations scoring top marks on every axis while going bankrupt and low scoring organizations that prosper.

The reason this approach is popular, particularly with the supplier support groups of large companies, is that it doesn’t require the auditors to know any specifics of the business, management, and technology of the organization they are scoring. They can see whether all the visible signs of 5S are in place, each team has a performance board and each machine an andon light,… You can even check how many Kaizen events they have run in the past 12 months.

The suppliers learn how they must appear to humor the auditors and engineer their facilities to look right. To them, it’s a cost of doing business, not an opportunity to improve, which explains why they don’t perform any better than competitors. If instead of complying with external mandates, an organization is driven by its management to improve its practices at all times, in pursuit of broader goals than certification by a customer, it will not follow a generic script and perhaps undertake projects that are not as spectacular to auditors but are more useful. For example, instead of putting

If instead of complying with external mandates, an organization is driven by its management to improve its practices at all times in pursuit of broader goals than certification by a customer, it will not follow a generic script and perhaps will undertake projects that are not as spectacular to auditors but are more useful.

For example, instead of putting andon lights on every machine, they may pursue setup time reduction on one of the machines and later propagate the new method to all similar machines. Or they engineer retrofits to old machines that double their capacity or eliminate an entire category of defects.

Having mental or physical check lists when visiting a plant is useful. You want to notice the type of containers used on the side of an assembly line, idle machines, or empty tool pockets in machining centers,… to use as input to an analysis, not just to summarize as a simplistic score.

#MaturityAssessment, #RadarChart, #20Keys, #ShingoModel, #ExternalMandate, #ChristianHohmann

10 comments on “The fallacy of maturity assessments | Chris Hohmann

  1. Comment on LinkedIn:

    Worth reading . I agree with your comments Michel Baudin. Maturity assessments can be very useful if well applied and understood by stakeholders involved on this process, mainly management team.

  2. Comment on LinkedIn:

    Great article and content Michel Baudin as usual! Covering this, sometimes the very subjectiveness of conducting maturity assessments.

    I have seen some complicated 60 survey questionnaire, where the executive leadership rates themselves @ level 4+ out of 6 of which achieves basic conformance and compliance, then I have seen the front line teams at the coal face complete the same audits @ 1.5/6, then an independent certified auditor come on @ 0.5/6. It is similar to the ISO9000/14000 compliance audits, where you can see/site, no proactive improvement activities occur during the year at some organisations, then about 2-3 weeks prior to the annual ISO audit, the poor quality and production supervisors are redirected to generate all the statistical charts and paperwork to support the upcoming audit on the non compliances, after the audits they return to their normal work mode of putting out all the daily bushfires and knee jerk reoccurring type problem solving efforts etc!
    This is like rubber stamping your Operating /Standards/Quality systems!

    cheers Ralph.

  3. Comment on LinkedIn:

    I agree with Michel Baudin in the summary of limitations of most maturity assessments and herein lies the answer, the construct is often incomplete and the assessors not proficient in assessing the real capability maturity issues such as sustainability, improved market position and EBITDA / ROCE improvements over time.

  4. Comment on LinkedIn:

    Tnx Michel. Maturity assessments using questionnaires .. to subjective. Using operational PDCA / KPI based assessment the way to go – MAYBE!. Assessment must be conducted by process owners. Add to this designed threshold validation criteria (inclusive of external forensic review for index scores abobe 55), and integrity is assured. Currently with a client where operational results directly linked to snr and Exec performance assessment. 3 years down the line … total maturity change.

  5. Excellent post. I agree with both of you. And the situation gets worse when factories or business units are compared against one another – especially when the results are used in reviews/ratings. It’s always good to know where you are in terms of your vision, but a subjective assessment is not the way to do it. Enlightened leadership teams know where their problems are and, if they possess even a basic kaizen mindset, can get more out of working on closing the gaps than wasting time conducting and arguing about a maturity assessment.

  6. It is hard to disagree with you.

    However, how should a big company have an idea of how good its suppliers are? Imagine a supermarket chain, for example, with zero understanding of what drives good performance in production.

    • Please feel free to disagree.

      You don’t need to understand what drives good performance for your suppliers in order to monitor them in terms of delivery, quality, and price but you can’t help them improve unless you do understand it.

      A common mistake for companies is to staff supplier support groups with rookies and give them audit checklists as crutches. To suppliers, it is arrogant and disrespectful.

      Supplier support groups should be comprised of seasoned professionals who have been successful in internal projects and are intellectually nimble enough to understand the differences between their company’s business and that of each supplier.

      They must be willing and able to dive into each supplier’s specific issues deep enough to hold meaningful discussions with its leaders. They must come across as respectful and credible.

      And then, not only must you do without their help on internal projects, but you have the challenge of retaining them.

      • That makes sense and matches my observations!
        To most suppliers, “assessment” means “audit”. And they feel they have to look good. They keep the important data to themselves. Don’t expect them to disclose and break down their costs of quality, for example.
        Outside the auto industry, I have seen very few organizations doing a good job at helping their suppliers improve. Ikea does a good job. Apple sends a lot of resources to set up processes & lines correctly. I am sure there are others but not many. And in each case the supplier’s top management is open to help/deep involvement from their customer.

      • Big companies, like Ikea, Apple, or Toyota can, unless they act with restraint, be perceived as bullies by their suppliers. That happens even if they don’t intend to, and they must be especially careful. But the roles are reversed when the customer is a small company buying from larger suppliers.

        Can a small customer provide supplier support? I know at least one case where it was successfully done.

  7. I’d say, it all depends how maturity assessments are used. Like with any other tool, if it is applicable and used with focus on an actual problem, then it can be powerful by creating new insights and initiating the right actions.

    I am using the Lean Factory Audit and the Lean Office Audit (https://www.leanmap.com/benchmarking/lean-audit/) to engage the management team into an (often intense) discussion, identifying the weak areas that contribute or even cause the problem the team is trying to solve.

    Because the outcome is a score, and performance driven people like scores, they stay engaged and interested. The process is then viewed as a sporty challenge to improve the score. New energy is created that can now be directed towards the problem. If properly steered, any gain in maturity will then correlate well with the performance gain that team is going after, e.g. increasing productivity to reduce the cost base.

    And I agree, a maturity assessment is not very helpful when just used in a compliance context (not much learning created), while it can be very effective when used as an anchor point and integral part of an improvement system – with focus on a well-defined business problem. Just my five cents.

    Joerg Muenzing

Leave a Reply

Your email address will not be published. Required fields are marked *