Metrics and Key Performance Indicators (KPIs)

Metrics and KPIs indicate how an operation or system is performing relative to trend or a defined goal. They may take the form of point values (ninety-five percent uptime for a manufacturing process), ranges (plus or minus one percent of target), or probabilities (less than fifteen minute wait time, eighty-five percent of the time). They can be measured over time periods of any length.

Many types of values can drive decisions, but what makes something a metric or KPI is how the decision relates to strategic organizational goals. For example, a data point associated with a document or a record may affect whether that item is diverted to one processing operation or another. That data point drives a decision (and potentially thousands of them a day), but that decision does not indicate progress toward a goal or standard. Providing service in an effective and timely manner, reducing waster, hitting quality or financial targets, and meeting or exceeding market standards are all examples of strategic assessments and can be thought of as governing metrics.

Note that values that serve as metrics and KPIs are likely to be derived from the operation of a running process or system, as shown above (at least in part), and not values that describe or govern it, as shown below. Note also that the above figure, at least in this context, refers to a working solution, while the figure below describes the engagement during which it was created or modified.

Indicators may take many forms, but should have many of the following important characteristics (item names from the BABOK, think about how this advice is similar to that for writing user stories):

  • Clear: understandable and without ambiguity
  • Relevant: germane to the activity
  • Economical: obtainable without excess time and effort
  • Adequate: embodies enough information to provide effective guidance
  • Quantifiable: can be expressed in values that can be validated
  • Trustworthy and Credible: deriving from agreed-upon, accurate, authoritative sources

It’s nice when metrics are expressed in terms of simple values, but they can become very complex and synthetic. I worked on one program that tried to improve the “readiness” of large quantities of equipment. The equipment needed a high degree of maintenance and inspection, and was considered “not ready” whenever such work was waiting to be performed or even in progress. As a result, the value of the metric (expressed as a percentage), turned out to be surprisingly low. However, small changes to that value were regarded as highly significant. The important thing is to understand the source, derivation, and context of the metric or KPI, no matter what form it takes. (I could tell you some other stories about that and other programs, but those are discussions for another day. Ask me over a coffee sometime.)

Another issue with KPIs is that they may not capture what is really wanted. For example, I recently encountered a situation where a manager established a goal of having an eighty percent “win rate” for sales encounters. Someone else asked the question of whether the metric would have the desired effect. Going back into history, industrial planners in the former Soviet Union used to set quotas for the production of nails. The problem was that the KPI was expressed in tons of nails, and this was done without consideration of how difficult different kinds of nails were to make. As a result, producers tended to manufacture nails that were easier to produce. Since thicker and wide-headed framing nails are easier to produce than finer and small-headed roofing nails, the shortages of roofing nails were so severe that buildings would be constructed all over the place — except for the roofs. This rendered the buildings unusable and they would just sit and rot in the weather.

The Soviet nail example is meant to illustrate the advantages of the price system, where the (presumably higher) price of roofing nails incentivizes producers to manufacture what is needed.

I don’t know if we always have to go as far as saying Goodhart’s Law is always applicable, but one has to be mindful of what is being measured and what it means.

Returning the “win rate” example I started with, is the real goal to ensure that sales teams get wins at a certain rate, or is it to make the most sales? Or is it to make the most sales per a given amount of effort? Or is it to bring in the most money? One possible problem with targeting the win rate alone is that tales teams may game who they try to sell to, or they may do inappropriate things (like reduce the price to buy sales, thus lowering the brand’s value). None of those are desirable.

Metrics and KPIs should be revisited and reviewed over time to ensure they represent what is wanted, and that they incentivize the desired actions. I often mention the need to be thorough, and to look at problems from multiple points of view. As always, different people with different perspectives should be consulted where it makes sense. I try to illustrate this by including the image below in many of the talks I give.

This entry was posted in Tools and methods and tagged , , . Bookmark the permalink.

Leave a Reply