In the last blog posting on “The Characteristics of Metrics that Matter: Relevancy”, I summarized the characteristic of relevancy in determining metric effectiveness. A good metric must be relevant, i.e., it must align to one or more organizational objectives, limit conflict with other metrics, and relate to a well-understood impact.

To understand metric alignment and impact better, you should also make sure that the metric’s calculation is accurate and reliable. To me, accuracy and reliability are the basis for metric “trust” and should be examined to minimize misinterpretation of the metric and its values.

“Accuracy” here means that the metric’s calculation must be done “right” – the formula or calculation method must be correct for what you are trying to measure. Therefore, one should confirm the calculation method with one or more third-party sources that also use and understand the metric. You should also confirm with individuals in your organization that compile, review, and are affected by it.

In the real world, these individuals may not always agree on a “perfect” calculation definition so, document and discuss their arguments openly. Disagreements may or may not completely disallow any one metric, but not openly discussing them will undoubtedly decrease the metric’s motivational effectiveness. In fact, working through accuracy disagreements is one of the best ways of identifying the foibles in the formula.

Standardizing a metric’s parameters like sales, profits, customer, etc., also enhances its accuracy. If the parameter definitions are widely questioned, the metric will find little loyalty. Accuracy also includes understanding and accounting for how a metric formula could go wrong. For instance, inflation could cause a false jump in worker productivity if it is measured as total revenue/total number of employees. Absenteeism could be skewed if you accidentally include employees that are caring for sick family members or are out for other valid out-of-office reasons.

Spelling out exactly how to calculate a metric, how often, and from what data you compile it is extremely important in ensuring its accuracy.

Closely linked to metric accuracy is reliability. Reliability in this case is not how a metric is calculated but how it’s reliably interpreted once determined. Consistent interpretation begins with thoroughly understanding a metric’s “metadata” at any one time: honestly gathered input data, the accountable metric owner, time last updated, known and reliable systems of data origin, and other parameters.

Reliability is obviously hurt by “gaming” a metric, that is, circumventing honest and accurate interpretation to obtain a desired outcome for ego, laziness, or greed. Depending on the relationships between metrics, such corruption can negatively affect activities that can range from individual corporate processes like reliable employee appraisals to damaging the entire performance management system.

Simple questions organizations can ask to assess a metric’s reliability and accuracy are:

• Do users understand the calculation or formula and its interpretation? Note: The simplest formula may be the best starting place for building a metric, but it may miss the complexity needed to be truly useful
• Have we minimized unnecessary “human influences” in its interpretation?
• If a metric value changes, are we confident that a parameter that we thought would change actually did change?

Do you check for accuracy and reliability when crafting performance metrics? How? Drop me a note and let’s discuss.