The Fallacy of Numbers

In 1988 I was sitting in calculus class when I learned something that has stayed with me ever since. I’ll paraphrase Mr. Massey without his permission.

“It is a pity,” my teacher began, “that we use numbers for so much but few people understand what they mean.” He proceeded to draw out on the chalkboard how our grades for his class would be calculated. He drew boxes representing four exams and a final box to represent the average of those exams.

“We like to use numbers to communicate. Numbers have power. If you put a number on something, we like to think we have a better understanding of what’s happening. The entire field of statistics is based on this premise, but consider the following.”

He drew in the grades of two students. Both students received grades of A, B, C, and D. One received them in ascending order while the other received them in descending order.

“Which one has a better understanding of the subject matter?”

Obviously, the student that started slow and finished the semester with an A had a better understanding of calculus. His point wasn’t lost on any of us. Both would end up with the same average grade. He then went on to explain how we would be graded on a different scale where later exams would have heavier weight. He reserved the right to nudge our grades upward if he believed we deserved it.

I was thinking of this a few years ago when the University of Florida’s Human Resources department redesigned their method for performing annual performance appraisals. Each year managers across UF are expected to file performance appraisals for their staff members. There would now be a greater emphasis on not only expecting those appraisals to be done but there were now more criteria on which a staff member could be evaluated. Each criterion would be based on a number of 1-5 and then summed to a final score of 5-25.

We were guided by numerous presentations and exhortations to treat performance evaluations as a chance to discuss performance with our staff. The assignment of numbers was less important than the chance to truly communicate with our subordinates.

I carried out the evaluation of my staff and moved on with my work. It wasn’t very shocking that six months later I was asked to defend my request for a raise for one of my subordinates. Apparently, his performance numbers were lower than a number of other candidates. As an explanation, I offered up that I am a critical supervisor and expect a great deal from my staff.

Comparison of performance numbers assigned by different managers is problematic at best. Some supervisors are best friends with their staff. Some managers have difficulty criticizing staff and some subordinates do not respond well to any criticism. Communication is key, but reducing that interaction to a number implies an impartiality that simply does not exist.

Both examples serve to illustrate the dangers of relying upon numbers to reflect and evaluation of performance. Unfortunately, in any significantly large institution there comes a point where we must rely upon numbers to record performance. People need to remember what assumptions lie behind all of those numbers. When we forget those assumptions, we put ourselves in a position where we are relying on “fuzzy Washington math.”

Organizations are constantly looking for metrics in order to evaluate their own performance. Managers look for appropriate metrics in order to justify budgets, request new staff, defend their policies, and for any number of other reasons. Managers need information in order to make decisions and there need to be metrics in order to inform those decisions.

When organizations spend so much time concerned with litigation, metrics can be used to not only inform but defend decision making. Efforts to remove someone from an organization are usually greeted with calls for documentation. Has the employee been counseled? Have they been informed of their sub-par performance? Is there record of their sub-par performance? And, the power of numbers here cannot be understated. A steady record of low appraisal numbers recorded over time has as much impact as a record high performance numbers.

But, as time has passed, the metrics used for evaluation have evolved. More and more, recorded opinions of management are giving way to more objective criteria: how many tickets resolved, average ticket lifespan, customer survey based information. I believe this trend has serious power and great potential to help management make informed decisions about the organizations they run.

As long as we understand the assumptions behind those numbers, all is good. Without true understanding though, numbers can lead us more astray than forward.