Counting is Hard#
On day one of a statistics class we are taught that counting is hard. Depending on what question is asked, the numbers will vary.
Reading the following will take a few minutes. Not reading it will cause hours of misunderstanding in quarterly meetings.
Three kinds of report#
Time domain reports show results from a time period. You can see results by progress category. You cannot see yield, or what came from what.
Cohort reports show how a group of people ended up in business results. You cannot see the overall business results for a time period.
Operational reports show what is happening now. Similar to the dashboard in a car, where speed and fuel level can change every time you look.
An example from marketing#
How many leads did we generate this quarter? 4,000. That is a time domain answer.
How many new opportunities did we create? 200. Also a time domain answer.
So we got 200 new deals from 4,000 new leads? No. That is a yield question, and the 200 are not “from” the 4,000. Some derived from the 4,000. Some came from prior quarters.
Five questions#
In general, business metrics are asked to answer five things:
- How did we do last quarter? (Results)
- How many Y did we get from X? (Yield)
- What is our current state right now? (Operations)
- How is something trending? (Usually results over time)
- How does a particular person get compensated? (Complex, and tricky to count)
Answering them#
“What happened?” — usually a time domain report. Show the results of what occurred during a quarter, month, or year. The results should not change if you keep asking the question about the same time period.
“What happened to the new leads from that trade show?” — usually a cohort report. Show what happened to all the members of a group over time. The results could change every time you ask, as more members progress. There is an asymptote after about two years, at which point things stop changing.
“How effective is a particular program?” — a variant on the cohort report.
“What is happening this month, so I can manage it?” — an operational report. A snapshot of right now, ignoring how we got here or where it came from. The results can change within a meeting simply by clicking refresh.
Operational reports make terrible results reports and terrible yield reports, and we cannot live without them. They are the fuel gauge. You would not use one to work out your average consumption over the year, and you would not drive without one.
What to do about it#
The three kinds are often not properly labelled, which is where the confusion starts. The numbers between them will not line up. This is not an error.
In mixed management meetings, focus on time domain and operational reports. Cohort reports are for people digging into the nuances of yield optimisation — and a cohort number that has moved since the last meeting will be read as a mistake by anyone not expecting it to move.
Then there is the wild card: how do various groups get measured? Commission and MBO reports are frequently combinations of all three, with many corner cases and adjustments. Treat them as their own problem.
A formal treatment of the above — the Lexis surface underneath the three report types, and the conditions under which each is valid — is in the technical note counting.