One of my least favorite kind of PM compatriots are those who insist that they are purely data driven. Like all zealots, they’re often tiresome bores, and while there is a lot of truth in what they advocate, their smugness and certainty is off-putting. But, of course, numeric measurement is useful! So here’s a few downsides of shallow quantitative methods, as a dose of skepticism for all you quant advocates.
I think the greatest downfall of data driven approaches is picking the wrong number to build a goal around. If you work with smart people, even the best intended will game a simple numerical target, when selected without care and consultation. In one of my favorite stories about this, a company decided to set sales quotas by weight of items shipped from the warehouse. After months of record setting sales, accounting realized the weights shipped didn’t match the actual revenue the invoices reflected. The sales people had paid off the warehouse guys to add bricks to the shipments.
Another pitfall is not actually understanding whether the KPI you are tracking really means what you think it does. If a number is a “proxy” or “simplification” of some more complex behavior, beware! Always understand what the numbers really represent, not what they have come to represent to the team, especially when working with a mature product. What events trigger them? Don’t assume something labeled “conversion” is actually when revenue is generated. Look past the numbers to the underlying activity.
The next step is to move beyond description to analysis. Always be looking for the story. Why a number is, not just what it is. What does it mean? Being data driven doesn’t mean putting graphs in your decks and having dashboards. It means using quantitate data to make decisions, and understand how people use your product. Explore your data. Don’t just refute or prove theories with it, discover hidden failure points and identify new opportunities. If you’re lucky enough to have a good analytics system, get immersed.
And woe, beware the data driven emergency! I have seen many many times an urgent alarm is raised based on a single chart or number. Something is down, something is up, this looks way off. I would estimate that 90% of these end in one of two ways: a bug in the data, or a misunderstanding/change of definition. Analytics are tricky systems, and events and calls can easily move or change implementations without everyone knowing. Before you freak out, eliminate these options.
Numerical data is not an inherently superior or more honest. It’s as easy to lie with numbers as stories. It’s as easy to have bad analysis as bad taste. Always be scientific with data when you need to know the truth: taking measurement without prejudice and being clear eyed in your analysis. But be lawyerly with data when you use it with others. Pick and choose the evidence that best supports you, not lying, but treating the data as secondary to the narrative. In the end, I believe, data is there to build and support the narrative, not t’other way round.