Analysis – PM'ing

The Limiting Factor

In high school chemistry, my excellent instructor, Mr. Gleich, drilled into us the concept of the limiting factor. It was so counter intuitive. Why wouldn’t speeding up any part of the reaction not speed up the overall reaction? It made no sense to me and many others, but he drilled it in. He used analogies, exercises and demonstrations… eventually, even though it’s counter-intuitve, it was clear: only the limiting factor matters.

As a newbie PM, I instantly recognized the same math applied in many situations I faced. I had a product or milestone, a chain of reactions turning raw materials into a final result, over time. I wanted to know how long the reaction would take, and where to apply the catalyst. And I still think this is the key to optimizing any process. Reducing pain, playing politics, and playing favorites has their place in deciding where to put your energy, but the only thing that will really make a process faster is ruthless focus on the limiting factor, enabled by a clear understanding of the rates at each step. You need to understand what the slowest, least efficient step is, but also keep an eye that optimizations to that step are not offset elsewhere in the process. Although a full model is helpful, action anywhere outside the limiting factor is not. Know the whole system, but act on one factor at a time.

To give a concrete example, say you have an enterprise product that has a per-client implementation process. Say you want to increase the number of customers you can onboard per month. If your can make the install process faster and the IT training smoother, but the real blocker is legal sign-offs on the contracts, your revenue will not really budge. If legal can only approve 4 deals a month, the number of trainings you can do is irrelevant to your goal. Of coarse, after you get legal to pick up the pace, you might find those trainings are limiting you after all, but that is what iteration is for.

That’s a simplistic example, but this applies to things you might not normally think of as “processes”, but can be modeled and attacked this way. For example, revenue is actually the residue of a reaction. Your product/marketing/sales meeting the market atmosphere. If there is a slow reaction, a low revenue per unit of time is the result. If revenue is slow, or inert, limiting factor analysis can identify what part of your hoped for, theoretical, market reaction is not happening. Marketing message not landing? Missing a key feature? Pricing wrong? Lay out your factors and their rates. Then act ruthlessly to improve the limiting one.

Unknown Unknowns

I fear the day that the MBAs and people that would have become “financial analysts” in the late millennium fully take over the culture of software product management. These are fine people, but they are abstract risk managers and value optimizers. They could be structuring financial instruments, selling jars of pasta sauce, or running software systems with almost identical approaches.

That said, my favorite framework from the finance industry that serves me every day as a PM are the concepts related to risk. Debt, insurance, investment and even team management are often about the accurate evaluation and management of risk.

The future is unknowable and we’re always dealing in probabilities and manipulating the likeliness of outcomes, not guaranteeing them. How do you handle this as the person most responsible for delivering success?

For any major undertaking, it’s important to think through and spend time on likely threats to the project’s success. But that only helps with the known unknowns, what about the unknown unknowns (to quote the wisdom of war criminal Donald Rumsfeld)?

Even when you don’t know the exact cause, it is still possible to manage vague, generic risk. Unknown threats generally come from a limited set of source types. The most common disruptions, in my experience, are the hidden stakeholder and the implicit requirement, both usually discovered late in the development process.

The hidden stakeholder means just that: it becomes clear that there is a function or person who might kill, block, or otherwise limit your success. The implicit requirement, similarly, is a story, feature or capability that limits or threatens your success, which you failed to identify during research and planning.

Think about what you would do to prepare. You’re gonna get hit by something: keep your head up and at least see it coming. Should one of these manifest, triage dispassionately: is it in your control? is it really a blocker or problem? Some things, like personnel losses and context changes should be accepted and accommodated. Some, like exec meddling, team stalemates or nice-to-haves becoming blockers, can be worth fighting to overcome. Some things, like technical risk, dance in the grey area between the two.

Of course, the real solution I always recommend agile development processes, and indeed agile thinking in general. The answer to not knowing the future is just to embrace that things will change, you will learn more and change is as much about opportunity as risk.

Data Games

One of my least favorite kind of PM compatriots are those who insist that they are purely data driven. Like all zealots, they’re often tiresome bores, and while there is a lot of truth in what they advocate, their smugness and certainty is off-putting. But, of course, numeric measurement is useful! So here’s a few downsides of shallow quantitative methods, as a dose of skepticism for all you quant advocates.

I think the greatest downfall of data driven approaches is picking the wrong number to build a goal around. If you work with smart people, even the best intended will game a simple numerical target, when selected without care and consultation. In one of my favorite stories about this, a company decided to set sales quotas by weight of items shipped from the warehouse. After months of record setting sales, accounting realized the weights shipped didn’t match the actual revenue the invoices reflected. The sales people had paid off the warehouse guys to add bricks to the shipments.

Another pitfall is not actually understanding whether the KPI you are tracking really means what you think it does. If a number is a “proxy” or “simplification” of some more complex behavior, beware! Always understand what the numbers really represent, not what they have come to represent to the team, especially when working with a mature product. What events trigger them? Don’t assume something labeled “conversion” is actually when revenue is generated. Look past the numbers to the underlying activity.

The next step is to move beyond description to analysis. Always be looking for the story. Why a number is, not just what it is. What does it mean? Being data driven doesn’t mean putting graphs in your decks and having dashboards. It means using quantitate data to make decisions, and understand how people use your product. Explore your data. Don’t just refute or prove theories with it, discover hidden failure points and identify new opportunities. If you’re lucky enough to have a good analytics system, get immersed.

And woe, beware the data driven emergency! I have seen many many times an urgent alarm is raised based on a single chart or number. Something is down, something is up, this looks way off. I would estimate that 90% of these end in one of two ways: a bug in the data, or a misunderstanding/change of definition. Analytics are tricky systems, and events and calls can easily move or change implementations without everyone knowing. Before you freak out, eliminate these options.

Numerical data is not an inherently superior or more honest. It’s as easy to lie with numbers as stories. It’s as easy to have bad analysis as bad taste. Always be scientific with data when you need to know the truth: taking measurement without prejudice and being clear eyed in your analysis. But be lawyerly with data when you use it with others. Pick and choose the evidence that best supports you, not lying, but treating the data as secondary to the narrative. In the end, I believe, data is there to build and support the narrative, not t’other way round.

do not doing

One of the biggest mistakes I see PMs, and people in general, make is their bias towards action. I don’t come to my job with the protestant work ethic and preference for action in every possible situation. Instead, I find much of my thinking about labor/effort better reflected in the Dao De Jing. Among it’s messages is to act as naturally as possible, to spend effort on identifying the flow and going with it, rather than activity for its own sake. This is sometimes boiled dow to the translation “do not doing”. In product management, I apply this to mean “sometimes it’s okay to CHOOSE not to do something as the best strategy”.

Resist the temptation to view work and activity as a valuable end, in-and-of itself, or as the solution to all problems.

This kind of “inactivity by choice” can be the right strategy in many situations, here are three examples:

Sometimes don’t build, repackage

Often, in software companies the temptation is to keep building new things. There’s a genuine need to deliver new features and updates every cycle for sales and marketing. And we’re paying a lot of engineers, we better keep them busy with interesting work! But I have had some of my biggest successes not with new things, but with old things applied and packaged differently. Often, the desire to do something new is because the existing product is not gaining the market or stakeholder acceptance expected. But if you do your research and listen to what users are saying, you may already have the solution with very little technical work. Instead of 6 months developing a new product, a change in interface emphasis and new marketing story may sell just as well.

don’t overreact to Analytics Noise

I worked in a very data driven organization with a hard driving (but quite effective) chief executive. The PMs and C-levels lived in fear of the email with a screenshot or link to a dashboard: “this number was down, do we know why?”. Many of these were valid queries, and they occasionally uncovered genuine issues. But more often than not, it was an unexplainable short blip, analytics bug, or most often, just a misunderstanding somewhere. And this chief executive was a genius (really). Most of us lesser mortals have an even lower “spot-a-real-problem” hit rate. That’s not to say you shouldn’t care or monitor your numbers (quite the opposite) but also learn how much noise and error exists in your context, and learn to triage these numerical mysteries as you might with bugs. Develop a methodology and intuition to know when to take action, and when to do nothing.

sometimes pass on an opportunity

In dealing with sales, or functions driven by month/quarter bound commissions or bonuses, there is a constant pressure to deliver something for a client or timeframe that does not account for your team’s reality. As with a marathon, just because you can run a 6 minute mile, doesn’t mean you should, if you want to finish the rest of the race. Asking the team to work nights and weekends better pay off, at least more often than not. Rearranging your roadmap for a marquee client can work, but it can also become a golden goose chase, giving the team whiplash and undermining any strategic thinking or product theories. Too many start-ups have been destroyed by taking on too much business or chasing a series of mega-opportunities before they were ready. Sometimes the best answer to inbound business is not “YES!”, but “not yet, but let’s talk”. Don’t go bankrupt “screening” your customers, or rigidly maintaining your scope in the face of product failure, but, also, be willing to be the leader at the table that says, “no, let’s not put in this in our sales proposal”. (Of course, a compelling “and here’s why…” should follow).

Keeping sales and marketing in their place

This is one of a series of essays about how to relate to common stakeholders. Here, I’ll provide some tips for working successfully with revenue people, like marketing and sales.

I have never worked with marketing or sales people that didn’t want some or many features and products I did’t have or ever plan to have. Many of these requests came with promises and guarantees of success, revenue, friendship and more (though I was never offered a bribe or kickback, sadly). It doesn’t mean the product is deficient, or you need to chase the supposed opportunities. Triage them reasonably, and individually, but just know that the revenue people always want more.
Similarly, these functions always want or expect things on impossible time frames. Especially in large or silo’d organizations, where these people often have little concept of the work needed to deliver on their promises. Again, be cynical but optimistic. Gauge the risk, and if you can’t meet their timelines communicate that as soon as you’re sure of it. Get the pain out of the way.
A corollary of the above is that these are the stakeholders (besides execs) who always want hard deadlines, and are therefore usually the most adamant about thinking about things in waterfall type processes and phased, scheduled, commitments. Do what you need to accommodate them, but never let their need for “dates dates dates” become a thing in-and-of-itself. Deadlines are for customers, press, partners or revenue, not because sales thinks you should have them.
Sales people are like customers. Listen to them, yes, but pay attention to what they really say and do. Go on sales calls. Go to conferences. How do they talk about the product(s)? What do they really focus on? What do competitors focus on? What is the zeitgeist they’re selling in? Study them in their native environment and discover their true needs.
Like bonding with engineers over programming languages, one good way to bond with sales (and marketing) is over competitor intelligence. Talk about your competitors, their product, their silliest shortcomings or recent flops. Be genuine and show them you know what they’re up against. It will help them open up about what they think of your product, and focus you on your common enemies.
Let marketing and sales own public terminology, but listen to the words they actually use — always participate in naming and major branding decisions, if possible. Names matter.
Just because they bring in the money doesn’t make them the boss. Beware tolerating toxic sales people or allowing them to set the product roadmap. Being pushy, or good at sales, does not make them good product people. Additionally, unlike engineers, in many industries they are generally low skill and replaceable. Be respectful of how vital they are to your livelihood, but they are, more than other functions, best kept contained in their domain. Visit them, do not let them in your domain too often.
If you are selling a new product, or are in a startup doing enterprise, or any kind of in-house sales, and your first sales person is failing, quickly move on from them. As stated above, engage with the reasons for their failure to iterate on your product, but never let a lone salesperson determine the fate of a product. Now… if your second salesperson also flounders, you probably have some product market fit problems. Or your product might just be a piece of shit.

Keeping power users in their place

This is one of a series of essays about how to relate to common stakeholders. Here, I’ll provide some tips for working successfully with power users or internal user stakeholders.

For a while I liked to watch small-business-makover reality shows. In these shows the hosts, who are successful entrepreneurs, sweep in and yell at a small business owner, remodel and fix any simple errors and get hugged. I found these shows about barber shops and hotels, bars and restaurants to be informative in the ways of business, entrepreneurship and the elements of success. For bars/restaurants, one of the most common issues was older establishments that were long overdue for an update. This tension between keeping regulars happy and keeping things fresh to get new customers coming in is remarkably applicable to the issue of keeping an audience of users happy, while also trying to make changes to bring in new users. There are no easy answers in restaurants or in software, but there aren’t TV shows about power users, conversion funnels and MAU decline, like there are about restaurants. Not yet.
Like people that rep themselves in court, or self-diagnosing a medical condition, it’s a bad idea to let the user define their own product roadmap. They lack the needed perspective. Most people understand this, and will either defer entirely to your team or will work in whatever way you want with them. But there are always a few that equate using a product with owning it. Like the fans that nitpick every change from a comic turned into a movie, they equate their attachment and loyalty to lay claim to being not just critics but owners of ideas.
BUT it is super important to genuinely listen, not to their solutions but their needs. Don’t take them for granted, and in particular keep your ear out for change. New complaints, new attitudes, new usage patters, these are your core community speaking.
Or, sometimes, just give them what they want, so they shut up. In some cases, if a task is small and inconsequential, even if it is totally non-strategic, it is well worth doing. Sanding down the edges of software goes a long way with people who have to use it a lot, and it makes them know they are valued and heard. Find the simple annoyances and fix them.
The later in the product lifecycle the more important this balance becomes. As growth slows, learn how to make user-priroitized changes and maintenance a core of your process. It takes different norms to fix complaints and keep things going than to discover value and build new features.
My spectacularly insightful co-worker, Anne Gomez, reaches out to new PMs at our organization when they are, inevitably, confronted with the intensity of our power users, especially when you do something they don’t agree with. She tells them about an early episode of Parks and Rec, the first of many where Leslie is harangued by irrational townsfolk. Leslie’s response is this:

Always keep that in mind when strangers and customers yell at you, call you names, and denigrate your teams work in reviews, forums or whever the pitchfork wielding mobs of your platform gather. At least they care. It’s much worse if no one gives a shit.