The Product Manager Makeover: The Matter of Metrics
I’ll start with an anecdote.
After about eight or nine months as a product manager, I began to feel frustrated by the lack of autonomy I had been expecting. Previously, I had enjoyed complete freedom to manage multi-thousand dollar cases independently for entire quarters. This new environment felt stifling, so I decided to talk to my manager. It went something like this.
“I think I am ready to handle some of the smaller products by myself. I have been working on them for more than 3 quarters and have an excellent understanding of the upstream and downstream services and blah blah……”
My manager after pausing for a bit “ Ok let’s see, how many daily requests do we see on the ‘X’ service?
“Umm, I don’t have it on the top of my head right now but I can look it up real quick”
“How many people do we get daily who do not complete a transaction?”
“Umm, again I can look it up..”
“Ok, how many of our users are on Android vs iOS?
“Umm….” ”
Do you still think you are ready?”
“I guess not. Ok bye”
While this is an oversimplified view of how the conversation went down, the crux of the discussion was - I did not know my own product.
And why did I not? - Well, no one had ever asked me before.
This is Post 4 of my series - The Product Manager Makeover where I discuss skills needed to make a top-performing product manager. If you haven’t read the previous post, you can read it here.
My daily work revolved around writing PRD, attending meetings for new products, and troubleshooting/ fixing any outages/ issues in production. I had a reliable operations team who did a great job of monitoring for any anomalies and fixing/ calling them out. Whatever little time I had left, it went towards deep diving into the tech. Amidst the packed schedule and a quest to become more technical, I had overlooked a crucial part of my job - metrics.
Of course, I was still thinking about new metrics for products I was launching but what about the existing metrics?
Who will take care of them? Who owns them?
The answer is you. As a PM, the buck stops with you. period. Not your operations. Not your analytics. No one else.
But I have 30 products already and 5 more I am launching this year. How do I take care of all of them?
You prioritize. That’s literally one of your core competencies. Some products are more important than others. Some metrics are more important than others. If you cannot prioritize and own all 35 products, you are not ready for it yet!
Why do metrics matter?
Metrics tell you everything about your product. The past, present, and future.
Now that I have told you about the importance of metrics, one very real problem which you will still face and I faced as well is that metrics are not present in the first place. If I had a dollar for every time I needed some data points on existing products and checked with engineering for events and those events never really existed, I’d be a moderately wealthy man :p
Having said that, there is nothing you can do at that point. You can ask them to create an event and then wait...
You need to make sure as PM that when building a new product, you build an exhaustive list of events and metrics so that those who come after you do not curse you incessantly.
How to do that? Glad you asked.
The ultimate guide for metrics when building a new product
When you start writing a PRD
Include a clear section in the PRD called Metrics
List the North Star Metric at the top and why you chose this metric - this metric is what is going to determine if your product was a success or not
This metric will vary a lot by the product and the team you are in but essentially what you need to think about is if I were allowed to monitor just one metric for this product, what metric would that be
List down all the other metrics that you want to measure; you can skip metrics that will be ingested automatically as they will rarely add any value
For example, if I am launching a payment product, it doesn’t make sense to call out # no of failed transactions and # of successful transactions separately because they will get measured by default for any payment product
What might be interesting are metrics around the Number of failed transactions classified by the reason or classified by services/ stages they failed at
Have a section for events to be ingested and keep this section as WIP as you will keep iterating on this list
When designs are ready
Once designs are ready, go over every screen and list down the events you want on each screen. This will be influenced by the metrics you want to measure as you had decided earlier.
For example - if you want to track at which step a user abandons a transaction, you need to have click events for every CTA
Be thorough and comprehensive; it might be expensive ( time and effort wise) to add an event later post-release
If confused, err on the side of including more events that are needed and useful but never less
Present the list to your lead app devs on both Android and iOS; both platforms have different constraints around what can/ can not be captured
For some events, your engineers will tell you that the particular event already exists; still, let it remain on the list because we will need it later (I’ll explain why)
Once the list of events has been finalized with app devs, include it in the PRD
When engineering solutioning is done
Do the same process that you did with frontend events for backend events
You’ll have less work to do here because most of the time your backend engineers will already have these events in mind for monitoring purposes but it is your job to think from a user lens and make a comprehensive list of backend events
During testing
From the list we made earlier, make sure that each and every event is getting properly ingested. Don’t do random sampling, check each and every event.
This might be the last time you are able to add extra events before release so put your thinking cap on and see if you missed anything.
Pre-launch
Sit down with your analytics team ( you might have a different team that takes care of reporting and dashboarding) and walk them through the product and requirements - they should be involved with the product since the first PRD walkthrough but it is good to recap and revise everything
List down your requirements clearly for the team in terms of
Dashboards to be built
Reports to be sent out daily/ weekly/ monthly/ any other frequency
Alerts to be sent out ( this is sometimes taken care of by the engineering) but analytics will help with the thresholds
For each of the above categories, be clear about the views you want and metrics you’d like to track
A good analytics team will add to the list and come up with valuable suggestions; a bad analytics team will try to reduce their work and therefore the scope but stand your ground
Get timelines for each of the assets required and follow up
Dashboard and alerts need to be tested and ready before the launch
Reports should ideally be ready as well but prioritise the other two
In the meeting with the analytics team, involve your business and operations team as well. Since your business and operations team have been part of the product from the start ( you are doing something wrong if they are not), they should have a good idea of what they need in terms of additional business and operations metrics. For example Business might want to track revenue per day or cost per day or operations might want to track the payment success rate on an hourly basis.
This meeting aims to finalize all the requirements from business and operations along with product requirements of cours
Your operations team will need to come up with an SOP as to how they will monitor the product and how will they report issues/ escalate
This will include a primary POC
Escalation matrix
Relevant actions to be taken in case of outage/ degradation
Contacts/ emails of external stakeholders to resolve issues (optional)
Other details depending on the product
If you have a short-term as well as permanent database where events are persisted for different amounts of duration, you’ll also need to figure out which events need to be persisted in the permanent database for analytical and reporting purposes
Often you might have to raise tickets with the database team to do that or your engineers will do that for you
Make sure data is being ingested into the permanent database before closing the ticket; your devs or analytics team can do that for
Publish a list of all the metrics for the product with your team; this should have the flow chart of services and the metrics along the flow from start to end including the names of events and the description; This will serve two purposes
You’ll have a handy list of all the events in one place for whenever you need to troubleshoot
In case you are not available/ leave the org, the incoming PM will be able to run queries/ troubleshoot without turning mad!
Your teams - business/ analytics/ ops won't bother you each time they want to find a data point
If you have access to the database for running queries, build a dashboard for the most important metric for your personal use; Instead of combing through rows of data, you can keep track of the 5-to 6 metrics you need to track daily
You cannot 100% rely on analytics to build the dashboard on time; you cannot pause the launch because the dashboard is not ready. Be prepared with your own rudimentary dashboard.
During Launch
Stare at your dashboard and see the numbers go up frustratingly slow. Lookout for any signs of trouble.
The first few weeks post-launch
Find 15-20 mins daily to go through the metrics; if there are bugs in the release, this is the best time to find them and fix them in the next release
Partner with your operations team and check with them daily for any anomalies reported
After few weeks
Figure out how important is the product for you; you should already have a very good idea.
If it's critical, you’ll need to monitor the metrics daily yourself + this should be P0 for your operations team.
It is important but not critical, you should still take a lot at the metrics yourself weekly. You can rely on your operations team to bring any anomalies to you.
If the product is not important at all, you can choose not to spend your own time on it. It should still be covered during the weekly/ bi-weekly metrics call.
Stay tuned for the next part of The Product Manager Makeover, where I talk about communication and why it might be the single most important skill in a PM’s repertoire.
