PRODUCTHEAD: How product teams can measure discovery
PRODUCTHEAD is a regular newsletter of product management goodness,
curated by Jock Busuttil.
products out #
every PRODUCTHEAD edition is online for you to refer back to
We can and should measure discovery activity and its impact on the team and users
Discovery cycle time and cadence are critical to adopting a continuous mindset
Avoid vanity quantity metrics such as number of research activities; measure quality instead
As product managers we’re always told that we need to be measuring things. But sometimes, particularly when it comes to assessing whether we’re doing product management well, figuring out what to measure can be tricky. As an example, I suggest some performance metrics for two teams: one new to discovery research, and one experienced in it.
Finding leading indicators of good product management behaviour #
If you’ve read any articles or books about metrics or analytics, you probably already know that it is more useful to find leading metrics (which predict what is likely to happen in the future) than lagging metrics (which tell you what has already happened). But it can be difficult to find the right indicators that typically lead to good product management behaviour.
The other wrinkle is that the right behaviours to measure and encourage will probably differ depending on both the stage of your product in its lifecycle and how adept your organisation is at product management.
Learning to go through the motions #
It’s a bit like learning to dance. Early on, the desired behaviour for the novice dancer is purely to overcome their awkwardness, start moving, and copy the choreographed steps. Even that may be enough of a physical challenge to begin with. But as they relax and the steps become second nature, then the focus can shift to more nuanced movements, adding intent and expression. Later with practice, the dancer will be able to bend the rules slightly, to improvise and perhaps add their own interpretation to the dance.
Helping a team to learn how to start learning about user needs for the first time (for example) can feel like teaching them to dance. To begin with, all I want them to start doing is having open-ended conversations with actual users about what they do, how they do it and why (no sales pitches allowed).
After a little theory and role-play to get used to the concept, they’re going to need to overcome their awkwardness, and start talking to people. And at the outset they’re probably going to be rubbish at it. That’s fine — they will quickly improve with practice.
Measuring the performance of a team new to discovery #
It’s far less important for an inexperienced team to worry about generating valuable insights at this stage. What matters more is that they get used to organising and having chats with actual users, then repeating the steps until talking to users becomes second nature. At that point they can start to be more directed with the questions they’re trying to answer, and on triangulating ideas and opportunities for new products and features.
Leading indicators that a novice product team is on the right track: #
» They understand why talking to real users helps to turn guesses about user needs into evidence of what they actually need.
» They understand why they have to understand the users’ problem or need before starting to figure out how to solve or address it.
» They know how to find real users to speak to, and not just from existing customers.
» They know how to have a conversation to learn about a particular topic, using open questions and active listening to maximise what they can learn.
» They’re each conducting several user interviews a month and bringing back their insights to the team for analysis.
Although the ultimate goal is to generate worthwhile opportunities for possible new products and features, at this stage it’s more important to establish the right mindset and to start forming the team’s new habit.
Measuring discovery effectiveness in an experienced team #
For a more experienced team, the act of doing user research is engrained as a habitual practice. Now the goal becomes to improve the effectiveness of their discovery research. This is to become better at finding opportunities that are worth trying to solve with a new product or feature.
Leading indicators that a more experienced product team is on the right track: #
» They track and seek to minimise the time between discovery research activities.
» They track and seek to minimise the number of days since a validated idea was fed into experiments and prototyping (to establish whether it was worthwhile to solve the problem). The goal of discovery is not solely to generate worthwhile opportunities, but also to act upon them.
» They track and review how often they are discarding ideas. They do this to encourage them to investigate more ideas, which in turn increases their odds of finding a worthwhile idea. It also serves as a check that they are being sufficiently objective to identify ideas that are not worthwhile to progress.
» They have periodic retrospectives to reflect on:
- how they could do discovery more effectively;
- what significant assumptions they missed or failed to check;
- how they could have learned what they did sooner;
and then take action to do it better next time.
Final thoughts #
If your team is new to doing discovery research with actual users, start by going through the motions to establish the habit, and not worry too much about the results. As user research becomes more a more engrained habit for your team, shift the focus to improving its effectiveness.
Speak to you soon,
what to think about this week
If you’ve read Lean UX and are practicing product discovery, you’re likely being asked regularly for measures of progress. How will we know if the team is “doing it?” How will we know if they’re doing it well? How will we know it’s having an impact on our ways of working? And an impact on our products and services?
Setting discovery outcomes is no different from setting any other outcome. Start by asking, what does success look like? How will we know when we are doing discovery well?
[Teresa Torres / Product Talk]
I often hear “teams should run more experiments” used as a goal or Key Result to structure Discovery work. By itself, that doesn’t sound so bad. But when you unpack that statement, it becomes apparent that this is “business as usual,” just packaged differently.
When I say I don’t really endorse any particular product management framework, it’s not because I think they hold no value. Rather it’s because it’s too tempting for people to think the framework is all there is: “if we follow the process really, really carefully, we’ll get a good result.” Not so.
[I Manage Products]
Job adverts present a chicken-and-egg problem: they all need you to have product management experience to secure a job, but you don’t yet have a product management job to gain that experience.
Don’t let this discourage you!
[I Manage Products]
Recently I was explaining to a client why I focus my efforts on finding “force multipliers”. These are what I call activities that allow us to extract multiple benefits from a single piece of work. You could think of it a little like a workplace fusion reaction, where the output ends up far greater than the input effort.
[I Manage Products]
can we help you?
Product People is a product management services company. We can help you through consultancy, training and coaching. Just contact us if you need our help!
Helping people build better products, more successfully, since 2012.
PRODUCTHEAD is a newsletter for product people of all varieties, and is lovingly crafted from a spoopy disco.
Read more from Jock
The Practitioner's Guide To Product Management
by Jock Busuttil
“This is a great book for Product Managers or those considering a career in Product Management.”— Lyndsay Denton