PRODUCTHEAD: How product teams can measure discovery

products out #

every PRODUCTHEAD edition is online for you to refer back to

---

tl;dr

We can and should measure discovery activity and its impact on the team and users

Discovery cycle time and cadence are critical to adopting a continuous mindset

Avoid vanity quantity metrics such as number of research activities; measure quality instead

---

hello

As product managers we’re always told that we need to be measuring things. But sometimes, particularly when it comes to assessing whether we’re doing product management well, figuring out what to measure can be tricky. As an example, I suggest some performance metrics for two teams: one new to discovery research, and one experienced in it.

Finding leading indicators of good product management behaviour #

If you’ve read any articles or books about metrics or analytics, you probably already know that it is more useful to find leading metrics (which predict what is likely to happen in the future) than lagging metrics (which tell you what has already happened). But it can be difficult to find the right indicators that typically lead to good product management behaviour.

The other wrinkle is that the right behaviours to measure and encourage will probably differ depending on both the stage of your product in its lifecycle and how adept your organisation is at product management.

Learning to go through the motions #

It’s a bit like learning to dance. Early on, the desired behaviour for the novice dancer is purely to overcome their awkwardness, start moving, and copy the choreographed steps. Even that may be enough of a physical challenge to begin with. But as they relax and the steps become second nature, then the focus can shift to more nuanced movements, adding intent and expression. Later with practice, the dancer will be able to bend the rules slightly, to improvise and perhaps add their own interpretation to the dance.

Helping a team to learn how to start learning about user needs for the first time (for example) can feel like teaching them to dance. To begin with, all I want them to start doing is having open-ended conversations with actual users about what they do, how they do it and why (no sales pitches allowed).

After a little theory and role-play to get used to the concept, they’re going to need to overcome their awkwardness, and start talking to people. And at the outset they’re probably going to be rubbish at it. That’s fine — they will quickly improve with practice.

Measuring the performance of a team new to discovery #

It’s far less important for an inexperienced team to worry about generating valuable insights at this stage. What matters more is that they get used to organising and having chats with actual users, then repeating the steps until talking to users becomes second nature. At that point they can start to be more directed with the questions they’re trying to answer, and on triangulating ideas and opportunities for new products and features.

Leading indicators that a novice product team is on the right track: #

» They understand why talking to real users helps to turn guesses about user needs into evidence of what they actually need.

» They understand why they have to understand the users’ problem or need before starting to figure out how to solve or address it.

» They know how to find real users to speak to, and not just from existing customers.

» They know how to have a conversation to learn about a particular topic, using open questions and active listening to maximise what they can learn.

» They’re each conducting several user interviews a month and bringing back their insights to the team for analysis.

Although the ultimate goal is to generate worthwhile opportunities for possible new products and features, at this stage it’s more important to establish the right mindset and to start forming the team’s new habit.

Measuring discovery effectiveness in an experienced team #

For a more experienced team, the act of doing user research is engrained as a habitual practice. Now the goal becomes to improve the effectiveness of their discovery research. This is to become better at finding opportunities that are worth trying to solve with a new product or feature.

Leading indicators that a more experienced product team is on the right track: #

» They track and seek to minimise the time between discovery research activities.

» They track and seek to minimise the number of days since a validated idea was fed into experiments and prototyping (to establish whether it was worthwhile to solve the problem). The goal of discovery is not solely to generate worthwhile opportunities, but also to act upon them.

» They track and review how often they are discarding ideas. They do this to encourage them to investigate more ideas, which in turn increases their odds of finding a worthwhile idea. It also serves as a check that they are being sufficiently objective to identify ideas that are not worthwhile to progress.

» They have periodic retrospectives to reflect on:

• how they could do discovery more effectively;
• what significant assumptions they missed or failed to check;
• how they could have learned what they did sooner;

and then take action to do it better next time.

Final thoughts #

If your team is new to doing discovery research with actual users, start by going through the motions to establish the habit, and not worry too much about the results. As user research becomes more a more engrained habit for your team, shift the focus to improving its effectiveness.

Speak to you soon,

Jock

---

---

what to think about this week

How do you measure product discovery?

If you’ve read Lean UX and are practicing product discovery, you’re likely being asked regularly for measures of progress. How will we know if the team is “doing it?” How will we know if they’re doing it well? How will we know it’s having an impact on our ways of working? And an impact on our products and services?

Moving from measuring activity to impact

[Jeff Gothelf]

Doing discovery well: how to measure and guide your team

Setting discovery outcomes is no different from setting any other outcome. Start by asking, what does success look like? How will we know when we are doing discovery well?

How do you know that you are doing discovery well?

[Teresa Torres / Product Talk]

Going beyond quantity for measuring product discovery adoption

I often hear “teams should run more experiments” used as a goal or Key Result to structure Discovery work. By itself, that doesn’t sound so bad. But when you unpack that statement, it becomes apparent that this is “business as usual,” just packaged differently.

Push beyond quantity and into quality

[Tim Herbig]

---

---

recent posts

More than making the right moves with frameworks

When I say I don’t really endorse any particular product management framework, it’s not because I think they hold no value. Rather it’s because it’s too tempting for people to think the framework is all there is: “if we follow the process really, really carefully, we’ll get a good result.” Not so.

Relying solely on the guardrails is missing the point

[I Manage Products]

Getting your first job as a product manager

Job adverts present a chicken-and-egg problem: they all need you to have product management experience to secure a job, but you don’t yet have a product management job to gain that experience.

Don’t let this discourage you!

Practical tips for breaking into a career in product management

[I Manage Products]

Force multipliers

Recently I was explaining to a client why I focus my efforts on finding “force multipliers”. These are what I call activities that allow us to extract multiple benefits from a single piece of work. You could think of it a little like a workplace fusion reaction, where the output ends up far greater than the input effort.

Getting more out than you put in

[I Manage Products]

PRODUCTHEAD is a newsletter for product people of all varieties, and is lovingly crafted from a spoopy disco.