How to measure product management – ProductBeats talk

I gave a talk on Tolpagorni’s ProductBeats weekly webinar series about how to measure product management. You can also read the article the talk was based on, if you like.

Video, slides, sketchnotes and transcript after the break!

Video

Slides

Magnus’s write-up of the talk is available behind a login page on ProductBeats’s Haaartland page, but it’s free to sign up.

Sketchnotes

Transcript

Introduction

[MAGNUS] With us today we have Jock Busuttil. Hello Jock can you hear me?

[JOCK] I certainly can! Let me just start my video as well. Hello!

[MAGNUS] Good morning, Jock! You are working with Product People all around the world and you’re based in the UK. What do you see when you work with these product people? You’ve also been writing a book on product management.

[JOCK] Yeah!

How do product managers evolve in their roles?

[MAGNUS] How do people evolve in their roles as product management? How do you do that?

[JOCK] So I think the main way people evolve was, as they get more experienced they can either take on more complex products, or they can take on larger numbers of products, and that’s kind of a show of their technical development as a product manager. But to show their personal development as a product manager as well, one of the most important things I would do with my teams would be to ensure that, as people got more senior, they took on more responsibility for sharing their experience with the rest of the community. So in other words they took on more of a coaching or a mentoring role within – alongside their hands-on product management. And I think that regardless of your job title, even if you’re just starting out there’s a role every product manager can play in sharing their experiences with other people within the product management community. And as you get more senior that only becomes more of a practice.

[MAGNUS] Wonderful I think that’s a that’s a great uh thought. I think whenever you know when trying to teach or trying to share your experience then you really need to reflect on your knowledge. And I think that reflection of your knowledge is really something that is – uh – we all benefit from [it], whether we do it in training classes on or either on teaching other ones which might be the best way of learning something. But also there is something that you want to share with us today, Jock, around what is it that we want to improve within. Then we also need to measure. We need to measure our work and how well are we doing this. So Jock Busuttil, author, speaker, founder of Product People in the UK and just between London and Cambridge, the screen is yours. And I would like you to share your thoughts on how do we measure product management.

[JOCK] Perfect!

[MAGNUS] Let me make your screen seen.

[JOCK] Can you see me, everyone?

[MAGNUS] Yes we can see the screen perfectly, thank you.

How to measure product management

[JOCK] Excellent! Good! Well thank you very much for letting me come and speak this morning, Magnus. Yeah, I’m going to talk about how we measure product management. So, let me just – you know who I am, so we don’t need to see this. My book is there on my website if you’re interested.

And I believe, Magnus, you’ll be making the sketchnotes, the graphic representation of these slides available after as usual on the on the newsletter. So let me tell you about this story, then. So about five or six years ago, I was head of product for the UK’s Ministry of Justice digital team. And yes, maybe it’s still a little bit strange for government departments to have digital teams, but now it’s fairly commonplace, at least here in the UK.

My team at Ministry of Justice Digital

So within my team we had 12 product managers and they had differing levels of experience. Each of these product managers had their own multi-disciplinary team, so they had a product manager they had user researchers, designers, developers, delivery managers, which is kind of another way of saying scrum masters, but with any discipline rather, not just Scrum. And other roles would come into the team as needed. So a nice, autonomous, multi-disciplinary team that were empowered to build product.

Now while this happened all quite a while ago, and things have changed since those days, at the time the senior management team was really struggling with the idea of a flat organisation so they kept trying to reassert a command and control structure a top-down structure to the organisation. And that was very much to reflect the way in which government and the civil service works. So they’re struggling a little bit with this flat hierarchy. And so we have this team that’s thriving on autonomy, really responsible to achieve the goals with their product, and to solve user problems, and they have this kind of conflict a little bit with senior management.

The dreaded stack rank

So senior management are there trying to cut costs. They’re saying – they’re looking at these teams. We’ve got 12 product managers, we’ve got 12 teams at the moment, we need to cut costs a little bit. So what they asked me to do was to create what’s called a stack rank. A terrible idea, which was essentially a way of ranking all of the product managers in order of performance so that the worst ones essentially would be fired, okay? And this is a terrible idea for a whole variety of reasons.

Now my team was diverse. It was – they’re all strong performers and a stack rank is a way of really devaluing people who are used to working autonomously. It turns them into a commodity and not something of value. It ignores their experience, their individuality of their skills and their approaches, and it kind of constrains. It’s a very reductive way of pulling everything down to a single measure of measurement, a single dimension of measurement. And even if it was possible to kind of pull and rank all these people on just a single measurement the team would rightly object to it and would probably just leave on principle, which was which was terrible.

And you have to bear in mind that, well at least at the time, recruiting someone would take at least six months just due to the slowness of the processes and the security vetting and that kind of thing, whereas at the time with the contractors, if one of them left, they could literally leave the next day. So you know it’s a very difficult thing to make sure there’s enough people available at the right times. So they’re all product managers. They each did product management differently. And that’s what I really needed them to do because they’re all solving different user problems.

So I was being pushed by senior management to measure performance. I couldn’t refuse to do it completely. All that would happen is they’d get rid of me, replace me with someone who didn’t care as much, and then they’d rank my team and find the bottom 30 percent anyway. So what would serve my team better, as their head of community was by mostly doing what senior management wanted, but in a way that made it really, really hard for them to actually rank my team. So this is the system of measurement I came up with. I called it a talent map.

The talent map

Now what this means is – I’ll take you through what we’re going to do. So step one was to sit down with my team. And in one of our weekly meetings we discussed what was going on. I gave them the pressures that we were under, discussed the need for cost-cutting, but also discussed with them how it was so important for me to show that actually a stack rack wasn’t possible. So what we did was we discussed the dimensions for how we’re going to measure product management performance. So we discussed what were the most important skills a product manager needs. And we had three groups of these skills: we had the emotional or soft or people skills that we sometimes think about; we had the technical skills like product roadmapping or prioritisation, or that kind of thing; and then we had other disciplines. So not necessarily part of product management but the ones that we need to have an understanding of to be able to have sensible conversations with our multi-disciplinary team. So we need to understand a bit about design, we need to understand enough about user research to have a conversation with our user researcher, but we’re not necessarily expert in doing those things ourselves. And so these three sets of skills we used as our metrics.

Next step was for each of my team – so each of the product managers – to do a quick survey to assess their own experience. And what they were going to do was to score each of their skills honestly out of five. So zero – no experience of that. Five they’re an expert. Three was kind of in the middle and they’re okay. And what we did was then I plotted the results, averaged off the team, and compared the individual results to the team averages, and then shared those results back with just the individuals. I only shared the team averages with the whole team, as it were. So each individual got to see their own little report. And this was really interesting – we’ll look into kind of some of the ways we can see this – but it became very clear where they felt they were stronger and where they felt they were weaker. And this was really useful for sort of coaching later on.

Peer learning and knowledge share

But also a really interesting thing happened. When the team realised that some people were – needed help in certain areas or when they realised that they were reasonably expert in certain areas, what they would do is they’d work together to start filling in the gaps of their colleagues. So they would start to share their experience. So if somebody was more experienced in, say, kanban then they would buddy up with people who were less experienced in kanban and share that knowledge. Similarly if somebody decided that they were not as great at public speaking as maybe they could have been, then they took on more responsibility for putting themselves forward to do more speaking engagements.

The subjective perception

So this was all really valuable so far. But we had even more interesting stuff to learn. So the step three was, we asked the delivery teams, so the actual multi-disciplinary teams that worked with the product managers, to rate how they thought the product manager was doing, and this was anonymous. And so what I was interested in was the subjective perception, okay? So this isn’t objective in any way, it’s literally just opinion. But I wanted to see where the product manager and the teams were in agreement, so they both thought that the product manager was strong in a particular area or weak in a particular area, but also it was kind of interesting where the product manager rated themselves weak at something but the team rated them as strong, or the other way around. Because this led to useful areas for further investigation.

And then the last step was to get the same to run the same exercise, but this time at the top of the organisation with the senior managers. And this brought me to a very quick conclusion, which was that with some exceptions, it was clear that senior managers had absolutely no idea how to answer the question. They just didn’t know what product managers did, and so they were not really close enough to the product managers to make any kind of sensible assessment. And for me this was a dual failing: this was a failing of the senior management team, who were supposed to be closer to our team than they were in practice, but also it was a point of improvement for the product managers, who should have been better at stakeholder management, and should have been closer, putting themselves closer to the senior management team. Because bear in mind in government in particular, to keep things moving we absolutely needed to have the stakeholder,s the senior management team on board with what we were doing. So it’s very, very important to have good stakeholder management with that team. So the lesson we learned was invest more time in stakeholder management.

Personal development plan

So this led to the personal development plan. So when we got all these results together – I’ve actually got an example for you here. First of all, it’s anonymised. There was no product manager called Petra on my team, so there’s no problem with sharing any of this. But this was the kind of thing that I would use to keep track with my one-to-one coaching sessions with each of my product managers. And we’d have one of these for each quarter and update it every quarter. So at the top you can see the date. We can see the section about the individual’s goals. And this is what they want to achieve by when, okay? You could use OKRs for this if you wanted to, it’s the same kind of idea, but the idea being it’s a goal, it’s an outcome we’re looking for. Then we have the column which says how we’re going to achieve this, and this is some ideas that we come up together with in our one-to-one coaching sessions. And then to keep track of progress, we have what they’re doing to achieve this. And this is a way just to keep a quick summary of what they’re going on – how they’re doing as we go along each session. And then we’ve got the individual graphs. We saw these briefly earlier. But you can see how we can identify potential areas to work on a one-to-one coaching sessions.

So in this particular case, the person – Petra has decided that she’s you know lacking a bit of focus, lacking a bit of motivation, and maybe a bit of patience in the soft skills. And so these are things that we could work on in the coaching session. We go through the different assessments by self-assessment, by senior management, and by the delivery team. And we go on to the technical skills. And then in the self-assessment here we’ve got opportunity, because Petra is particularly good at metrics, to share knowledge with the rest of the team. And again there’s some potential areas around product vision, business case, that might be worth exploring in one-to-one sessions to develop further. So this is really good, because very quickly you can see what the topics need to be to ensure that we’re developing our product manager as well.

And then when we get on to the pink ones, what you can see are – these are the skills that are related to product management, but not actually product management themselves. So user research, writing copy, information security, development, design. So things that we’re not expecting them to be expert in, so we’re expecting the graph to be a bit smaller, but it is kind of interesting to pull out situations where individuals do have a special skill, alongside the product management that maybe we can draw on as we need to.

The results

So what were the results? Well, nobody got fired, so my objective was achieved! And I wasn’t just being sentimental or overprotective. I think the cost-cutting exercise at the time was short-sighted. It didn’t take into account the amount of work the team was being tasked with, and indeed how much we saw coming down the line. Because bear in mind we were a lot closer to the roadmap, and we could see the projects that were coming down the pipeline. So bearing in mind that you could get rid of a freelancer in a matter of day, but recruiting replacement takes six months, even if it was quiet right now, we were going to have suddenly three new projects pop out of the woodwork, and they’d be approved without warning and suddenly we would be understaffed again. So it was really, really important for me to make sure that we had a sufficiently good number of product managers available to do things.

Product manager performance is multi-dimensional

So five things we can learn from this exercise. So first of all, people performance is multi-dimensional. Product management’s a generalist role requiring multiple skills, which means you need to look at different aspects, different areas of performance when you’re assessing this. And to be honest, if you look at other roles other than product management, you’d realise they’re multi-dimensional too, so we can’t just go and pull out one metric and say everyone is going to be stack-ranked on this particular measure.

Stack ranks are reductive and devaluing

The second thing is stack ranks are such a devaluing thing to do to people. It’s really reductive. And if you do have high performing people in any discipline, attempting to do this to put people in this kind of competitive state with one another is going to be really devaluing for them. And if they’re sensible enough, they’ll probably either game the system to make sure that they get a really, really high score in that single metric, or they’ll leave because it’s a system they don’t want to work within.

Hurray for diversity

You want a diverse team because people have different strengths and weaknesses and that’s okay. User problems are not all identical, so you want a mixture of personalities, experience and abilities in your team to be able to tackle them in different ways. This is also the reason why I’m so keen to make sure that people recruit product managers from any market sector, not just the one they happen to be in – uh – that they’re looking for a role in. So you know if you’re working in automotive and you can apply that to analytics, or if you’re working in healthcare and you can apply that to developing APIs, then I think this background, this different perspective that people have, is really, really valuable, and actually something to to look for rather than to try and recruit out.

Opportunities to improve

Of course, the whole purpose of this exercise is to spot opportunities to improve. The product managers could assess themselves and identify their own areas to develop, and of course I could help them in the coaching sessions, but also the opportunities to share knowledge and experience with their peers. There’s such a power in peer learning. So this was a really valuable exercise, and we didn’t see gaps and experience as failings, we saw them as opportunities to grow.

Storytelling with data

And last of all don’t forget to think about your storytelling. If you have data to tell a story, good graphs have power – have conversational power. It’s never a waste of time to spend a bit more effort making your statistics easier to consume, particularly to people who just want to find out what the story is. So for me to tell that story back to senior management about the strengths and weaknesses of my team, these graphs were really, really, really useful.

Where to find the self-assessment toolkit

So there’s a toolkit available on my website. It’s all licensed under Creative Commons, so you’re free to reuse it. You need to attribute it to Product People and you can’t sell it or make money from it, because that’s not the point. But you are free to use it and apply that to your own teams if you wish to, or indeed just yourself.

Observations and questions from Magnus

[MAGNUS] Wonderful thank you very much! Thank you very much, Jock. Let’s do the digital clap for Jock and see how many claps we’re getting today for Jock. Let’s do the clap clap. We have some – uh hello Torbil – I didn’t see you before. Thank you very much Jock for sharing your thoughts. What was interesting is of course that you used the name Petra – that’s also one of our colleagues.

[JOCK] Oh I’m sorry! I’m very sorry, Petra, if you’re listening. I wasn’t intending to do that.

[MAGNUS] You’ll have to send that over and we’ll need to have a personal self-assessment on Petra to make sure that it fits. What was interesting, what we’ve also found – we do similar setups also when doing self-assessment. And both of the product managers and also with the stakeholders when we do these activities to measure and assess product management activities. And we have the same result as you do, that most of the time that the stakeholders are actually give a higher grade to a lot of the skills than the product managers themselves do. I could see that on your examples as well. Was that just on this example or was that something that you found as a pattern in your survey?

[JOCK] Yeah so it’s interesting. So one of the soft skills I think, if you were eagle-eyed enough to see it, one of the soft skills is humility, because we reckon or we as a team reckoned that a lack of ego, a sense of humility is very, very important in a product manager. And of course this has the double-edged effect of product managers being absolutely terrible about talking about how good they are when they’re in coaching sessions or in review sessions, or trying to get a pay rise. So sometimes we need to be a little less humble and we need to say, “Actually I’m quite good at these things.” But essentially, yeah, it was across the board that most of the way round, the team was saying, “I’m not so good at these things,” when actually everyone around was saying, “What are you saying? You’re brilliant! Don’t you know?”

[MAGNUS] I think that’s almost to some extent a personality of a lot of the product managers that …

[JOCK] Absolutely!

[MAGNUS] We are hardworking, gritty individuals and we know what we can improve. So since we’re looking for that improvement, we’re always judging ourselves maybe a little bit harsher than the environment is doing.

[JOCK] Absolutely. I think it’s important to be self-aware, and I think that if people – I don’t think arrogant product managers work very well.

[MAGNUS] You got a point there! A question here: when you do these assessments, how – what do you do if you get a new member, or if you learn new things, how often would you need to do these assessments that you shared. Would it be typically once a year that you normally do in these kind of areas, or once everyone – when you have new consultants coming in?

[JOCK] Uh it kind of depends. So with every new person then I’d make sure that we did one with every person on the team, so that we had something – because remember we’re using these on a weekly basis for one-to-one coaching sessions, so I needed to have a sense of where to develop these individual. But in terms of doing the overall survey I think maybe no more, maybe once or twice a year. So maybe once every six months at most. Because the things aren’t going to shift that much, that quickly. And also you need to give them a couple of quarters, but you know they’ve got stuff to do, they’ve got products to manage! You need to give them a chance to actually have the time to work on those skills and get better at them. And I think if you’re measuring it too often you won’t see the incremental change, whereas if you leave a decent gap between the big assessments, then you’ll be able to say, “Look, before you thought you were pretty bad at public speaking, but now you’ve had a chance to do six different talks in the time, and now you’re you’re feeling a lot more confident about it.”

Closing remarks

[MAGNUS] Wonderful! Thank you very much again for sharing, Jock, on measuring and assessing product management. I think that both doing the self-assessment and having stakeholder assessment and combining those things, and put them on some type of scale I think that’s a great way of doing it, and it has helps a lot in doing it. And of course it’s difficult to identify what really to measure here and the scales. And those you can find on Jock’s. I will share those also in the ProductBeats community, where we’ll have the graphical recording by Fia that we share that afterwards along with the presentation from Jock and our reflections.

In the chat I wrote down, I posted a link to the upcoming speakers. So there’s a long list of speakers till October right now. I think we have the list this is uh it’s public of all the speakers till October. However next week, we’re having vacation. Next week is our first Tuesday when we’re not having a session, so I’m sorry for that. I hope you’re able to cope one week without the ProductBeats webinar and good luck with that! I thank you all for listening today and wish you all a great vacation for those of you or who are now go on vacation, make use of dedication, read the book Grit or Peak or maybe buy the book from Jock and get some good summer reading and reach next level of high performance. Thank you very much! Enjoy your Tuesday and thanks again Jock.

[JOCK] You’re welcome, thank you very much.

[MAGNUS] Thank you, bye-bye!

My next talk

My next live talk is on Thursday 23rd July 2020 for Product Anonymous, Melbourne on how to save yourself from product management hell.