Starting with a guess: how to use data creatively
The good folks at Product Coalition interviewed me the other day as part of their “Data-Driven Product Manager” series.
The interview covers things like:
- thinking about outcomes, not outputs;
- alignment with broader goals;
- finding and measuring leading indicators; and
- the 5 most important metrics for product managers.
Have a read!
(Reproduced from Product Coalition, article by Laure Parsons.)
For me, as a product manager at Notion, a tool that helps product people understand and manage their team and product data, it was exciting to talk to Jock about his experiences with a wide variety of companies as they seek to make better decisions and ultimately, to use the data they have in an effective way.
Starting from scratch
Jock’s first exposure to using data came even before he was officially a product manager.
“I had a slow awakening to data. My first experience was in the early ‘noughties’ at Zeus Technology (a Cambridge, UK startup, long since acquired). I wasn’t even a product manager yet, but I wanted to know where we were coming up short with our products, I’d phone up people who had downloaded our software but had not gone on to purchase it.”
That experience proved to be surprising in two ways,
“…first that it wasn’t as scary to chat with people as I’d thought it would be — I think I’d been worried they’d be annoyed with me for some reason, when in fact they were really pleasant to chat with, and appreciated that I’d bothered to ask.
The second thing that surprised me was that after speaking to a few of them, some recurring themes about where we were going wrong, and what might be a better approach started to emerge — much more quickly than I’d anticipated.
Although I didn’t realise it at the time, I think I’d stumbled upon something really valuable: that having some data is a much better basis for deciding what to do, rather than having no data and simply guessing what to do next. In retrospect it looks pretty obvious, but at the time the whole Steve Blank ‘get out of the building’ thing hadn’t really happened yet, so it wasn’t a natural move.
Of course, things have become a whole lot more sophisticated since then, in terms of understanding, approach and the tools available — it’s been quite a transformation for product management over the last ten to fifteen years! But we’re definitely in a better place now that we’re using data more intelligently.”
Having some data is a much better basis for deciding what to do, rather than having no data and simply guessing what to do next.
Aligning your goals and your numbers
In trying to connect product goals with numbers and analytics, Jock believes the two are always closely related.
“I’m forever annoying product managers by pointing to their roadmap items and asking: ‘What’s the goal — the outcome for this thing? And how will you know you’ve succeeded? What evidence do you need to see?’
It sometimes challenges them to think about exactly why they’re working on something with their team, and drags them away from focusing too much on building shiny things for the sake of it.”
Jock suggests that the disconnect happens more when people are vague about their goals.
“Sometimes it’s due to an absence of understanding about the needs of their users and customers, so they find it hard to decide which direction to take their products, which in turn makes it really hard for them to define and measure how successful they’re being with data.
Other times, it might be because the goals of the wider organisation are vague, but it results in the same problem. Then you either get the situation where no measurement of success takes place at all, or the opposite, when teams try to measure absolutely everything because they don’t know what the most important things are to be measuring — all because their goals are vague.”
You either get the situation where no measurement of success takes place at all, or the opposite, when teams try to measure absolutely everything because they don’t know what the most important things are to be measuring — all because their goals are vague.
Most of the companies Jock works with are still grappling with setting measurable goals. “That’s possibly more of a reflection that product managers who are nailing the user-centric, evidence-led approach don’t usually need my help!”
At Notion we have found that one of the trickiest parts of getting started with using data is knowing what to track. That problem is common and Jock too has seen it with companies he’s coached.
They’re often a little bit lost when it comes to figuring what to measure and how. So often it’s about coaching them to think about what they expect to happen by doing something, and how they could tell whether things have changed, ideally for the better.”
Jock supports the idea of defining measurements that can help a team look toward the future. “The next piece is really about thinking creatively about ways to measure things before they happen and be more proactive, rather than reacting to stuff that’s happened in the past. Finding decent leading indicators of success is tricky, especially if there’s a general lack of awareness of what users do because there’s minimal analytics in place to begin with.”
Of course, as with any new practice, there can be challenges.
“People can also get a bit lost in the numbers — tying themselves in knots. Sometimes in the absence of any evidence to act as a benchmark for what ‘good’ looks like, you just have to start with a guess, then adjust it to something more sensible once the data and learning start to roll in.”
Sometimes in the absence of any evidence to act as a benchmark for what ‘good’ looks like, you just have to start with a guess, then adjust it to something more sensible once the data and learning start to roll in.
The 5 most important metrics for Product Managers
Though Jock resists the idea that data should be prescriptive, one of his central tenets is that “what you need to measure depends on the maturity and state of the product, the company, even the team itself,” so what the metrics are will always be evolving and changing.
1. If it was my product, I’d want to know how effectively we were allowing our users to achieve whatever it is they need to achieve — i.e. some kind of success measure from the user perspective.
2. I’d want some kind of equivalent success measure from the business’s perspective, whether that’s customer satisfaction, profitability or proportion of paying customers, customer lifetime value or whatever.
3. Churn rates and predictors of what tends to keep users coming back to the product are really useful — if you don’t have any users, it’s pretty hard for the product to succeed!
4. I also like to have some kind of view of the health of my product, whether that comes from bug counts, technical debt, customer support tickets, web ops stats or something else.
5. The last piece is around keeping ourselves honest by tracking how often we’re learning from our users and customers, and how effectively we’re using that feedback to inform our approach. This is probably the most important one — if we’re not continually learning, we’re going to go off track pretty rapidly.
The creative side of data
Though they might not fall into traditional “SaaS Metrics” that VCs promote, Jock thinks customer happiness and team morale are key to successful product development.
“I’m painfully aware that I can’t build a product by myself (nor should I try), so the success of my product is entirely dependent on the team of specialists working with me. So team morale and company culture more broadly are really important to keep tabs on.
An unhappy team is often a symptom of bigger organisational issues, and I think product managers have a responsibility to tackle these problems also. Taking some kind of quick ‘temperature check’ on team morale is a really useful exercise to run from time to time in team retrospectives. I like the simplicity of the approach taken by one company that measured company happiness with baskets of tennis balls.”
Taking some kind of quick ‘temperature check’ on team morale is a really useful exercise to run from time to time.
Jock also points out that leading indicators (that can give teams advance warning of future events) require creativity.
“Often the really creative ones come from a really close understanding of user behaviour and habits, and how events connect to each other in unexpected ways. The thing is that it’s difficult to come up with good generic examples because these leading indicators are often so specific to a particular product and user base.”
Thanks so much to Jock for sharing his insights!
Jock Busuttil was interviewed by Laure Parsons.
Read more from Jock
by Jock Busuttil
“I wish this book was published when I started out in product management”
KejiA (Amazon reviewer)