PRODUCTHEAD: Evals are just another rabbit hole
PRODUCTHEAD is a regular newsletter of product management goodness,
curated by Jock Busuttil.
inside my product #
every PRODUCTHEAD edition is online for you to refer back to
tl;dr
Evals are your way to check your AI is behaving
Writing evals is a task a PM can do, but shouldn’t
With AI, “done” becomes a calibration about an acceptable variance in output and behaviour
hello
I was trying to find a video I’d watched a few weeks ago, in which somebody was explaining evals clearly using a worked, real-world example. (Evals are for checking whether generative AI is behaving in its responses.) Most annoyingly, I’d neglected to bookmark it and could not for the life of me remember who was giving that particular talk. Searching vaguely for ‘how do evals work’ was equally a fool’s errand.
As a consequence, for you this week I have a few of the better quality articles I then found describing the ins and outs of evals so that you don’t have to wade through the millions of identikit articles and videos I did.
Evals are another example to add to the growing list of rabbit holes for product people to fall into. Much like ‘doing QA’, wireframing, and the perennial classic, being the only person writing user stories, product people have ended up doing stuff that they can but really shouldn’t. Sometimes it’s because the senior leadership team doesn’t know what a product person is meant to be doing to add real value, sometimes because they’re too stingy to hire specialists, and sometimes because product people can be more easily distracted than a dog seeing a squirrel. For whatever the reason, being superficially able to do something doesn’t make you an expert. And even if you happen to be quite good at doing it, you’re still taking time and effort away from the kinds of task that only a product person is in a position to do.
Evals feel like something a product person should be involved in. To be able to define what correct or acceptable responses should be in given situations would suggest having a good understanding of the nuance of user needs and their context. However, if the only person on your team who has a handle on that is you (the product person), then I’d argue you’re doing it wrong – everyone on your team should understand your users even if the idea of ever speaking to one is mildly off-putting to them.
Evals are not a million miles away from the discipline of test-driven development (TDD), in which you start with the unit test you’re trying to pass, then you write just enough code to make the test pass, then you iteratively improve both the code and the test. Substitute ‘code’ for ‘prompts, skills and tools’ and deal with the non-deterministic behaviour and you’re most of the way there. (Easy, right?)
However, writing the evals is only the tip of the iceberg. Just as with ‘regular’ tests, you also need to be able to iteratively assess and improve both the evals and whatever they’re evaluating, automate the testing process, and update everything when the latest, greatest frontier model unexpectedly starts wanging on about goblins. Sure, you can get super-involved in evals, but likely at the cost of fulfilling your product role as effectively. (And if there’s no discernible difference, that would be telling in its own way.)
I’m not saying that you shouldn’t be actively involved and interested in the evolving methods of building a good product, just that you shouldn’t be the ‘evals person’ any more than you should be the ‘wireframes person’. No matter how tempting it is to dive down the rabbit hole, as a product person it’s not your job to wear all the hats in the absence of dedicated specialists who can do that task quicker and more comprehensively.
Equally, if you do find the rabbit hole more interesting and fulfilling than product role, then that’s cool also. Just remember that someone else will need to become the product person in your absence.
Speak to you soon,
Jock
what to think about this week
Demystifying evals for AI agents
The capabilities that make agents useful also make them difficult to evaluate. The strategies that work across deployments combine techniques to match the complexity of the systems they measure.
How to detect when your AI is doing weird s**t
[Engineering at Anthropic]
Beyond vibe checks: A PM’s complete guide to evals
Writing evals is quickly becoming a core skill for anyone building AI products (which will soon be everyone). Yet there’s very little specific advice on how to get good at it. Below you’ll find everything you need to understand wtf evals are, why they are so important, and how to master this emerging skill.
(I still don’t agree it’s the PM’s job to write them, though)
[Aman Khan / AI Product Playbook]
What “done” means when you’re shipping AI features
Ah, the good old days of sprint reviews. Engineering says they shipped [something]. All tests passed. No P0 bugs. The demo worked like a charm. “Works as designed” had been achieved once again. Sadly, we don’t have the luxury of this simplicity any more (if we ever did).
The definition of “done” has always been the problem. For AI, it has to be completely rewritten.
Dealing with a non-deterministic system
[Jeff Gothelf]
The Engineer’s Guide to AI Evals: RAG, LLMs, and Everything in Between
I’ve shipped a few AI features in my time. And every single time, there’s a moment — usually right before a demo — where someone asks: “But how do we know it’s actually working?” And the uncomfortable answer, for most teams, is: we don’t. Not really. We tested it manually, it felt good, and we shipped it.
Jim doesn’t test. Don’t be like Jim
[Rishi Chhabra / Medium]
recent posts
How to form a go-to-market strategy
A go-to-market strategy isn’t something you tack on as an afterthought to an upcoming product launch. Instead it’s about understanding where value exchanges occur, and figuring out how to make them more frequent, consistent and predictable.
Go-to-market emerges from your discovery
[I Manage Products]
We’re all addicted to AI, but it’s going to be okay
We seem to stuck in a contradiction in which we worry about AI’s effect on our critical thinking, while finding it equally hard to resist using. Why is that?
[I Manage Products]
Canary in the mine: AAA game developers are unionising
Product management has had its own fair share of problems over the last few years. Nevertheless, there are early warning signs from AAA game studios that there may be another storm brewing in tech for us to weather.
Union-busting just isn’t a good look
[I Manage Products]
can we help you?
Product People is a product management services company. We can help you through consultancy, training and coaching. Just contact us if you need our help!
Helping people build better products, more successfully, since 2012.
PRODUCTHEAD is a newsletter for product people of all varieties, and is lovingly crafted from a slightly neater garden.
Read more from Jock
The Practitioner's Guide To Product Management
by Jock Busuttil
“I wish this book was published when I started out in product management. It gives a really wonderful overview of what product management is and involves on a day to day basis.”
— Keji Adedeji, product leader & coach
Jock Busuttil is a product management and leadership coach, product leader and author. He has spent over two decades working with technology companies to improve their product management practices, from startups to multinationals. In 2012 Jock founded Product People Limited, which provides product management consultancy, coaching and training. Its clients include BBC, University of Cambridge, Ometria, Prolific and the UK’s Ministry of Justice and Government Digital Service (GDS). Jock holds a master’s degree in Classics from the University of Cambridge. He is the author of the popular book The Practitioner’s Guide To Product Management, which was published in January 2015 by Grand Central Publishing in the US and Piatkus in the UK. He writes the blog I Manage Products and weekly product management newsletter PRODUCTHEAD. You can find him on Mastodon, X (formerly Twitter) and LinkedIn.