PRODUCTHEAD: Treat AI agents like interns

ok product #

every PRODUCTHEAD edition is online for you to refer back to

---

tl;dr

Delegate the same kinds of task to an AI agent as you would to an intern

Generative AI won’t help you find product differentiators

Evals are a way of checking the quality and effectiveness of your LLM and AI tools

---

hello

I’m writing this following a week in which Amazon learned the hard way why it was a false economy to sack their expensive but knowledgeable network engineers, and the rest of the world realised quite how much of their stuff depends on Amazon Web Services. As a keen sleeper, I was particularly traumatised to read about Eight Sleep’s customers being gently toasted and folded by their smart beds because of the outage.

For you this week #

I’m led to believe that everyone in product is now 100 percent familiar with all aspects of AI, or at least that’s the impression I’m getting. This edition of PRODUCTHEAD is for all the laggards, luddites, and hold-outs, who still want to learn but perhaps feel it’s a little late in the day to be asking basic questions. It’s a little like you’ve been talking to someone for three hours at a party and you don’t want to ask what their name is because you missed it first time. I shall be that helpful friend who habitually introduces everyone to everyone just in case.

This week, I have the inimitable Henrik Kniberg with his take on how AI agents are already acting like junior coworkers at his current company, Abundly; Leah Tharin on why attempting user research on AI-simulated customers instead of real ones is a profoundly flawed idea; and Aman Khan has written a helpful guide to evals (quality control for your generative AI).

Enjoy the edition, and don’t be afraid to say you didn’t catch someone’s name first time around.

Speak to you soon,

Jock

---

---

what to think about this week

AI Agents in Practice

Henrik Kniberg, famous for his clear explanations of what product management and agile, iterative development are all about, turns his attention to AI agents in his keynote at Product at Heart earlier this year. He talks about where agents are most productive, how we define our relationship with them, and how to define our trust in them. He then goes on to demo building an AI agent to illustrate his points.

[VIDEO] Think of it as an intern

[Henrik Kniberg / Product at Heart]

Why AI cannot simulate your customers behaviour

There’s a new paper on LLMs that’s making the rounds, and obviously, it’s the most mindblowing thing that you’ve ever seen, at least according to LinkedIn: “Omg we can simulate customers!”

Anyways…

“Founders would literally rather boil the ocean than talk to customers.”

[Leah Tharin / Leah’s ProducTea]

A product manager’s guide to evals

After years of building AI products, I’ve noticed something surprising: every PM building with generative AI obsesses over crafting better prompts and using the latest LLM, yet almost no one masters the hidden lever behind every exceptional AI product: evaluations. Evals are the only way you can break down each step in the system and measure specifically what impact an individual change might have on a product, giving you the data and confidence to take the right next step.

Evals quietly decide whether your product thrives or dies

[Aman Khan / AI Product Playbook]

---

---

recent posts

Are developers vibe coding themselves out of a job?

And is the increasing reliance by junior developers on AI coding assistants storing up a generational skills shortage for the future – ‘professional debt’, if you will?

So simple, anyone could do it. Wait – don’t fire me

[I Manage Products]

Cloud computing for non-technical product managers

To understand how cloud computing works, we’re going to start with the basic building blocks and work our way up.

And why is it a cloud anyway? (All is revealed)

[I Manage Products]

Navigating your product management career

Ross Webb and I have been chatting about product management career progression.

We cover topics including:

» Thinking of visibility as a strategic competency, not self-promotion

» Controlling your narrative through regular updates

» Building cross-organisational relationships deliberately

» Mapping your stakeholders’ preferred communication styles

A roundtable chat on moving into product leadership

[I Manage Products]

PRODUCTHEAD is a newsletter for product people of all varieties, and is lovingly crafted from painful stretches.