79: The product leader’s guide to interviewing
Because I tend to help organisations build up their product team from scratch, I’m often involved in the interviewing and hiring process, so I’d like to share with you my product leader’s guide to interviewing product managers.
Recruitment is time-consuming #
As a product leader, you will inevitably be hiring at some point. Be under no illusions — hiring is a lengthy and time-consuming process. In each of the head of product roles I’ve been in, at least half of my time was devoted to ongoing recruitment. The larger the product team, the more recruitment you’ll be doing.
Saddling one person with the whole kit and caboodle is effectively saying they can do very little other work for the duration. Your tactic here is to divide up the workload across as many people as is practical to do so.
Unconscious bias training #
How you go about interviewing will depend on the number of product managers in your team at your organisation. As a bare minimum, you as the product leader should receive training in recognising and avoiding unconscious bias. If you have a team of product managers, you should put as many as you can through the same training.
Unconscious bias training has recently been in the news, with the announcement that the UK government is planning to phase it out on the basis of studies that conclude there is little evidence of a positive impact.
It’s true that, by itself, unconscious bias training will not eradicate bias from your interviewing process. However, that’s not its purpose.
Even with the benefit of training, we still can’t help ourselves from jumping to conclusions — that’s why it’s ‘unconscious’. But training does at least highlight the problem areas so that we can take steps to minimise the effect of bias throughout the process. Here’s what you can do.
3 diverse people per interview panel #
You have lots of product managers #
If you have a relatively large product team with members experienced in hiring, you can have two or three interview panels running in parallel. Interview panels should have a diverse mix of three people.
It’s important to have one person who is on every panel to ensure consistency. As the product leader, this will probably be you, as you’ll need to be involved in reviews with all the interview panels you set running anyway.
Each panel needs a lead interviewer: if not you, then a product manager with hiring experience, who’s done some training to recognise and avoid unconscious bias. It can be useful to have at least one other product manager with less experience in hiring also on the panel.
The third member of the panel could be either another product manager or more likely a peer from a different discipline. This could be someone such as a delivery manager (e.g. a scrum master) or designer, developer or user researcher at roughly the same level as the role you’re hiring. Bear in mind that their perspective in the interview process will be coloured slightly by their own professional background. You’ll understand the feeling if you’ve ever participated in an interview panel for a role other than product manager.
If the other product managers on your team don’t yet have experience in hiring, you (the product leader) will need to be the lead interviewer on each panel.
You have no other product managers (yet) #
If you don’t have any other product managers (hey – we all have to start somewhere), then invite peers from other disciplines to join you to bring each panel up to three people. If you’re in this situation, you’ll almost certainly have to deal with all the initial sifting and screening calls yourself, and only bring in the peers when it comes to the face-to-face interviews in order to minimise their time commitment.
Again, as the product leader you would be on each panel as the lead interviewer. And it will be only fair to offer your time in return to participate on any of the peers’ forthcoming interview panels for their own disciplines.
Having a panel of three not only spreads hiring experience around (meaning you have more people potentially to draw on for panels next time around), but also the differing perspectives on the candidate will be helpful to defuse potential sources of individual bias. Research from Applied found having three people on the interview panel would drop the odds of picking the wrong person for a job from around 33% to around 15%. Having larger panels brought diminishing returns, as well as presumably frightening the poor interviewee.
Aim for consistency, but don’t sweat it #
For consistency, the same panel should follow their own cohort of job applicants through the whole process from initial résumé / CV sift through to face-to-face interviews. Set that expectation with the panel members at the outset. Stuff happens, so it’s not the end of the world if one or two of the panel members are pulled away due to unexpected work commitments. If you can’t postpone the interview until the panelists become free again, then you’ll need to call in some favours to find suitable replacements to step in.
Similarly, don’t swap candidates between panels if you can avoid it, unless of course you need a second opinion on a candidate that’s dividing opinion at the face-to-face interview stage (this happens occasionally).
Blind hiring and unconscious bias #
I’ve had the misfortune of working at a company where — and I’m genuinely not making this up — potential sales people would be asked in interview, “What’s your banter like down the pub?”
It should come as no surprise whatsoever that the resulting sales team at that company was an army of white, mostly male clones, each emulating and perpetuating the awful behaviours of the person who hired them. While they got on famously with each other in post-work drinks, their respective abilities as salespeople varied greatly.
Diversity on your team is good. Don’t only hire people like you, unconsciously or otherwise.
Cover letters and résumés are, in my opinion, a terrible way to get a sense of someone’s ability to do a good job. They’re an outmoded, archaic hangover from a time when employers were more bothered about whether a candidate’s face would fit in, rather than their actual aptitude for the job. As we can’t wave a magic wand and replace them instantly with something more appropriate, we have to stick with the current social dance.
The reason why cover letters and CVs are terrible is that they’re an absolute minefield when it comes to triggering unconscious bias. Studies have shown how socio-demographic information can affect hiring decisions, usually to the detriment of non-white, female and minority groups.
There’s even a sensible school of thought that interviewers should only look at a candidate’s résumé after they’ve put them through an exercise that allows them to demonstate their ability.
You should ensure the résumés are redacted to obscure information that identifies name, age, address (which can be a proxy for wealth and sometimes ethnicity), gender, religion, disabilities, marital status and so on.
A 1998 study by Schmidt and Hunter found that work samples (like the practical exercise I mention below) and structured interviews were far more effective ways of assessing a candidate than years of experience, years of education or hobbies.
Consider also removing previous company names in the job history section. Knowing a candidate previously worked for a top 10 company does not make them any more or less likely to be objectively better (or worse) at product management than a candidate from an otherwise unknown organisation.
Similarly, there’s a strong argument for removing the names of academic institutions in the education section, which again can be a proxy for a whole host of socio-economic biases.
It is also a good idea to remove hobbies, because they can arguably trigger a positive or negative association with the reviewer (and can also occasionally hint at gender and ethnicity), and because they have absolutely no bearing on the candidate’s ability to do the job they’re applying for.
It hopefully goes without saying that someone needs to redact all this information before the résumés and cover letters get into the hands of the interview panel. The most straightforward way to do this for someone to print everything out, take a black Sharpie to all the sensitive information, then go and plant a few trees in apology. Alternatively, invest in a system that handles recruitment and applicants in as unbiased a way as possible. Although I’ve not used them personally, Applied and FairHire appear to be on the right track with their products here.
Interview stages #
1: Résumé / CV sift #
Assuming you have a decent number of CVs coming in, the first stage is the initial sift. This is a relatively quick yes/no exercise for the panel based on the candidates’ résumés and cover letters.
The objective here is to weed out the candidates who:
- have clearly applied for the wrong job role (this is often the fault of poor-quality recruitment agents who don’t understand what a product manager is, so don’t judge the applicant too harshly);
- have applied optimistically for a job beyond their experience (think people who have recently left school or university putting themselves in for senior positions with no explanation);
- betray their lack of understanding of the role through their résumé and cover letter (don’t be too harsh in your judgement for this either);
- lack the necessary working visa (assuming your organisation won’t sponsor that process);
- fail to satisfy any other non-negotiable criteria (such as security clearance); or
- demand an unrealistically high salary (aside from being crass).
I would suggest all three panel members look over all the résumés together and agree on the candidates. If you can’t agree on a candidate in the sifting stage, keep them in. You’ll be able to make a better informed decision in later stages.
The proportion of candidates you sift out at this stage will depend on the quality of your sources. It can be worthwhile to note at the end of the recruiting process which sources tended to yield better candidates, particularly if you’re working with recruitment agents on a retainer and you want to figure out their value for money over other sources.
2: Screening interview #
The screening interview is usually a short, 15-minute conversation just to check some basic details with the candidate and to get an initial sense of their suitability. It’s not a formal interview and doesn’t need the whole panel to get involved, just one interviewer. The interviewer’s mindset should be that the candidate is suitable for the next stage unless they disqualify themselves during the chat.
I would typically do this by telephone simply to keep it quick and low pressure. Video calls are great and everything, but if technology faff is adding a 25% time overhead to each call, it’s not worth the effort. Plus candidates (and interviewers) don’t need to look presentable on a telephone call, which helps to keep the pressure off.
By this point, the interviewer should have read through their cover letter and CV and made a few notes. Overall, the interviewer should be expecting what the candidate says on the phone to stack up with what they’ve already written in their cover letter and résumé. More specifically, they’re checking:
- whether the candidate is organised enough to be in a suitable location, with phone signal, at the agreed time (it’s not a good look if I’m going to voicemail when I phone them up, but equally it’s not a deal-breaker if there are extenuating circumstances);
- why they’re looking for a new role (this can sometimes open up a whole can of worms);
- what they understand about the role they’re applying for (this can very quickly highlight people who have failed to prepare or have applied for every single job going);
- why they want to work for my organisation;
- working visa or other non-negotiable criteria (e.g. security clearance), if applicable;
- availability for face-to-face interviews (it’s also good to know if they’ve already received a job offer, or are expecting one soon); and
- notice period, if applicable.
At this stage, it’s best only to filter out the candidates that lack the required working visa or other clearances, or who have withdrawn themselves by accepting another job. You will occasionally get some weird answers to the obvious questions, but as long as they’re not showstoppers (such as the candidate is overtly racist or something equally unacceptable), it’s better to note these down for discussion at interview stage with the rest of the panel present, rather than outright rejecting the candidate.
3: Face-to-face interview #
Structure and scoring #
This is where the candidate will come in (or appear on a video call) to be interviewed by the panel. It’s important for the panel to sit down and agree the structure of the interview, what main questions to ask, and how the candidate will be scored in each relevant area. It’s up to you, but I tend to use a scale of 0-5 where 0 is ‘no evidence of capability’ and 5 is ‘exceeded the required level of capability’.
Note that this scale is relative to the job role you’re recruiting for. The way to earn a ‘5’ will be quite different for a new product manager versus a chief product officer.
Typically candidates scoring mostly 4s in each area with the odd 3 or 5 will be likely hires. Candidates scoring 0 or 1 in one or more area from all three panelists should probably be avoided. Total the scores in each area, applying any weighting on questions consistently to all candidates.
A practical exercise #
I like to start interviews by springing an unexpected exercise on the candidate. It’s perhaps a bit mean of me, but it helps to cut through any pre-prepared answers, puts them under a bit of pressure (that’s product management, right?), and usually causes them to revert to their usual persona rather than their polished interview persona. It’s not all bad, though. It’s also an opportunity for the candidate to demonstrate how well they can do the job.
The kind of exercise I use is a realistic scenario for the level of role I’m recruiting. It’s not an armageddon or Kobayashi Maru, nor is it an ‘estimate the number of gas stations in the US’, because none of those is helpful in determining aptitude as a product manager. Even when recruiting someone into their first product manager role, a realistic scenario is helpful to tease out the candidate’s instincts and how they reason things through.
I leave the candidate to mull over and work through the exercise scenario for 15 minutes, then come back to debrief them with the rest of the interview panel for another 15 minutes or so. I tend to apply more weighting to the score for the exercise because it’s a demonstration of what the candidate can do, rather than what they say they can do.
We then spend the rest of the interview time going through the remainder of the questions. It’s a good idea to alternate the panelist asking the questions. Each panel member will be scoring the candidate independently and taking their own notes, which in some organisations will be made available to the candidate on request. Don’t forget to allow a reasonable amount of time for the candidate to ask questions at the end.
After the candidate has been escorted away, it’s a good idea for the panel to reconvene to swap impressions and scores. Explore any major discrepancies or disagreements, but each panel member should not adjust their scores for the candidate to align with the rest of the panel. Take the average of the three panelists’ total scores.
Offering jobs (or not) #
Particularly if you have a small number of candidates to begin with, it can be tempting to make an offer to whoever scores the highest in their cohort, even if that score isn’t particularly high. I would counsel against that, at least until you’ve had a chance to interview more people. It’s not that you’re holding out for a truly exceptional candidate, but making a bad hire will cause you much more of a problem than having a vacancy open for longer that you’d prefer.
Keep a record of unsuccessful candidates and their interview notes. Flag candidates you would have offered the role to if they hadn’t been pipped to the post by an even more exceptional candidate. You should also expect an unsuccessful candidate to ask you for feedback, though not all will. Be honest and offer constructive suggestions to improve, but don’t crush the poor candidate’s spirit.
On a related point, it’s helpful to cross-check new applicants against previous ones. If you have lots of recruitment going on, you’d be surprised at how many unsuccessful candidates will optimistically re-submit their résumé for other roles advertised.
Final thoughts #
Interviewing as I’ve outlined above takes several people a lot of time and effort, but it is very much worth it. We need to compensate for our inherent tendency to jump to conclusions from very little data, and unconscious bias is — well — unconscious. We can’t deprogram our accumulated set of cues, instincts and triggers, so we take steps to minimise or counter their effects.
Interviewing someone should be about assessing as objectively as possible whether they have sufficient aptitude, interest, motivation and emotional intelligence to carry out the role you’re recruiting for. It shouldn’t be about anything else.
Further reading #
“What is blind hiring?”, Kate Glazebrook, Applied, 27 November 2018 (retrieved 23 November 2020)
“How to anonymise CVs”, Andy Babbage, Applied, 12 December 2018 (retrieved 23 November 2020)
“How to stop getting hiring wrong”, Emily Burt, People Management, 24 January 2019 (retrieved 23 November 2020)
“Is blind recruitment the secret to the perfect hire?”, Annie Makoff-Clark, People Management, 24 January 2019 (retrieved 23 November 2020)
“Best practices in recruitment and selection (according to science)”, Joe Caccavale, Applied, 22 September 2020 (retrieved 23 November 2020)
Get articles when they’re published
My articles get published irregularly (erratically, some might say). Never miss an article again by getting them delivered direct to your inbox as soon as they go live.
Read more from Jock
The Practitioner's Guide To Product Management
by Jock Busuttil
“This is a great book for Product Managers or those considering a career in Product Management.”— Lyndsay Denton