The 3 Best Ways to Assess Candidates

Enter Schmidt and Hunter, in slow motion… to AC/DC

Back in 1998 (when the Spice Girls were huge, Justin Timberlake was the dorky one in N-Sync, and Madonna’s Ray of Light album finally clued the US into the fact that dance music is a thing, which they then preceded to rename “EDM” and get entirely wrong) Frank Schmidt and John Hunter published a meta-study pulling together 85 years of research on how to assess candidates.

‍

It’s a nutrition label for hiring processes

The key output of their study is a simple table (below) showing predictive validity of assessment types, i.e. how likely it is that someone who does well at each type would actually turn out to be good at the job.

(This paper may as well have been delivered to the publisher while walking down the road, in sunglasses, with “Back In Black” playing in the background).

The top validity score here, r=0.54, is actually pretty mediocre, you wouldn’t want to hire on that alone, so you can see why hiring methodology has built up layers of assessment to improve the odds of making a good hire. The data fits.

But to achieve a low false positive rate (bad hires) using layered assessments you need to discard people who fail any one stage… so it follows that low-validity assessments typically also have a high false negative rate (good people turned away), meaning you need to process far more candidates to get the same number of hires.

But what’s most striking about this table is that the assessments most companies use have very low predictive validity. Most companies don’t even use structured interviews, so they best they’re getting is a cross-reference between a handful of very weak indicators.

CVs, for example, contain some of the weakest indicators of ability and the strongest sources of potential bias… peoples names, age, the names of institutions they studied at, their years of experience in an area, and the names of companies they worked at.

(Quick note: we ran a study to test this and our tools had triple the predictive power of even a well-run CV process that had no gender or ethnic bias… fist bump... mic drop).

Not only do individuals broadly suck at hiring, but companies suck at designing hiring processes.

‍

How we decided what to build

As you‘ve guessed we thought it’d be worth trying something different.

We’re looking at a complex problem, but we know the boundaries of it, and thanks to Schmidt and Hunter, as well as all the people researching different aspects of hiring bias, we have rough hint of a recipe.

‍

Our approach

1. Take the most predictive types of assessment
2. Look at the ways those assessments could work at scale, what known bias problems they’re susceptible to, and how easily we can supply the content and supporting tech
3. Build tools that support those processes while minimising any bias risks
4. Get real data from real hiring rounds
5. Continuously monitor each part of the process for both bias and predictive power*

*Note: One day we might have to make a trade-off between predicting the best candidate and supporting diversity, but we haven’t yet been faced with that. So far (touch wood) the two things seem very closely aligned.

‍

Our winning assessments

It’s about time I got to the point and told you what we’ve tried out so far. So far on our platform we’ve implemented (and used both internally and with customers):

‍

1. Work samples

These are just written answers to questions directly relating to the job. They typically test some combination of job knowledge and situational judgement, and are most effective when covering types of work that accurately reflect the job itself.

These answers are rated by a human review team in the hiring organisation, in such a way as to reduce affinity bias, stereotype bias, halo effects, confirmation bias, ordering effects, comparison friction and groupthink… but that’s a story for another blog post.

‍

2. Structured interviews

To make it harder to base interview results off first impressions, biases, or lack of discipline, we built out a process through which hiring managers can define the structure and content of interviews, and interviewers can record their notes and assign scores. The gains here come from ensuring consistency across candidates, helping teams focus on the skills important to the role, and pushing interviewers away from ‘System 1’ thinking towards ‘System 2’ thinking (and hence away from all the associated cognitive biases).

‍

3. Multiple-choice tests (yep)

Multiple choice tests (while obviously limited) are a great tool for larger scale rounds where time constraints demand a filter that scales with candidate volume. They can be used either to test basic domain knowledge, or for general skills, e.g. numeracy.

We see these tests as having the loosest correlation with performance of the three, mainly due to the limiting format of multiple choice. We’d recommend using these as a loose filter, i.e. use the resulting scores to drop the bottom 20%, not the bottom 80%, but broadly speaking it’s a great way to manage bulk hiring. We tend not to use these for smaller rounds or more senior hires.

The commercial numeracy testing market also had severe problems with gender and minority bias to the tune of 8% to 25%, (the gender skew in maths at key stage 4 is around 3%). To illustrate how bad that skew is in practice; if you set a pass mark at the 80th percentile with the industry leading numeracy test provider, you’d get three times more men than women. That’s 3x. Triple. 300%. Our numeracy test still has a skew, but it’s the lowest in the sector and we’ve only just started.

‍

That’s all for now

We’ll follow up with some posts later showing some data analysis of these talent assessment types. Here’s a sneak peak of some work in progress to show how we go about unpicking which interview questions are fair and objective, and actually predict job performance.

In the meantime we’d love to hear what you’re using, and where you think we could be doing things better.