Structured interviews and scorecards: how to build them

What the research supports

Predictive validity is the correlation between how candidates score on a hiring method and how they later perform in the job. Schmidt and Hunter's 1998 review tabled 19 selection procedures and reported an average validity of .51 for the structured interview and .38 for the unstructured interview.^[1] Their table gave work sample tests the highest value, .54, and reference checks .26.^[1]

A 2022 paper by Sackett, Zhang, Berry and Lievens argued that earlier corrections for restriction of range often overcorrected, so many validities were overestimated. Most methods that ranked high before stayed high, with mean estimates lowered by .10 to .20, and structured interviews came first with a mean validity of .42.^[3][2] The 2023 follow-up reports a standard deviation of .19 for structured interviews and an 80 per cent credibility interval from .18 to .66, which the authors say is better read as .42 plus or minus .24.^[2]

The revision is contested. Ones and Viswesvaran describe the case for lowering the estimates as conceptual rather than empirical and argue that range restriction corrections in concurrent validation studies are appropriate.^[4] The practical reading for an employer is directional: structure helps on average, the figures are averages with wide spread, and they do not predict any single candidate. Our guide on choosing the right hire covers the debate in more detail.

What makes an interview structured

Schmidt and Hunter drew the distinction plainly. An unstructured interview has no fixed format or set of questions, the interviewer often asks different candidates different things, and responses are usually summed up in an overall impression. A structured interview is the opposite on each count, with questions usually set by an analysis of the job and responses scored individually. They noted that structured interviews cost more to construct and use.^[1]

That cost is paid once per role. The questions and the scale are written once, then used for every candidate. An office that hires several people a year into the same role recovers the effort quickly. One that hires into a role every few years may not, and that is a fair reason to keep the method light.

The five parts of a scorecard

A scorecard is a template bound to a role and to a stage of the interview process. Practitioner and vendor documentation, including that of applicant tracking vendors, converge on five parts. This is a synthesis of vendor material, not one platform's schema.^[5][6]

• Criteria. Four to eight dimensions of the role, drawn from the job. The guides say that more than about a dozen dilutes the signal.^[5]
• A shared scale. One scale for the whole organisation, usually 1 to 5 or four levels from does not meet to exceeds. Some practitioners argue that four levels removes the safe middle; that is asserted in the guides and not strongly evidenced.^[5]
• Behavioural anchors. A written description of what each score looks like for each criterion, so that one interviewer's 3 is not another's 4. Behaviourally anchored rating scales pair a numeric level with a concrete, observable behaviour derived from job analysis.^[7]
• Evidence notes. The specific quotes and examples that justify each rating.^[5]
• A recommendation. Hire, do not hire or undecided, with the reason.^[5]

The counts of four to eight criteria and the choice between four and five levels are convention. They are sensible starting points for a small office, and not findings.

An illustration for a property consultant role

The table below is an example written for this guide, not taken from a client or a source. The criteria should come from your own analysis of the job.

Ask every candidate the same questions in the same order, and score only against the criteria. Questions on matters unrelated to the job, such as age, family or religion, add noise and risk. Employment discrimination law is a subject for a labour-law adviser; the habit of scoring against the criteria alone is a good guard.

Score separately, then discuss

The single most important workflow rule in the guides is sequence. Each interviewer submits a scorecard independently, before any debrief. Locking scores first blocks anchoring on the loudest or most senior voice, halo effects and after-the-fact rationalisation.^[5][6]

The scores are then put side by side per criterion. A common heuristic is to flag any criterion where two interviewers differ by more than two points and discuss only those. The debrief is described as a short meeting, often 30 to 45 minutes and within about a day, that reconciles evidence rather than voting until everyone agrees, with the hiring manager making the call. Divergence is treated as information and not noise.^[5][6] These timings are practitioner convention, and the guides give no study showing that they produce better hires.

Structure and fairness

Practice guides also argue that structure reduces bias, because the same discipline of fixed criteria, anchored scales, independent scoring and evidence requirements attacks noise and unfairness together. One guide cites standardised subgroup score differences of about d = 0.23 for structured interviews against about 0.56 for unstructured ones.^[6] The figures come from a secondary summary of meta-analyses and not from the source papers, so treat them as indicative.

The 2022 revision also reported group mean differences for each predictor, a reminder that validity is one of several trade-offs in designing a selection system.^[3] A scorecard is a tool for consistency. It does not on its own make a process fair, and it should be reviewed with the people who use it.

Where the method does not fit

The authors of the 2023 follow-up note that work samples and job knowledge tests are not generally suitable for entry-level hiring where skills are built after hire, and that structured interviews are not easily used in high-volume screening.^[2] For an experienced consultant, a short exercise that samples the work, such as presenting a listing or handling a difficult enquiry, adds a second predictor to the interview. For a new entrant to the trade there is little to sample, so the interview carries more weight.

No source reviewed for this guide measures any of this in the UAE, so the figures describe the research literature, which draws mostly on other countries.

A one-page version

• Write four to six criteria from the job, each with a one-line description of a strong and a weak answer.
• Choose one scale and write what each level means for each criterion.
• Write the questions once and ask them in the same order of every candidate.
• Have each interviewer score and note evidence before talking to anyone.
• Flag criteria with large gaps between scores, discuss those, and record the decision and the reason.

A shared document is enough for a small team. The Works ATS, which The Works Group sells and which is in build, keeps one record per person with a date on each of nine stages and a reason for each exit. Its product page does not list interview scorecards, so a scorecard would sit alongside it in a document.^[8]

Questions