Calibrate Hiring Panel Scores and Flag Inconsistent Interviewer Ratings

Paste your panel's scores and notes to get a score matrix, flagged spreads over one point, evidence separated from impressions, and a timed debrief agenda.

You getScore matrix table, disagreement breakdown and bias and protected-trait flags

Prompt

Prompt
You are a hiring operations lead who runs calibrated panel debriefs.

I will paste each interviewer's scores and written notes for one candidate right after this message. Find where the interviewers disagree, check whether each rating is backed by evidence, and build a debrief agenda that resolves each disagreement.

Rules:
1. Flag any competency where interviewer scores differ by more than one point on the scale used. List flagged items first, largest spread at the top.
2. For every flagged rating, sort the interviewer's note into Evidence (something the candidate said or did) and Impression (feelings, "seemed", gut reactions, vague fit comments). Quote the notes. Do not invent content.
3. Name the likely cause of each spread: different evidence, different bar, topic not covered, or impression without evidence.
4. Give each agenda item one question the panel must answer, an owner and a time box. Keep the total agenda to 30 minutes or less.
5. Do not recommend hire or no-hire. Flag any note that mentions age, family, health, accent, religion or another protected trait.

Output format:
1. Score matrix table: Competency | one column per interviewer | Spread | Flag
2. Disagreement breakdown: for each flagged competency, Evidence and Impression bullets per interviewer, then Likely cause
3. Bias and protected-trait flags, or "None found"
4. Debrief agenda: numbered items with question, owner and minutes

End with one line on what is missing from the pasted material, such as unscored competencies or absent notes.

Example output

At a glanceTimed debrief agenda (28 minutes)Four agenda items, each with an owner and a guiding question
  1. Remove the family-status comment3 minRecruiter: confirm Ownership is scored on evidence only
  2. Experiment Design bar10 minMarcus: is sample-size reasoning required at Senior level?
  3. Stakeholder Communication evidence10 minPriya: slow pricing answer or fast VP reframe?
  4. Re-score5 minHiring manager: does anyone change a score, and why?

Example for: a Senior Product Analyst candidate (J. Okafor), four interviewers, 1–4 scale.

1. Score Matrix

CompetencyPriyaMarcusDanaLeoSpreadFlag
Stakeholder Communication24342⚠️
Experiment Design31332⚠️
SQL & Data Modeling43431—
Ownership334—1—

2. Disagreement Breakdown

Stakeholder Communication (spread 2)

  • Priya (2): Evidence: "Took 6 minutes to reach the recommendation in the pricing case." Impression: "Felt a bit rambly."
  • Marcus (4): Evidence: none recorded. Impression: "Great energy, execs would like her."
  • Leo (4): Evidence: "Reframed the churn chart for a non-technical VP in two sentences."
  • Likely cause: Different evidence plus an impression without evidence (Marcus).

Experiment Design (spread 2)

  • Marcus (1): Evidence: "Did not mention sample size or power." Impression: "Seemed out of depth."
  • Priya, Dana, Leo (3): Evidence: Dana notes "proposed holdout group and a 2-week minimum run."
  • Likely cause: Different bar. Marcus asked a stats-heavy follow-up the others did not.

3. Bias and Protected-Trait Flags

  • Marcus, Ownership note: "Not sure she'd keep up with our pace given her young kids." This refers to family status. It must be set aside and should not inform any score.

4. Debrief Agenda (28 minutes)

  1. Remove the family-status comment (Owner: Recruiter, 3 min). Question: Does the panel agree this comment is excluded and that Ownership is scored on evidence only?
  2. Experiment Design bar (Owner: Marcus, 10 min). Question: Is sample-size reasoning required at the Senior level, and did the other interviewers test for it?
  3. Stakeholder Communication evidence (Owner: Priya, 10 min). Question: Which example best predicts exec-facing work, the slow pricing answer or the fast VP reframe?
  4. Re-score (Owner: Hiring manager, 5 min). Question: After discussion, does anyone change a score, and what evidence supports the change?

Missing: Leo gave no Ownership score or notes, and Marcus's Stakeholder Communication score has no supporting evidence.

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • The one-point spread threshold gives an objective trigger, so the debrief focuses only on real disagreements.
  • Splitting notes into Evidence and Impression shows which scores rest on observed behavior and which rest on gut feel.
  • Banning hire or no-hire advice and flagging protected traits keeps the AI in a support role and surfaces legal risk early.

Use it right before a panel debrief, once all interviewers have submitted feedback. Paste every scorecard and note in full into ChatGPT, Claude or Gemini, and include the rating scale.

Common edits

  • Change the flag threshold to any spread of one point or more for executive roles.
  • Ask for a short summary email to send to the panel before the meeting.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in HR & Management