The release note says the vendor has “improved scoring accuracy.” The hiring team has 40 applicants in process, half scored last week. Can the recruiter put all 40 in rank order on Monday morning?

A scoring update calls for a review. The extent of validation work depends on what changed, whether scores still support the decision you use them for, and what evidence covers your jobs and applicants. A display correction that leaves stored scores, rankings, and recommendations intact presents a different question from a new model that changes who advances.

Criteria’s survey of 1,027 hiring professionals found that 56% used AI somewhere in hiring. For an assessment owner, purchasing the tool is only the first decision. Someone also has to own the decision to accept the next scoring version.

Follow one applicant across the change

Imagine a customer service assessment with a passing score of 70. A vendor starts giving more weight to response brevity. A candidate who scored 72 under the old version now earns 68 on the same answers. The labels on the dashboard look familiar. The candidate's next step has changed.

Ask the vendor to score an appropriate comparison set under both versions and identify who crosses the cutoff in either direction. Look at cases near the threshold, differences by role, and relevant subgroup results where the data support interpretation. If the vendor calls brevity a better predictor, ask which job outcome it predicts and whether that evidence applies to the work your agents do. Agreement between versions tells you about continuity. Evidence against job performance tells you something different.

Decide what happens to the open requisition

Andrea Smith, a people and culture business partner at Salal Credit Union, described who ultimately owns a screening decision in a Talent Signal News interview. “I still want to keep those humans accountable for what needs to be done.” When the vendor changes the model, that accountability needs a name on the employer's side.

Record the versions, the evidence reviewed, the decision owner, and the effective date. When comparability remains uncertain, avoid pooling scores across versions. Options include finishing a cohort under the prior version if available or using a consistently administered supplemental assessment for affected applicants. Have the assessment owner review the plan before anyone rescales scores or changes a cutoff. Recruiters need an instruction they can apply to the person already waiting.

Scoring model FAQs

Can we compare candidate scores from before and after the update?

Only if there is evidence the versions support that comparison for the intended decision. Ask for a side-by-side distribution, rank changes, and decisions at your actual cutoff. Two versions can have the same average and still move different people into the interview group. Ask about score linking if the units have changed. Until comparability is established, label every score with its version and keep different versions out of a single leaderboard.

Can we keep the same passing score after the update?

Keep it when evidence supports the original interpretation and the job requirement it represents. Start by retrieving the rationale for the old threshold. Then examine what crossing it means under the new model, including examples just above and below it. In the hypothetical 72-to-68 case, check whether the changed result reflects a meaningful difference in job-relevant performance. Moving the cutoff until the old percentage passes can conceal a change in who qualifies.

What evidence should we request from the vendor before the new model goes live?

Request a change log, a comparison of decisions under both versions, and evidence that the updated scores support the hiring decision for your roles. “Please provide the version and effective date, affected roles and scores, changes to inputs or weighting, and the evidence supporting our intended use. Please include paired results across versions, movement at our cutoff, subgroup sample sizes and uncertainty where available, and options for in-flight candidates or rollback.” Assign an assessment specialist to review the response. If evidence is missing for a material change, document the gap and decide what alternative evidence recruiters should use in the interim.

FROM THE TALENT SIGNAL NEWS ARCHIVE
Human Review Closes The Gap Between A Match Score And A Qualified Candidate