How to Evaluate AI-Assisted Screening in a Recruiting Agency

An agency that says it uses artificial intelligence in sourcing or screening is not making one operational claim. It may mean search expansion, résumé parsing, ranking, chatbot questions, interview transcription, or a recommendation that a recruiter never independently checks. Those uses have different risks and should not be compared as one feature.

This guide separates public evidence from buyer analysis. NIST's AI Risk Management Framework is voluntary and not employment law. The EEOC's employment-testing guidance explains the importance of job-related selection procedures and appropriate administration. The decision framework below is editorial guidance for comparing agencies; it is not a legal opinion or a guarantee that a process is compliant.

Start with the decision map

Ask an agency to map every automated step to the hiring decision it influences. A useful map has five columns: input, transformation, output, human owner, and candidate consequence. For example, a résumé parser may transform a document into structured fields, while a ranking model may turn those fields into a recommended order. The recruiter, not the software label, should be identified as the owner of the next decision.

Stage	Evidence to request	Control to test
Sourcing	Channels, filters, exclusions, refresh date	Does the search reproduce the role brief?
Parsing	Fields extracted and discarded	Can a recruiter correct a parsing error?
Ranking	Signals and thresholds at a high level	Is ranking advisory or a gate?
Outreach	Message logic and suppression rules	Can candidates opt out and be contacted only once?
Review	Human review and escalation record	Can the employer see why a person advanced or did not?

NIST describes four functions - govern, map, measure, and manage - and emphasizes that risk work continues across the system lifecycle. That is a useful procurement test: a one-time vendor demo is not evidence of ongoing control. Ask who reviews changes to the model, prompts, data sources, and thresholds.

Test role-specific validity

Generic accuracy claims are weak evidence for an agency search. A model that finds common keywords may perform differently for a construction superintendent, a nurse manager, or a confidential executive search. Give each agency the same redacted brief with must-have qualifications, acceptable equivalents, disqualifiers, location constraints, and evidence expectations. Compare outputs using a pre-agreed rubric.

The rubric should record false negatives, false positives, missing context, and recruiter corrections. Do not treat a score as truth. A candidate may be ranked lower because a résumé uses a different term, because a credential is described in another format, or because the system cannot interpret a career transition. The agency should show how those cases are reviewed rather than claiming the model removes judgment.

Measure fairness and explainability without overclaiming

Ask what groups, outcome definitions, time windows, and denominators appear in any bias or adverse-impact analysis. A single overall pass rate cannot establish fairness. Results can change when the comparison group, role family, stage, or missing-data treatment changes. Require the agency to explain what it measured, what it did not measure, and whether the test population resembles the proposed search.

The Uniform Guidelines provide a framework for evaluating selection procedures. The EEOC's employment-testing guidance also explains that using a vendor's documentation does not remove the employer's responsibility to ensure a procedure is appropriate for its intended purpose. An agency may help operate a process, but the hiring company should not assume the agency owns every legal or evidentiary obligation. Put responsibilities in writing and have counsel review the arrangement where appropriate.

Require a human correction path

Human review is meaningful only when the reviewer can disagree with the tool, has enough information to investigate, and records the reason. Ask whether recruiters see the underlying résumé and brief, whether candidates can be reconsidered, and whether a rejected recommendation is final. An approval click is not independent review if the reviewer cannot see the relevant evidence or is measured only on throughput.

Use a small pilot with a fixed sample, no automatic rejection, and a stop rule for material errors. Compare the tool-assisted process with the agency's ordinary process using the same role brief. Track time saved, candidates surfaced, candidates missed, correction rate, candidate complaints, and unresolved incidents. Those are operational measures, not proof of legal compliance.

Contract and data questions

Before sharing candidate information, specify the agency's purpose, permitted data, access roles, retention period, subprocessors, security incident notice, deletion process, and use of data for model training. Clarify whether the agency can reuse a candidate's résumé for another client and how consent or notice is handled. If the agency uses a third-party platform, identify who can answer a candidate's access or correction request.

Document the change-control path. A materially different model, data source, ranking threshold, or automated message should trigger notice and, where needed, a new evaluation. Preserve a versioned record of the brief, test set, result, reviewer, and decision. This makes a later review possible without pretending that a dashboard is a complete audit trail.

Decision rule and limitations

Prefer the agency that can explain the workflow and its limits over the agency with the most impressive automation vocabulary. Proceed when the tool's role is bounded, the evidence is role-specific, humans can correct it, candidates are treated consistently, and responsibilities are explicit. Pause when the agency cannot name the data source, cannot describe a human override, refuses to discuss errors, or presents a universal accuracy claim.

This framework does not independently test any vendor or establish that a particular tool is lawful. Laws and agency guidance can change, and state or local requirements may add obligations. Recheck the applicable rules for the worksite and ask qualified counsel to review a high-impact deployment.

A controlled test of ranking behavior

Use a synthetic set of applications built around the actual role requirements, with the employer's assessment specialist approving the cases. Include a clearly supported qualification, an equivalent qualification expressed differently, an ambiguous claim, and a missing qualification. These cases test different failure modes. They should not be populated with real candidate information merely because that is easier to obtain.

Have reviewers assess the cases before they see the tool's order. Preserve both the independent review and the tool output. If reviewers see the ranking first, the comparison may measure agreement with the recommendation rather than independent evaluation. Ask the agency to explain every disagreement using the underlying evidence and the role rubric. A numerical score without an explanation does not resolve the discrepancy.

Repeat a small subset with irrelevant formatting changes, such as moving an unchanged qualification to another section. A changed recommendation is a reason to investigate sensitivity, not automatic proof of discrimination or invalidity. The purpose of this pilot is to discover questions that require a stronger evaluation. It cannot establish accuracy across all roles or replace a qualified validation study.

Predefine the consequences of a failed case. For example, if the system ignores an explicitly accepted equivalent qualification, keep that use in advisory mode until the owner explains and corrects the behavior. Do not compensate by quietly changing the test set. Preserve the original case and compare the corrected configuration against it.

Separate parsing defects from ranking decisions in the result table. A parser that loses a credential and a ranking rule that discounts an accurately extracted credential require different repairs. Assign the technical owner for the former and the assessment owner for the latter. A recruiting assistant can maintain the case references and reviewer notes while those owners decide what the result means.

This protocol generates local evidence about a bounded use case. It does not justify marketing the agency as unbiased or the tool as universally reliable. Discuss candidate research support to keep administrative evidence gathering separate from screening decisions, or bring a workflow to a consultation.

Follow a missed candidate through reconsideration

Consider a fictional search for a maintenance supervisor where the brief accepts either a named certification or documented equivalent experience. A synthetic application describes the experience clearly but receives a low recommendation because the parser places it in an unrelated field. The agency's demonstration should follow this case through correction, reranking, and recruiter reconsideration. Showing that an administrator can edit a field is insufficient if the candidate remains outside the recruiter's review queue afterward.

Record three separate outcomes: whether the evidence was captured correctly, whether the recommendation changed, and whether the recruiter received the corrected case. If the first two succeed but the third fails, the operational defect sits in the handoff. Ask the agency to identify which pending decisions could have used the earlier output and how the authorized owner would locate them. Keep the pilot fictional so this exercise does not expose or alter real applications.

Then repeat the original case against the corrected configuration without changing its qualification language. Preserve the unsuccessful output alongside the new result. This gives the buyer a concrete example of recovery and prevents a successful demonstration from erasing the initial failure. The final procurement note should describe the tested role, failure, repair, and remaining uncertainty; it should not turn one recovered application into a general accuracy claim.

Sources

1. NIST, Artificial Intelligence Risk Management Framework 1.0 - checked October 2, 2026.
2. NIST, AI Risk Management Framework Core - checked October 2, 2026.
3. EEOC, Employment Tests and Selection Procedures - checked October 2, 2026.
4. eCFR, 29 CFR Part 1607: Uniform Guidelines on Employee Selection Procedures - checked October 2, 2026.

Frequently Asked Questions

Does human review make an AI screening process safe?

No. It can reduce risk only when the reviewer has authority, evidence, time, and a documented correction path. It does not replace validation or legal review.

Should an agency disclose its model source code?

Usually the buyer needs an understandable process description, test evidence, access controls, and change notices rather than source code. A sensitive search may justify additional technical diligence.

Can an agency reuse candidate data to improve its system?

Do not assume so. State permitted uses, notice, retention, deletion, subprocessors, and any model-training use in the agreement after privacy and employment-law review.