Validating Work Samples in an Agency-Led Hiring Process

A work sample can make a recruiting process more concrete: a candidate analyzes a real-like dataset, drafts a client update, diagnoses a technical issue, or plans a project handoff. It can also become an unstructured audition that rewards spare time, expensive tools, or familiarity with the evaluator rather than the capability the role requires.

This article uses the EEOC Uniform Guidelines and the Department of Labor's occupational information as evidence boundaries. The suggested rubric is an editorial tool, not a validity study or legal conclusion. Employers and agencies should review applicable law, collective rules, accessibility needs, and job-specific evidence before deployment.

Begin with the job task

Write the job task before choosing the exercise. Name the outcome, constraints, tools normally available, collaboration expected, quality threshold, and time horizon. A sample should represent important work without requiring unpaid production for the employer. If the role rarely produces a polished presentation, do not make presentation polish the main score.

Design field	Decision to record	Failure it prevents
Construct	Capability being assessed	Vague “culture fit” scoring
Task	Realistic output and constraints	Irrelevant puzzle solving
Materials	Permitted tools and information	Unequal preparation
Time	Expected duration and deadline	Candidate burden differences
Rubric	Observable anchors	Personal preference scoring
Reviewer	Trained owner and escalation	Uncontrolled interpretation

The agency should explain how each criterion connects to a duty in the brief. A criterion with no job connection is a candidate-experience and validity warning.

Standardize administration

Comparable candidates should receive the same instructions, materials, time window, submission channel, and permitted assistance, subject to reasonable accommodation. Record the version of the exercise and any clarification sent to one candidate. If the exercise changes mid-search, decide whether earlier candidates need a fair re-evaluation.

The recruiter should not coach one candidate on what “the client really wants” while withholding the same information from others. If a hiring manager answers a question, publish the clarification to the whole cohort. That is basic process control and makes later review possible.

Score observable evidence

Use a short rubric with behavioral anchors. For a search analyst, “identifies the decision-relevant data and states two limitations” is more useful than “strong analytical thinking.” Require reviewers to cite the candidate's work, not merely assign a number. Separate must-have failure conditions from weighted preferences and document who may override a threshold.

Run a calibration round before live use. Have two reviewers score the same sample independently, compare differences, and revise the rubric where disagreement reflects unclear instructions. Calibration does not prove validity, but it can expose a rubric that measures evaluator style.

Connect the sample to outcomes carefully

The EEOC guidelines describe validation concepts and caution that a selection procedure should be related to the job and used responsibly. Do not claim that an exercise predicts performance merely because hiring managers like it. Track whether candidates advance, are hired, complete onboarding, and perform against a job-relevant outcome, while considering sample size, role mix, censoring, and confounding.

An agency may be able to report process measures - completion rate, time to review, score distribution, rework, candidate withdrawals - but those are not the same as job-performance validity. Ask for the denominator and time window for every claim. If the agency has no evidence beyond anecdotes, label the exercise as a structured decision aid rather than a proven predictor.

Include accessibility and candidate cost

Ask whether candidates can use assistive technology, request additional time, or complete the task in an equivalent format. Consider internet access, scheduling, language, caregiving, and whether the exercise requires software a candidate cannot reasonably access. A shorter sample with a clear purpose may be more informative than a multi-hour assignment that filters for availability.

Tell candidates how the work will be used, who will see it, how long it will be retained, and whether it becomes the employer's property. Do not reuse candidate work for a client deliverable without explicit authorization and appropriate compensation or terms. Provide a contact for technical and accessibility questions.

Agency diligence and handoff

Ask a prospective agency to show the exercise, rubric, reviewer training, candidate message, accommodation path, data controls, and reporting template. Give every agency the same role brief and score the answer for relevance, standardization, burden, fairness, evidence, and ownership. Verify that the client, not the agency alone, approves the final exercise.

The handoff from recruiter to hiring manager should include the candidate's consented submission and the work-sample record, not informal opinions that bypass the rubric. Keep identity and evaluation data appropriately protected. A candidate should not have to repeat an exercise because the agency failed to transfer the version or score.

Limitations and decision rule

Proceed when the sample mirrors real work, administration is standardized, scoring is anchored, accommodations are available, candidate use and retention are clear, and outcome claims are modest and evidenced. Pause when the exercise is long, speculative, unpaid production; the rubric is secret; reviewers cannot explain scores; or the agency markets a preference as predictive science.

No exercise is universally valid. Job duties change, scores can be affected by instruction quality and reviewer behavior, and small cohorts limit inference. Revalidate when the role, tools, location, or evaluator group changes materially.

Examine reviewer disagreement before setting a cutoff

Consider a fictional exercise asking a recruiting analyst to reconcile a pipeline report. One reviewer values a polished chart, while another notices that the candidate excluded unresolved applications from the denominator. If the role requires accurate reporting, the disagreement may reveal an unclear rubric rather than a difference in candidate ability. Ask both reviewers to identify the exact work evidence behind their scores before discussing an overall recommendation.

Create a calibration table with one row per criterion, each reviewer's independent score, the cited evidence, and the resolution. Preserve the initial scores. Replacing them with a consensus score would hide how much interpretation the rubric required. The assessment owner can then revise an ambiguous criterion and decide whether the same clarification must be applied to earlier candidates.

Test the instructions as well as the scoring. Give a designated pilot participant the exercise without verbal coaching and ask them to identify unclear assumptions after completion. If the expected output depends on a hidden convention that only insiders know, either state that convention or establish why knowledge of it is actually job-related. Do not treat an avoidable instruction gap as a candidate deficiency.

A cutoff should follow the intended use and supporting evidence, not a round number chosen after looking at favored candidates. Document whether the exercise informs a discussion, contributes to a combined assessment, or controls progression. Those uses have different consequences, and a diligence review should not accept the same evidence claim for all of them.

Track administration exceptions separately from performance. A corrupted source file or interrupted session can invalidate comparison even when the rubric is sound. The recruiter decides whether a comparable opportunity can be restored. An assistant can preserve version references, delivery times, and the approved recovery action without interpreting the candidate's ability.

Interview scheduling support can coordinate the approved logistics and reporting preparation can organize review records. Bring a sanitized exercise to a workflow consultation to define those administrative tasks while the qualified assessment owner retains design and scoring authority.

Interpret missing exercises separately from weak performance

Suppose a fictional agency invites twelve candidates to complete a short scheduling exercise. Eight return usable submissions, two withdraw before opening the materials, one reports a damaged attachment, and one does not respond. An average score for the eight usable submissions describes those submissions only. It does not describe the capability of all twelve invited candidates, and assigning zero scores to the other four would mix performance with withdrawal, technical failure, and unknown status.

Prepare an administration ledger before the reviewer receives the scoring packet. Record invitation, material version, delivery confirmation where available, approved deadline, exception status, and usable submission status. Keep the reason for an accommodation or sensitive personal explanation outside the general scoring packet unless an authorized owner determines it is necessary. Reviewers need enough context to interpret the evidence without acquiring unrelated candidate information.

For the damaged attachment, ask the assessment owner to decide whether to supply the same materials again or offer an equivalent approved opportunity. Record the decision before reviewing the candidate's eventual score. A recovery arrangement should not quietly become extra coaching, a different task, or an unrecorded extension. If comparability cannot be established, mark that limitation rather than forcing the result into the ordinary ranking.

When reporting the exercise, show invited candidates, usable submissions, unresolved administration cases, and scored submissions separately. Explain which denominator supports each completion measure and which candidates enter the score summary. A small fictional pilot can reveal burdens or broken instructions, but it cannot establish predictive validity from these counts. The buyer's next decision is whether the administration process is ready for a larger, appropriately designed evaluation, not whether the highest score automatically identifies the best employee.

Sources

1. eCFR, 29 CFR Part 1607: Uniform Guidelines on Employee Selection Procedures - checked October 2, 2026.
2. EEOC, Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines - checked October 2, 2026.
3. U.S. Department of Labor, O*NET OnLine - checked October 2, 2026.
4. U.S. Department of Labor, Employment and Training Administration, O*NET Content Model - checked October 2, 2026.

Frequently Asked Questions

How long should a work sample be?

There is no universal duration. Set the shortest window that reveals the defined capability, then test completion burden and accessibility before launch.

Can candidates use AI or outside help?

State the permitted tools and evaluate the work the role actually requires. If tool use is part of the job, assess judgment and verification rather than pretending the tool does not exist.

Does a consistent rubric eliminate bias?

No. It can reduce avoidable variation, but the task, criteria, reviewers, accommodations, and outcome evidence still require review.