Testing Recruiting ATS Export Completeness with a Reconciliation Sample

A recruiting report can be internally consistent and still omit the records that matter. Its percentages add up, every row has a candidate identifier, and the spreadsheet contains no obvious blanks. Yet the export may exclude archived applications, hide records owned by another team, or flatten several stage events into one current-state row. Before an agency relies on the report, it needs evidence that the export represents the population its owner intended to measure.

This research protocol evaluates export completeness and meaning. It does not rank applicant tracking systems or report findings from an agency dataset. The examples below are synthetic. The proposed method can be used in a controlled pilot, with a system owner approving access and a reporting owner defining the population. A virtual assistant may prepare comparisons and exception records; those owners retain responsibility for the conclusions and any system changes.

Research question and unit of analysis

Ask whether a specified export contains the applications and events required by a specific reporting definition at a named cutoff. The unit might be an application, a candidate, or a stage transition. Choose one primary unit and document its relationship to the others. A file with one row per candidate cannot directly answer a question that requires one row per application without additional information.

Define the target population before extracting the file. For example, it might include every application associated with approved requisitions during a quarter, including applications later withdrawn or archived. A report limited to currently active candidates would not represent that population, even if its title says “quarterly pipeline.” The system label is not a substitute for the research definition.

Record the fields required to support the intended measure. A stage-conversion analysis needs event or transition evidence; a current workload count may need only current stage and ownership. The protocol should not demand every available field merely because completeness sounds desirable. Additional personal information increases handling burden without necessarily improving the analysis.

Construct an independent comparison inventory

The comparison inventory should come from an approved source that does not simply repeat the same export query. Depending on the system, this may be a requisition-level application list, a separately scoped administrative report, or documented test records. Explain how independent it actually is. Two buttons that invoke the same underlying filter do not provide independent confirmation.

Include known positive cases that should appear and known negative cases that should not. Positive cases can include an active application, an archived application within scope, a transferred application, and an application with multiple stage movements. Negative cases might belong to another client or fall outside the chosen period. Use synthetic records where practical and avoid introducing real candidates solely for testing.

Freeze the comparison inventory and export close together in time. If candidates move between stages while the files are collected, a difference may reflect timing rather than omission. Preserve both extraction times and identify records that changed between them. Do not force a match by deleting the inconvenient records from the comparison set.

Stratify around plausible failure mechanisms

A purely convenient sample of the first twenty rows may miss the very conditions most likely to cause omissions. Instead, define strata around the export's boundaries: active versus archived, current versus former owner, single versus multiple applications, recent versus older events, and records with restricted visibility. The reporting and security owners should decide which strata can be inspected lawfully and operationally.

Within each approved stratum, choose a reproducible selection method and retain the selected identifiers in a restricted test record. If selection is purposive because the goal is to exercise known edge cases, say so. A purposive diagnostic sample cannot support a statistically representative estimate of the agency-wide error rate.

Do not describe the absence of observed defects as proof that the entire export is complete. A small sample may establish that specific test cases behave correctly while leaving other cases untested. Report the sample design, population size if known, cases reviewed, and the conditions not covered. If management needs a population estimate, have a qualified analyst design the sampling and uncertainty calculation.

Compare membership before comparing values

First determine whether each expected unit is present, absent, duplicated, or not matchable. Use stable identifiers approved by the system owner. Names alone can create false matches, and candidate IDs may not distinguish multiple applications. Preserve the mapping between person, application, requisition, and event where the question requires it.

Only then compare required field values. An application present with the wrong stage is a different defect from a missing application. A blank date can mean unknown, not applicable, access-restricted, or lost during export. Record the observed condition and ask the owner to establish the cause. Treating every blank as the same error would make the findings less useful.

For each difference, retain a minimal evidence reference: expected membership or value, observed export result, relevant filter settings, and the responsible review owner. Keep candidate documents out of the general findings report. The FTC's information-protection guidance supports restricting access and retaining only information needed for the task.

Use a synthetic reconciliation example

Suppose the defined inventory contains 80 applications across ten requisitions. The export has 76 rows. Investigation finds six in-scope archived applications missing and two duplicated current applications. The net difference of four rows initially concealed eight record-level defects. Comparing totals alone would have understated the problem and left the duplicated applications unidentified.

Now suppose another three records changed ownership between the two extraction times. Keep those timing exceptions separate while reconstructing the relevant cutoff. They are not automatically export defects. The final reconciliation should distinguish confirmed omissions, confirmed duplicates, timing differences, and unresolved cases, even if that makes the result less tidy.

Test the corrected configuration against the same diagnostic records and a fresh approved sample. Reusing the original cases checks whether the identified defect was repaired. A fresh sample checks whether the repair introduced an unexpected effect elsewhere. Neither step justifies expanding access beyond the agreed population.

Evaluate history and repeated events separately

An export may include every application while still losing event history. For a selected application with several stage transitions, compare the sequence against the authorized event view. Check whether repeated entries into a stage remain distinct and whether corrections are distinguishable from original events.

Document which timestamp each column represents: occurrence, entry, update, or export time. An analyst who mistakes synchronization time for interview time can produce a plausible but misleading response-speed measure. The assistant should preserve the field definition and flag ambiguity rather than choosing the interpretation that makes the chart easiest to complete.

The BLS JOLTS definitions provide an example of explicit units and reference periods in labor-market statistics. This protocol does not turn an agency export into a JOLTS measure. It applies the narrower lesson that a count is interpretable only when the underlying event and observation window are defined.

Report findings by consequence

Organize the results around the measures affected. Missing archived applications may distort historical conversion rates. Duplicate applications may inflate current workload. Missing event timestamps may prevent a time-based analysis altogether. A single overall data-quality score can hide those different consequences and should not replace the defect table.

For each affected measure, state whether it can be used as defined, used with an explicit limitation, or held pending repair. The reporting owner makes that decision after reviewing the evidence. An assistant should not publish a corrected-looking dashboard while the population remains unresolved.

Document the approved repair and rerun result. If the export cannot supply the required history, the appropriate conclusion may be that another data source or a different reporting question is needed. Do not manufacture historical events from current state to satisfy the original request.

Limitations and operating handoff

This method can reveal omissions and interpretation defects within its tested scope. It cannot prove every record is accurate, establish that a hiring process is fair, or attribute business outcomes to an agency. Access restrictions may also prevent direct comparison of some records; those limits belong in the findings rather than being silently removed from the denominator.

Retain the population definition, extraction settings, timestamps, sample selection rule, reconciliation results, and owner decisions under the organization's retention policy. The public report can summarize counts without exposing candidate identifiers. Temporary working files should follow the approved disposal process after the evidence is accepted.

For administrative support, review recruiting reporting preparation and ATS pipeline management. A workflow consultation can help define the extracts and handoff, while the system and reporting owners retain control of access and interpretation.

Sources

1. Federal Trade Commission: Protecting Personal Information, A Guide for Business.
2. Bureau of Labor Statistics: JOLTS Data Definitions.

Frequently Asked Questions

Does a matching row count prove an export is complete?

No. Omissions and duplicates can offset each other. Compare membership using the correct stable identifiers before interpreting the total.

Can a diagnostic sample estimate the overall defect rate?

A purposive set of edge cases cannot provide a representative rate. A qualified analyst must design an appropriate probability sample and uncertainty method when that estimate is required.