How to Evaluate AI Hiring Tools: 8 Questions to Ask Vendors

More than half of hiring teams now use AI somewhere in their process, and employers carry the legal and reputational risk when those tools get it wrong. Before buying, employers should press vendors on whether the tool predicts job performance, whether it has been independently audited for bias, how its scores are explained and where people stay in charge of decisions. Governance, fraud safeguards, limits on the data the AI evaluates and the candidate experience deserve the same attention. A vendor that cannot answer these questions in writing carries risk regardless of how well its product demos.

AI is now a standard part of the hiring stack. Fifty-six percent of hiring professionals use it somewhere in their process, according to Criteria's 2026-2027 Hiring Benchmark Report, most often to screen candidates, run assessments and schedule interviews. Seventy-one percent are optimistic about the technology's role in hiring.

The legal exposure has grown just as quickly. Under Title VII, an employer is responsible for the selection tools it uses whether it built them or bought them, and courts have started testing how far that responsibility extends to vendors. In 2024, a federal judge in California allowed discrimination claims against an HR software provider to proceed on the theory that it acted as an agent of the employers using its screening tools, and in 2025 the same court certified a nationwide age-discrimination collective action. State and local rules have multiplied as well. New York City requires annual independent bias audits of automated hiring tools, Illinois has prohibited AI that has a discriminatory effect since January 2026, California now requires employers to keep automated-decision data for four years, and Colorado's revised AI law takes effect in January 2027.

That combination makes vendor evaluation one of the most consequential decisions an HR team makes. Chris Daden, chief technology and AI officer at Criteria, has asserted that "AI in hiring must be held to a higher bar than AI in other domains." The eight questions below give employers a way to hold vendors to that bar.

Eight questions to ask AI hiring vendors

1. What evidence shows the tool predicts job performance?

A hiring tool is only useful if its outputs relate to how people actually perform once hired. Ask for validation studies that connect scores to job outcomes such as supervisor ratings, productivity, sales or retention, and ask how many people and which kinds of roles the studies covered. Good vendors can explain the type of validity evidence they rely on, describe how the tool compares with established methods, and offer to run a local validation study using your own employees' results.

The research on hiring methods gives a useful benchmark. A 2022 re-analysis by Paul Sackett and colleagues found that structured interviews, job knowledge tests, work samples and cognitive ability tests all predict performance far better than unstructured interviews or years of experience. Any tool under consideration should be able to show that it outperforms a resume screen.

2. Has the tool been independently audited for bias?

Ask for the most recent bias audit, who performed it and how independent the auditor was. New York City's Local Law 144 is a reasonable model. It requires an annual audit by an independent party that calculates impact ratios by sex, race and ethnicity, including intersectional groups, and it requires employers to post a summary of the results. Employers should also understand how the vendor applies the four-fifths rule from the federal Uniform Guidelines on Employee Selection Procedures and whether it monitors adverse impact continuously or only once a year.

Criteria, which sponsors Talent Signal News, reports that an independent audit completed in June 2026 covered more than 300,000 candidates over five years using the data standards required by Local Law 144 and found no practically significant disparate impact by gender or race. Audits capture a single point in time, and Daden has cautioned that a Local Law 144 audit "does not certify that a tool is free of bias," so ongoing monitoring matters as much as the audit itself.

3. Can the vendor explain how scores are produced?

Hiring managers, candidates and regulators increasingly expect an explanation of why a tool reached its result. Ask what inputs the model uses, how those inputs map to job-related skills or traits, and what a hiring manager will see when reviewing a score. Colorado's revised AI law, which takes effect January 1, 2027, will require employers to give candidates a plain-language explanation of an adverse decision and the tool's role in it, according to international law firm Seyfarth Shaw. Tools whose scoring cannot be explained will make that requirement difficult to meet.

4. Who makes the final hiring decision?

Ask where human judgment sits in the workflow and whether the tool can automatically reject candidates without review. Many employers prefer tools that rank, score or flag candidates for a person to evaluate. "AI should not make a complete hiring decision," Daden wrote. Colorado's law will also give candidates the right to request human review of an adverse decision.

5. How does the vendor govern its AI?

Governance covers how a vendor builds, tests, monitors and updates its models over time. Ask whether the vendor follows a recognized framework such as the NIST AI Risk Management Framework or holds certification to ISO/IEC 42001, the international standard for AI management systems published in 2023. Ask how often models are retrained, how changes are tested before release and how customers are notified. Criteria announced its ISO 42001 certification in May 2026. "We don't just test our models once and deploy them," Daden wrote at the time.

6. How does the tool protect against candidate fraud?

About four in ten hiring professionals in Criteria's benchmark survey said they're struggling with candidates who use AI to misrepresent themselves, and more than a third said they can't reliably detect it. Ask what safeguards the tool uses, such as timed formats, large question banks, identity verification and AI proctoring that flags outside help or a second person. Ask how flagged incidents are reported and reviewed, and how candidates are told about monitoring in advance.

7. What data does the AI evaluate?

The inputs a tool scores determine much of its risk. Systems that analyze facial expressions, voice or video backgrounds raise more questions about bias and accuracy, and the EU AI Act already prohibits emotion recognition in the workplace, including during recruitment. Ask exactly which data points the model uses, where candidate data is stored, how long it is kept and who can access it. Criteria's interview scoring, for example, evaluates only the transcript of what a candidate said. Employers in California should also confirm the vendor can support the state's requirement to retain automated-decision data for at least four years.

8. What will candidates experience?

Candidate experience affects whether qualified applicants finish the process at all. Ask how long the process takes, whether it works on mobile devices, how accommodations for candidates with disabilities are handled and what candidates are told about the use of AI. Disclosure is also becoming a legal requirement. New York City requires notice at least 10 business days before an automated tool is used, and Illinois requires employers to notify candidates when AI is used in hiring decisions. Most candidates are open to a well-designed process. Sixty-eight percent of job seekers in Criteria's 2026 Candidate Experience Report said they would be willing to drop the resume in favor of science-based assessments and structured interviews.

Red flags during vendor evaluation

Some answers should slow a purchase down. Be cautious if a vendor describes its validation evidence only in general terms or cites "proprietary research" it won't share, if it can't say when its last bias audit took place or who conducted it, or if it describes its model as too complex to explain. Contracts deserve the same scrutiny. Colorado's new law voids clauses that try to indemnify a party against its own discrimination liability, and in general a vendor's terms do not shift an employer's legal responsibility for its hiring decisions. Claims that a tool is "bias-free" or "100% accurate" are also a warning sign, because no hiring method can honestly promise either.

How to run a pilot

A structured pilot gives employers evidence before they commit. Start with one or two high-volume roles where outcomes are easy to measure, and agree with the vendor in advance on what success looks like, such as better quality of hire, lower early turnover or a shorter time to hire. Run the tool alongside the current process for long enough to collect performance and retention data on the people hired. Then compare selection rates across demographic groups using the four-fifths rule and ask the vendor to walk through any gaps.

The results can be significant when the tool is a good fit. After Heartland ECSI added Criteria's assessments to its call center hiring, turnover fell from about 75% to 26% and 90-day retention rose from 50% to 91%. A pilot shows whether results like those hold up in a specific organization.

For background on how the main types of hiring tests work and what the research says about each, see Talent Signal News' guides to pre-employment assessments and structured interviews.

FAQ

What should I look for in an AI hiring tool?

Start with evidence that the tool predicts job performance and an independent bias audit with ongoing monitoring. Beyond that, look for scores hiring managers can explain, human review of decisions, recognized AI governance such as ISO 42001 certification, safeguards against candidate fraud, clear limits on the data the AI evaluates and a transparent, accessible experience for candidates.

Are employers liable if a vendor's AI hiring tool discriminates?

Yes. Under Title VII and state anti-discrimination laws, employers remain responsible for the selection tools they use, including tools built by vendors. Courts have also begun to hold that vendors can share liability in some circumstances, and California's regulations treat vendors acting on an employer's behalf as agents.

What is a bias audit for AI hiring tools?

A bias audit measures whether a tool's outcomes differ across demographic groups. New York City's Local Law 144 requires an annual audit by an independent auditor that calculates impact ratios by sex, race and ethnicity, and employers must publish a summary of the results before using the tool.

Which laws regulate AI in hiring in 2026?

Key rules include New York City's Local Law 144, Illinois's amendment to its Human Rights Act (effective January 2026), California's automated-decision system regulations (effective October 2025) and Colorado's revised AI law (effective January 2027). In Europe, the EU AI Act classifies hiring AI as high-risk, with those obligations now scheduled to apply from December 2027. Federal anti-discrimination law, including Title VII, applies everywhere in the United States.

What is ISO 42001?

ISO/IEC 42001 is an international standard, published in 2023, that sets requirements for establishing, maintaining and improving an AI management system. Vendors certified to the standard have had their processes for building, testing and monitoring AI reviewed against it.