AI skills now show up on nearly every job description, but "proficient in ChatGPT" on a resume proves nothing. Here's how talent teams can define AI fluency, test it with real tasks, and keep the process fair and defensible.
AI skills have become a standard line on job descriptions faster than most hiring teams have figured out how to measure them. Gartner predicts that by 2027, 75% of hiring processes will include certifications and testing for workplace AI proficiency. Employers are already moving in that direction. Shopify CEO Tobi Lütke told staff that "reflexive AI usage is now a baseline expectation," and Zapier now holds every new hire to a minimum AI fluency bar.
The requirement is clear. The measurement isn't. Most candidates now list AI tools on their resumes, and many of those resumes were written with AI. A line that says "proficient in generative AI" tells a recruiter almost nothing about whether someone can use these tools to produce better work, or whether they'll hand over a confident, polished, wrong answer.
Testing AI fluency well means treating it like any other job skill: define it in terms of observable behavior, give candidates a realistic task, score the result against a consistent standard, and keep a human in charge of the decision.
Why a resume can't show AI skills
Self-reported AI skills have the same weakness as every other self-reported claim: the candidate wrote it, and now an AI may have polished it. As AI-written applications flood hiring pipelines, the phrases recruiters would use to spot AI experience are the same phrases a model inserts to match the job description.
Hiring teams are already feeling the effects. In Criteria's 2026-2027 Hiring Benchmark Report, one in four hiring professionals said they've lost confidence in resumes as a reliable signal altogether. Forty-one percent said candidates are using AI to misrepresent themselves, and 35% said they can't reliably catch it.
That leaves recruiters with a familiar problem in a new form. Criteria Founder and CEO Josh Millet says the way employers evaluate talent was already changing before generative AI arrived, and AI has accelerated the shift. "It's shifting from a model where it was, 'Tell me why I should hire you,' to a new mode of, 'Show me why I should hire you,'" Millet says.
AI fluency fits that pattern. It shows up in how someone actually works with the tools, not in which tools they list. The only reliable way to see it is to have candidates do the work in front of you.
AI fluency goes beyond tool knowledge
Knowing how to open a chatbot and type a prompt is now a baseline skill. The skill that separates candidates is judgment: knowing when to use AI, how to direct it, and when not to trust what it produces.
Research shows why that distinction matters. In a field experiment with 758 Boston Consulting Group consultants, researchers from Harvard Business School and other institutions found that GPT-4 boosted speed and quality on tasks within the model's capabilities. On a task deliberately set outside those capabilities, consultants using AI were 19 percentage points less likely to reach the correct answer than those working without it. The researchers call this uneven boundary the "jagged technological frontier." An AI-fluent employee can tell which side of that frontier a task falls on.
Several frameworks now describe the skill in observable terms. Anthropic's AI Fluency Framework, developed with professors Rick Dakan and Joseph Feller, breaks it into four competencies: delegation (deciding what to hand to AI), description (communicating the task clearly), discernment (judging the quality of the output), and diligence (taking responsibility for the result). Zapier's hiring rubric scores candidates on AI mindset, strategy, building, and accountability. Its guiding principle: "With AI, you can delegate the work, but not the accountability."
For hiring teams, the practical definition is short: an AI-fluent candidate can use AI to do better work faster, and can explain and defend every part of what they hand in.
5 ways to test AI fluency in candidates
1. Start with the job, not the tool
AI fluency looks different for a plant scheduler, a bank analyst, and a customer service lead. Before building any test, identify two or three tasks in the role where AI already helps or soon will, and what a good result looks like for each. Avoid testing for a specific product. Tools change every few months, and a test tied to one tool ages as fast as the tool does.
2. Run an AI-assisted work sample
Give candidates a realistic task from the role and let them use AI to complete it, with a time limit and a short written explanation of their approach. Ask them to share the prompts they used or watch them work live. The finished product matters less than the process behind it. Zapier, which observes candidates using AI in real time during its skills test, puts it this way in its hiring guidance: "A rough result with strong reasoning and real iteration is a better signal than a polished one with no visible process behind it."
3. Ask candidates to find the errors
Discernment is the hardest part of AI fluency to fake and the most important to test. Hand candidates an AI-generated output relevant to the role, such as a summary, a forecast, a customer reply, or a piece of code, that contains planted mistakes: a wrong figure, an invented source, a recommendation that ignores a key constraint. Score how many errors they catch, how they verify them, and how they would fix the work. The jagged-frontier research suggests this is exactly where uncritical AI users fail.
4. Use structured, behavior-based interview questions
Interview questions add context a task can't, such as how a candidate has handled AI at work before. Ask every candidate the same questions and score answers against a defined scale, the core of structured interviewing. Useful prompts include: "Tell me about a time an AI tool gave you an answer you didn't trust. What did you do?" and "Describe a task you decided not to use AI for, and why."
A behaviorally anchored rating scale keeps scoring consistent. A low anchor might describe a candidate who used AI output as-is without checking it. A middle anchor describes someone who checked facts but couldn't explain their approach. A high anchor describes someone who verified the output, caught a meaningful problem, and changed their process as a result.
5. Test the reasoning underneath, without AI
AI fluency depends on skills that exist without the tool: critical thinking, problem solving, and the ability to learn quickly. Gartner predicts that through 2026, atrophy of critical-thinking skills due to generative AI use will push half of global organizations to require AI-free skills assessments. Validated pre-employment assessments of cognitive aptitude and problem solving give teams that baseline, provided they're protected against AI-assisted cheating. The combination shows both whether a candidate can think and whether they can think with AI.
Learning ability doesn't expire
Any specific AI skill a candidate has today may be out of date within a year. The tools change, the interfaces change, and the tasks change with them. Millet points to his own engineering team: "Three years ago, my software engineers were only writing code. Now they're basically editors of code."
He says the stronger long-term signal is a candidate's capacity to learn, rather than any fixed skill set. Millet recalls a platform rebuild that was done in a framework none of the 12 engineers on the project knew when they were hired. "We hired smart people who were good learners, who were intellectually curious. They learned these new languages as they came along."
The same applies to AI. As tools take on more of the routine tasks in a role, what remains is the durable skills like critical thinking, judgment and relationship building. A good AI fluency test captures how a candidate works with today's tools. A measure of learning ability and problem solving shows whether they'll keep up with next year's.
Keep AI skills testing fair and defensible
An AI fluency test is a selection procedure like any other, and the same standards apply. Under the federal Uniform Guidelines on Employee Selection Procedures, employers should be able to show that a test relates to the job and doesn't screen out protected groups without justification. A generic "AI quiz" bolted onto every role is hard to defend. A task built from the actual work of the role is much easier.
Access is a fairness issue, too. Candidates who pay for premium AI tools, or whose current employers provide them, will have more practice than candidates who don't. Providing the same tool to everyone during the assessment and telling candidates in advance which tools are allowed reduce that gap. Clear rules about when AI is permitted, and when it isn't, also keep an AI-assisted work sample from blurring into the cheating problem employers are already trying to solve.
Finally, the scoring itself should stay explainable, whether a person or software does it.
That principle matters more as AI hiring regulation spreads across states and countries. Teams that can explain what they tested, why it relates to the job, and how a person made the final call are in a much stronger position. TSN's guide to evaluating AI hiring tools covers the questions to ask vendors.
FAQ
What is AI fluency in hiring?
AI fluency is a candidate's ability to use AI tools to do better work: deciding what to hand to AI, directing it clearly, checking its output for errors, and taking responsibility for the result. It's different from tool familiarity. A candidate who knows many AI products but accepts their output without checking it is not AI-fluent.
How do you test AI skills in candidates?
The most reliable methods are an AI-assisted work sample based on a real task from the role, an exercise where candidates find and fix errors in AI-generated output, and structured interview questions scored on a behaviorally anchored rating scale. Pairing these with an assessment of problem solving and learning ability shows the reasoning underneath.
What AI fluency interview questions should I ask?
Ask behavior-based questions that reveal judgment, such as: "Tell me about a time an AI tool gave you an answer you didn't trust. What did you do?" "Describe a task you chose not to use AI for, and why." "Walk me through how you checked the accuracy of something you produced with AI." Ask every candidate the same questions and score answers against defined anchors.
Will AI proficiency testing become standard in hiring?
Gartner predicts that by 2027, 75% of hiring processes will include certifications and testing for workplace AI proficiency. Companies including Zapier and Shopify have already made AI fluency a baseline expectation for employees.
Should candidates be allowed to use AI during assessments?
It depends on what the assessment measures. Tests of AI fluency should allow, and observe, AI use. Tests of underlying abilities such as cognitive aptitude or problem solving should be AI-free and protected against cheating. Tell candidates the rules in advance and apply them consistently.
Is testing for AI skills legal?
Yes, provided the test relates to the job and is applied consistently. Under federal guidelines, selection procedures that disproportionately screen out protected groups must be job-related. Employers should also make sure candidates have equal access to any tools the assessment requires.
