Key Takeaways
- What an AI recruiting experiment is: A structured, hypothesis-driven test where recruiting teams compare AI tools or workflows against a defined standard of “good” (such as a job post audit) to measure real output quality, not assumptions or hype.
- Tools tested: Anthropic Claude, OpenAI ChatGPT, and Google Gemini – all given the exact same prompt and job description to generate a marketing job post.
- Outcome: Gemini produced the strongest and most accurate job post, Claude hallucinated details (like location), and ChatGPT leaned heavily on generic buzzwords without strong specificity or signal.
How To Conduct An AI Recruiting Experiment
As a job post writing expert, I decided my first big AI recruiting experiment should be a battle of the LLMs. Which tool can write the best job post: Claude, ChatGPT, and Gemini. The test? If I gave them all the exact same prompt and information, which one would spit out the best job post according to my job post audit? I built this prompt without my expert hat on to simulate how I know this process would go in a corporate environment if the hiring manager was in charge of the first draft. They would probably give very little intel, reference an old job description, and expect brilliance.
Here’s the prompt: Using this job description, write a marketing job post that will attract the right candidate.
You can see the results from each model, the original job post, and my take on each of them in this longer video.
Here’s the TL;DR version: Gemini wrote the best draft with the same information the others had. Claude actually hallucinated different details into the job post like location and ChatGPT was a buzzword bingo winner.
| LLM Model | Strengths/Weaknesses |
| Gemini | – Strongest breakdown of required skills with clearer structure and prioritization – Frames responsibilities as examples, making expectations easier to interpret – Communicates scope and scale effectively within the role – Lists tools appropriately, though lacks deeper context or explanation of usage |
| Claude | – Includes more specific contextual detail, but omits key basics like full-time vs part-time status – Well-written and polished in tone, but lacks clarity and complete role framing – Introduces hallucinated or unsupported details not present in the original input |
| ChatGPT | – Relies heavily on common AI-generated content patterns – Uses broad buzzwords without grounding them in concrete role context – Overly verbose, with long sections and repetitive bullet structures |
Why Experimentation is Mandatory for Recruiting
This kind of experimentation is what recruiting teams should be doing – creating hypotheses and testing to get results. Instead, a lot of people are still doing the same things with the same tools. I know, I see all the headlines you do. But when I actually talk to recruiters? The majority of their work looks a lot like it did a year ago, with a copilot writing first drafts of emails and tools that help them screen resumes for quality and authenticity.
They are the definition of doing more with less. Applications are up 95% and team sizes shrink quarterly. There’s no time or incentive for experimentation. That’s an easy formula to make your team complacent, not curious, about technology.
This complacency is only exacerbated by all the “AI is going to kill your job,” headlines. I surely would not be at all interested in using a tool that threatens my paycheck every week. Would I do it to keep my check? Yes. Would I do it well? For myself, not at work. Anthropic reported in December 2026 that 69% of people experiment with AI, but don’t tell anyone at work.
But, that’s the nature of people who are facing mass uncertainty. There’s simply too many distractions. It’s easier – emotionally and literally – to just keep doing the same thing the same way until one day, you leave.
How to Set Up an AI Recruiting Experiment in 3 Steps
Where I hear most people getting stuck is they don’t have time to even figure out where to start. I mean, is taking an afternoon off to poke around and quit an hour in to check email a real recruiting experiment? No.
So. I built this quick exercise to frame up a simple, results-oriented AI recruiting experiment. I recommend this model for operational tasks that have a right answer – the “if this, then that” category (at least with the AI capabilities currently available to us).
- Identify a problem that impacts your workflow. So, in the job post example, the problem is that posting generic job posts are driving generic results. Remember, everything people post about on LinkedIn is not worth doing. Pick an actual problem that wastes your time. I would suggest things you’re doing really manually like adding an invite to your calendar or Googling a company name to check what industry they are in every time you get a new resume.
- Define what good means. Literally, what are the benchmarks that tell you this will do what it’s supposed to do? In my job post example, this is where the job post audit comes in. This is a very important step because if you don’t know what good means in recruiting, it’s easy to build a tool that does a task but doesn’t drive better results. We don’t build for fun. Build for utility if we’re going to waste this many natural resources on tech.
- Make time to experiment. I’m talking thirty minutes, maybe an hour, once a month, where you all sit in the same room and do show and tell. I know you’re busy. I know everything is important. Doing more with less. Blah blah blah. It’s still worth it. Make time to try these tools – if only for the benefit of upskilling your team for their next jobs because you don’t know what’s coming.
I’m currently running in-house recruiting labs to help teams get started because we can’t just wait for a rollout. We need to start this growth and skill development now. I’m booked until June with teams who want to inspire experimentation, but if you’re looking for team training this fall – let’s talk now so I can do a kickoff training that will give your team confidence to experiment again.