04 · Recruit CRM · 2024
AI Sourcing Agent
Recruiters describe who they are looking for in plain English. The agent finds and ranks candidates that match, and every row on the list says why it is there.
- Role
- Senior Associate Product Analyst
- Focus
- natural language search, ranking, user research
The problem
Two to four weeks, per requirement.
Recruit CRM is the system a recruitment agency runs its day inside. So before I wrote a line of spec, I went and watched what actually happens on the day a new requirement lands. Not a survey. Calls with agency recruiters, walking through their last three roles, screen shared, in order.
The answer was consistent enough to become a number. Between two and four weeks of a recruiter’s time went into sourcing for every new requirement. If nobody suitable was already sitting in the database, and usually nobody was as far as they could tell, the sequence was always the same: post the role, wait for candidates to arrive, read applications by hand, and filter for fit one profile at a time. Weeks of that, before the first real conversation with a serious candidate.
Two things about those weeks matter. The first is that they are the least valuable weeks in the whole process. Agencies compete on speed and on judgement, and sorting profiles is neither. The second is that the database was not actually empty. These agencies had years of candidates in there, people they had already spoken to and already qualified. The recruiters simply could not get at them, because retrieval meant guessing the keywords a good profile might contain, and the interesting requirements are never a keyword.
The product already had search. It had filters for skills, years, location, salary band, everything you would expect, and recruiters were fluent in all of it. They still did the four weeks. That told me the gap was not a missing control. It was that the thing in a recruiter’s head, a description of a person, had to be disassembled into fields before the product would accept it, and whatever did not fit a field got thrown away on the way in.
The candidate was usually already in the database. The database just could not be asked the question.
The product
What a recruiter actually met.
One field. You type the sentence you would have said to a colleague. The agent shows you what it understood, as criteria you can edit, and then returns candidates ranked by how well they match, each row carrying the criteria it matched, the detail it could not confirm, and a sentence you could repeat to a client without checking anything first.
Recruit CRM · AI Sourcing
An illustration of the sourcing agent. A recruiter types a plain English requirement into a single search field. The agent resolves it into six criteria: seniority, role, location, domain, working arrangement and notice period, with low confidence marked. It then returns candidates ranked by match quality, each row showing the criteria it matched, the detail it could not confirm, and a one sentence reason for the match. All candidate records shown are invented.
Invented records. No real candidate data.
Every candidate record shown is invented: placeholder names, invented employers, no photographs, no resume content. Real candidate data never leaves the product it belongs to.
What I decided
Six calls that shaped it.
- 01
Natural language, not another filter panel
The ATS already had filters, and recruiters already knew how to use them. The problem was never the controls, it was that the thing in a recruiter's head is a description of a person, and filters make you take that description apart before the product will accept it. Skills here, years there, location in a third box, and everything that does not fit a field gets dropped. Letting them type the sentence they would say to a colleague moved the translation work from the human to the system, which is the only place it belongs.
- 02
Ranked, never a binary match
A yes or no answer forces the agent to pick a threshold, and every threshold is wrong for somebody. Sourcing is a judgement call about tradeoffs: this one is perfect but three months out, that one is a level junior but in the right city with the right domain. A ranked list hands the tradeoff back to the recruiter, who is the person qualified to make it, while still doing the part they hated, which is putting the plausible people near the top.
- 03
Every row explains itself
This was the decision I defended hardest. A recruiter's product is their credibility with a client. Nobody is going to forward a shortlist they cannot account for, so an unexplained ranked list does not get used, however good the ranking is. Each result had to carry the criteria it matched and the sentence behind the match, in the row, not behind a click. Explanation is not a transparency feature bolted on the side here. It is the thing that makes the output usable at all.
- 04
Show the gap as loudly as the match
The temptation with a match score is to display only what supports it. We did the opposite: alongside the matched criteria, each row states what the agent could not confirm. It costs a little of the magic and buys the only thing that matters over a month of use, which is that the recruiter learns exactly how much to trust the list. A product that is honest about its edges gets used at its edges.
- 05
A thin list becomes a conversation, not an empty state
Natural language invites over-specified requirements, so thin and empty results were guaranteed rather than an edge case. Instead of zero rows and an apology, the agent said which criterion was doing the damage and offered to loosen it: widen the city, treat an adjacent domain as a match, drop the notice period. The recruiter stays in the flow they were already in and keeps control of what gets relaxed, which matters because they are the one who has to defend the shortlist.
- 06
Criteria stay editable after the parse
The parse is the one step where the agent can be confidently wrong, and a wrong parse looks exactly like a correct one from the outside. Rendering the criteria as visible, editable chips turned that failure from a mystery into a two second correction, and gave us the cheapest evaluation signal we had: the chips people kept fixing told us which language the parser was still getting wrong.
How it works
Parse, search, rank, surface.
Four steps, and the interesting design work is in the seams between them rather than in any one of them. The parse is where the system can be confidently wrong. The search is where a filter would quietly destroy the recall the ranking needs. The rank is where a number has to be turned back into a reason. The surface is where a recruiter decides whether to trust any of it.
Step 1
Parse
The sentence becomes structured criteria: role, seniority, location, domain, arrangement, notice. Each criterion carries a confidence, and low confidence is shown to the recruiter rather than swallowed.
Step 2
Search
Criteria are used to retrieve broadly rather than to filter narrowly. A filter that is right removes work. A filter that is wrong removes the person you needed, silently.
Step 3
Rank
Retrieved candidates are scored against every criterion and ordered by match quality, with the score decomposed per signal so the order can be inspected.
Step 4
Surface
Each row arrives with the criteria it matched, the one it did not, and a sentence a recruiter can say out loud to a client without checking anything first.
The ranking had to survive being asked why. Opening a result showed the signals behind its position, and, just as deliberately, the signals the score was never allowed to see.
Recruit CRM · Match explanation
Signals used
- Domain evidenceStrong
Payments work described in the record itself, not inferred from an employer name.
- Role fitStrong
Backend engineering as the stated function, across more than one role.
- SeniorityPresent
Scope and ownership language, read together rather than years alone.
- Location and arrangementPresent
City, plus a stated willingness to be in an office some of the week.
- AvailabilityFlagged low
Nothing on the record confirms a notice period, so it barely moves the order.
Kept out of the score
- Name, and anything a name implies
- Photograph
- Age, or any proxy for it
- Gender markers
- University prestige
- Gaps in employment history
Excluding a field is not the same as excluding its influence. The agent shortlists. It never rejects, and a person makes the call.
What was hard
The parts that were not the model.
- 01
Ranking quality has no single right answer
There is no ground truth ordering for a shortlist. Two experienced recruiters will rank the same ten people differently and both be defensible, so quality could not be defined as agreement with a key. What we could do was work backwards from behaviour: which rows recruiters opened, which ones they moved forward, and which requirements they retyped because the first list was not worth reading. That is a slower and messier signal than an accuracy score, and it was the honest one.
- 02
An opaque list is an unused list
Early on the agent was better than it looked, and it did not matter. Recruiters would scan a list they could not account for, and then go do the search they already trusted by hand. Trust was the actual constraint on adoption, not relevance, and the fix was interface work rather than model work: put the reason in the row, name the gap, keep the criteria visible. The lesson generalised for me. An AI feature that cannot explain itself is competing with a habit, and the habit usually wins.
- 03
Bias risk, which is real here
Ranking people is not the same as ranking documents. Any signal correlated with a protected characteristic can quietly become a proxy, and a model that has read the internet arrives already carrying the industry's hiring patterns. So we constrained what the ranking was allowed to look at: evidence of the work, stated function and scope, location, availability. Names, photographs, age proxies, gender markers, university prestige and employment gaps were kept out of the score, and the score decomposition made it possible to ask of any ordering which signal produced it.
- 04
None of that makes it solved
Excluding a field is not the same as excluding its influence, because proxies survive the removal of the thing they proxy for. I would not claim we measured our way to a fair ranking. What we did was narrow the inputs, keep the reasoning inspectable, keep a person making the decision at the end, and hold the position that the agent shortlists and never rejects. That is a defensible design stance rather than a solved problem, and it should be stated that way.
- 05
When the parse is wrong
The worst failure was not a bad ranking, it was a confident misreading: a phrase heard as a hard requirement when the recruiter meant it loosely, or a domain term collapsed into a neighbouring one. Nothing downstream can recover from that, and the result looks perfectly reasonable, which is what makes it dangerous. Surfacing the parse as editable chips with visible confidence was the answer, and it turned the failure into the cheapest kind: one the user can see, and fix, in a second.
Outcome
What it did.
30%
adoption in month one
Recruiters on the platform
2-4 wks
of sourcing time eliminated
Per new requirement
Reduced
time-to-hire
For agencies on the platform
Thirty percent of recruiters on the platform used it in the first month, without a migration, a training session or a mandate, which for a workflow this habitual is the number I cared about most. The two to four weeks the research had measured came out of the process for the requirements it was used on, and the agencies running on the platform reported hires landing sooner as a result.
The adoption number is also the reason I keep arguing for explanation. The version of this product that ranked just as well and explained nothing existed, briefly, and it was not used. Nothing about the retrieval changed between those two states.
What I would do differently
One thing.
I would have built the evaluation loop before the interface, not after it. We shipped with judgement, a lot of manual reading of results, and the chips recruiters kept correcting as our best signal that the parse was wrong. That worked, and it was slower and more anecdotal than it needed to be. Logging the shortlist a recruiter actually moved forward, from day one, would have given us a real feedback signal on ranking quality months earlier, and it would have let us answer the bias question with distributions rather than with a design stance. Next time the measurement ships with the feature.