Shipping AI Features Broke Three Rules I Used to Build By
Authored by: Surya N
I was once asked by a recruiter using our platform – why one candidate ranked below another when both had nearly identical experience on paper. So I traced the request through our code and found nothing that could answer her question. We had given that scoring feature the way we submit a filter or a sort, and a filter need never be justified under normal circumstances.
That week changed how I plan engineering work afterwards.
What I learned from developing conventional web applications transferred over easily to developing AI solutions. However, three assumptions I made were immediately incorrect once the model was invoked in the request path, and all of them took us weeks to figure out.
Old Code Discipline Is Now Load-Bearing
The model supplier releases new models while phasing out older ones according to their own schedule, not yours. Five places in our product ask an AI model to do something: run an AI resume screener, score a resume against a role, draft a job description, summarise a profile, suggest interview questions such as while using the AI video Interviewer.
Earlier, each of those passes used to call the provider directly with its own specific instructions written.
The provider then decommissioned the version of the model we had all been referring to. They all malfunctioned on the same day.
All calls to each model now go to an internal service instead. The instructions are present in files with version numbers that get reviewed just like any other code.
We keep track of both the model version and the instruction version beside each score we produce. When the client asks for an explanation of why a particular candidate scored what they did six weeks ago, we run that exact combination of versions and show them.
Waiting Is Fine. Being Kept In The Dark Is Not.
Latency panic is the first instinct engineers have when it comes to AI features, and it is usually misplaced. Artificial Analysis clocks the quickest models at roughly 0.3 seconds to first token. However, even if a system thinks through a pile of resumes, it still does not get anywhere close to that, nor is engineering capable of bringing it closer. The recruiter can easily sit and wait as long as the monitor shows signs of action. A running count of resumes processed will do it. So will showing partial results as they are received.
The explanation is the tougher part of the response. In our system, when we score the candidate, the recruiter knows the criteria behind the scoring: what criteria were satisfied, which not, what points from the CV were used for this. And she has to be able to say no to this. Every single decision made by the AI in our product is reversible, and every single rejection is logged, with the reason; this guarantees the neutrality of the system. Recruiters trust the system that they can fix themselves.
Every Request Now Carries A Price Tag
SaaS pricing components have almost zero marginal cost. Model inference behaves nothing like it, which is why I say the same thing to every engineer who joins us. “Normal features cost money once to build. AI features cost money every time they’re used. So we check the cost before building to keep the business profitable.”
Our estimate before building anything now looks like this: what does one run of this cost at current provider prices? How many runs will a typical account make in a month, and what does the heaviest account plausibly do? What does that account pay us? If the answer to the third question does not comfortably clear the first two multiplied together, we do not build the feature as designed.
If you check OpenAI’s pricing page today, gpt-5.6-sol asks $5 per million input tokens and $30 per million output tokens. Their gpt-5.4-nano does the same work for $0.20 and $1.25 on the same per-million basis. So the small model makes the first pass over a large applicant pool, and the expensive one only ever reads the shortlist.
We ask those same three questions of every AI feature we ship, our time tracking software included.
What I Would Do Differently
Log what the model was, what it was told, and what the call cost. Do it from day one, even if nobody reads that log for months. We added ours late and burned a fortnight working out which model had scored which candidate.
Then price the feature before you design the screen. If the gap makes you wince, the fix is a smaller model or a usage cap. Not a bigger sales target.
We all rent pretty much equal intelligence now. The difference in products comes from whether the AI is able to justify itself and whether it pays for its own electricity. That’s the whole point of being for-profit or not.
About the author: Surya N is a software engineer at Hyring, where he leads engineering for the platform’s AI-assisted hiring workflows. He writes occasionally about the unglamorous (yet critically useful) side of shipping AI products, involving cost control, accountability, and other challenges that development faces in the AI era.