11 min read
AI Engineering · Forward Deployed Engineering
AI Won't Replace Engineers Who Think: What a 29-Second Wait Taught Me
Every week someone asks me the same thing: will AI replace software engineers?
After this project, my answer is clearer than ever. AI is not replacing engineers. It is replacing the habit of copying whatever the AI says without asking if it is right. The engineers who keep questioning, measuring and owning the decisions are becoming more valuable, not less.
This post walks through a real project that shows the difference. You will see the problem, the questions that mattered, where AI helped and where it would have led me the wrong way, and the result.
Client, vendor and model names are left out on purpose (NDA). Every number here was measured on staging before launch, on the same code that is now live, with all 4,700+ automated tests passing.
(opens the full-size image)Will AI replace software engineers? My short answer
AI will replace engineers who only follow it. It will not replace engineers who think with it.
AI is excellent at producing answers. It is far less reliable at knowing which question to ask, which answer to trust, and what is safe to put in front of real people.
That gap is the job. In my work as a forward deployed engineer, I sit close to the client, the users and the real constraints, and make AI work inside them. Those constraints are rarely the model. They are busy days, slow outside services, limited server capacity, and people under pressure.
The problem: a 29-second spinner in a timed exam
The platform is an enterprise AI assessment system used for hiring. Recruiters build written, spoken and adaptive tests. Candidates take them in the browser. AI transcribes spoken answers and rates pronunciation and open answers against a marking guide. The platform's own scoring rules turn those ratings into the final result.
While testing before a big release, we timed the test the way a candidate experiences it. Here is what we found:
| Moment in the test | What the candidate experienced |
|---|---|
| After a short spoken answer | About 6 seconds of waiting |
| After the last spoken answer | 28.6 seconds of waiting |
| A recording with only silence and a breath | Refused on every attempt. The candidate was stuck |
The cause was simple. Every time a candidate pressed Submit, the system sent the answer to several AI services and waited for all of them to finish before showing the next question. The AI was sitting directly between a nervous candidate and their next question. Engineers call that being on the "critical path".
And it was about to get worse. We were switching to a new AI provider that was slower at grading.
What "just follow the AI" would have produced
If I had opened an AI assistant and typed "this is slow, make it faster", I know the kind of answers I would have got:
- Use a faster model
- Shorten the timeouts
- Add a nicer loading animation
None of those are wrong on their own. But every one of them keeps the candidate waiting. A faster model still takes seconds. A shorter timeout turns a slow answer into a failed one. A nicer spinner is still a spinner.
The AI would have optimised the thing I pointed it at. The real fix needed someone to step back and question the setup itself.
The three questions a person had to ask
1. Does anything need the AI score right now?
This one question unlocked the whole design. If the next question depended on the AI's score, grading would have to stay where it was.
So I checked every type of test on the platform: fixed question sets, language tests and adaptive tests. In every case, the next question is chosen without any AI-graded score. The score only matters to the recruiter, after the test ends.
That meant the candidate never needed to wait for it.
2. What happens on a busy hiring day?
A slow answer for one candidate is annoying. The same delay for many candidates at once is a capacity problem.
On a hiring day, lots of candidates submit at the same moment. Each one was holding the system busy for 6 to 29 seconds. That pile-up could slow the platform for everyone, including recruiters who had nothing to do with the test.
3. What happens when the AI services disagree?
This was the nastier problem. A candidate recorded four seconds of room noise and one breath. One AI service heard nothing. Another "heard" a couple of words that were never said.
Our system treated that disagreement as a temporary error and told the candidate to try again. But the recording never changed, so every retry failed the same way. The screen hid the error. The candidate could not move forward.
No single bug caused this. The system simply had no safe answer for "the AI cannot agree". Trusting the AI blindly would have kept trapping real people.
Where AI earned its place
Once I knew what I was solving, I brought AI in as a thinking partner. Not as the decision maker.
I explained the system, my concerns and my first idea, and asked it to challenge the design and research how established testing companies handle automated scoring. The findings were clear:
- Pearson Versant, Duolingo English Test, Cambridge Linguaskill and ETS all build and check their automated scoring against expert human raters.
- At Duolingo, answers the AI flags as off-topic or unintelligible are reviewed by a person.
The research also challenged my first design. For example, it pointed out that a burst of finished tests could use up the database's connections and slow the whole platform, which shaped how we limit background scoring.
Here is how the work split between us:
| What AI did well | What I had to own |
|---|---|
| Researched how the industry solves this, quickly | Asked whether the score was needed at all |
| Challenged my design and found risks | Judged which risks mattered and how to handle them |
| Helped write and test the code at speed | Decided that AI disagreement goes to a person, not an error |
| Drafted the proposal for the team | Got each release step approved, one at a time |
AI made me much faster. It did not make the calls.
The solution: save now, score later
(opens the full-size image)The new design is easy to explain, which is usually a good sign.
When a candidate submits an answer, the system now does only quick, certain work. It checks the answer belongs to the right question, saves it, marks it as "waiting to be scored" and shows the next question. No AI calls. Under a second.
When the test ends, a background process scores every waiting answer using the same AI services and the same marking guides as before. Only the timing changed.
If an AI service has a bad moment, the answer simply waits and is retried automatically every few minutes. The candidate never knows.
If the AI services disagree or hear nothing, the answer gets a "no speech" score and a Check recording flag. A recruiter listens and can set the level themselves.
On busy days, each server scores at most four answers at a time, so a rush of finished tests cannot overload the database. That keeps the platform fast for everyone, even in a rush.
And if anything goes wrong, one setting switches the whole thing back to the old behaviour, with no new release needed.
The best part: most of this machinery already existed. It had been built as a backup, so written answers would survive an AI outage. We turned the backup into the main path and added spoken answers to it. No new infrastructure.
For the engineers: pending answers are claimed with
SELECT ... FOR UPDATE SKIP LOCKED, so several servers share one database-backed queue without scoring an answer twice. Language and adaptive tests store progress in a single JSON column with no row lock on the answer endpoints, so scoring for those runs after completion to avoid a lost-update race.
The bottleneck that wasn't AI
After the first release, every answer was fast except the last one. It still took 12 to 15 seconds.
The obvious guess was "the AI again". It wasn't. When a test finished, the system emailed each reviewer one at a time, while the candidate waited. That delay had always been there, hidden inside the AI wait. Once the AI wait was gone, it became the slowest thing left.
We moved those emails to the background too, and the last answer dropped to about half a second. The same day, that fix was live.
If I had assumed AI was the only problem, we would have shipped a test that was fast everywhere except the final answer. Measuring found it. Assuming would have missed it.
The result
| Before | After | |
|---|---|---|
| Wait after a spoken answer | 6.3 seconds | Under 1 second |
| Wait after the final answer | 28.6 seconds | About half a second |
| Silent or breath-only recording | Candidate stuck in a loop | Accepted and flagged for a recruiter |
| AI provider hiccup mid-test | Candidate asked to record again | Retried quietly in the background |
| Results for the recruiter | At the end, unless AI failed | Within a minute of the test ending |
The worst moment of the test became about 55 times faster, with the same grading rules. The issue was found on a Thursday and the fix was live on Friday.
What this means for you
If you are an engineer: use AI hard, but keep the judgment. Ask what actually depends on the result before you optimise it. Measure in the real environment. Treat the AI's output as a strong draft, not a verdict.
If you are a recruiter or hiring manager: "uses AI tools" tells you little. Ask candidates about a time they disagreed with an AI suggestion, and why. That answer shows you whether they lead the tool or follow it.
If you lead a business putting AI into real workflows: the model is rarely your bottleneck. The system around it usually is. You need people who understand both your users and how the system behaves under pressure, and who can make AI safe in that setting. That is what forward deployed engineering is for.
more enterprise AI case studies
FAQ
Will AI replace software engineers?
AI will replace engineers who only copy its output. It is far less likely to replace engineers who ask the right questions, check the results and own the decisions. In this project, AI sped up research and coding, but the key decisions came from human judgment.
What is a forward deployed engineer?
A forward deployed engineer works close to the client and the real users, adapting a technical product to their actual constraints. In AI work, that means making models reliable inside real workflows: busy days, slow outside services, security rules and people under pressure.
How should engineers use AI tools without losing critical thinking?
Use AI to research, challenge your ideas and speed up building. Keep three things for yourself: deciding which problem to solve, deciding which output to trust, and deciding what is safe to release. Always measure the result in a real environment.
Why not just use a faster AI model?
A faster model still makes the user wait, and you rarely control an outside provider's speed. If nothing the user sees next depends on the AI's result, the better fix is to stop making them wait for it at all.
What does human in the loop mean in AI systems?
It means a person reviews the cases the AI cannot handle confidently. Here, when AI services disagree about a recording, the candidate moves on and a recruiter gets a flag to listen and decide.
The takeaway
AI is the most useful tool I have ever worked with. This fix went from problem to production in under two days because of it.
But the turning points were not things AI offered on its own. It was asking whether the score was needed at all, refusing to trust two AI services that disagreed, and measuring instead of assuming. That is the part of engineering that is getting more valuable every month.
If you are putting AI into a workflow where real people wait on it, I'd be glad to compare notes. Get in touch
I'm Muhammad Abdullah, a forward deployed engineer building AI systems for enterprise clients. More about my work