Hiring Engineers When Take-Home Tests Are Meaningless

Every implementation-based screening signal degraded at once. Take-homes can be completed by an agent. Coding exercises test the automated skill. Portfolio projects demonstrate tool access. The signals didn't become slightly noisier — they stopped measuring the thing they were proxies for.

Which is a problem, because they were measuring something real: whether this person can think about a problem and produce something that works.

What the old signals actually tested

Worth being precise, because the replacement has to test the same underlying thing:

• Can they decompose a problem?
• Do they notice edge cases?
• Can they produce something that works?
• Do they write code others can read?
• Do they know when they're done?

Implementation was the medium, not the point. The point was judgment, expressed through implementation because that was the available channel.

What still works

Reviewing code, live. ✅ Give them a real change — ideally one with a subtle problem — and discuss it. This tests exactly what's scarce: can they tell right from plausible, do they notice what's missing, do they ask what the change was supposed to do.

It's also honest about the job. Review is a large and growing share of the work.

Debugging something real. A failing system, a reproduction, and a conversation. Tests reasoning from evidence rather than pattern-matching, and it's hard to fake because the process is visible.

Discussing their own past work in depth. Not "tell me about a project" — what specifically did you decide, what did you reject, what went wrong, what would you do differently. Someone who did the work can go three levels deep; someone who didn't runs out.

Working through an ambiguous problem together. Give them something underspecified and watch whether they ask, assume, or guess. This is the single most predictive signal for the current job, because resolving ambiguity is what's left.

A take-home where using agents is expected and the conversation is about the choices. ⚠️ Rather than banning tools — which is unenforceable and tests the wrong thing — assume them and interview on the judgment: why this approach, what did you reject, what did you check, where were you unsure.

What to stop using

• Algorithmic puzzles. Tested a proxy that's now automated, and they were always a weak signal.
• Unsupervised take-homes evaluated on the artifact. The artifact says nothing about who produced it.
• Portfolio volume. Access to tools, not capability.
• Framework trivia. Lookup, and always was.

💡 The uncomfortable consequence

The replacements — live discussion, past-work depth, ambiguous problem-solving — are all more subjective, more time-expensive, and more prone to bias than a scored exercise.

That's a real cost, and it lands hardest on candidates without a track record to discuss or a network to vouch for them, which usually means people early in their careers or switching fields. The old exercises were flawed and they were at least comparable across candidates.

Nobody has a good answer to this. Partial mitigations worth using: structure the discussions with consistent questions, score against a written rubric, use more than one interviewer, and give candidates without a long history something concrete to work through so they're not disadvantaged by having less to discuss.

✅ A workable loop

1. Short screen: discuss a piece of their past work in depth.
2. Review exercise: a real diff with a real problem, discussed live.
3. Ambiguous problem: underspecified, worked through together, agents allowed.
4. Debugging: a reproduction and a conversation.

None of it tests implementation speed, all of it tests judgment, and it's honest about what the job is now.

The takeaway

Implementation-based screening stopped measuring what it proxied for. Replace it with live code review, real debugging, depth on past decisions, and an ambiguous problem worked through together with tools allowed. Accept that these are more subjective and more expensive, structure them against a rubric to limit the damage, and give candidates without a long track record something concrete — because the cost of this transition falls on them hardest.