When a Worker Returns Garbage: Validating at the Handoff
Thirty-nine extracts were fine. The fortieth had every field present, correctly typed, and null — and the supervisor aggregated it, because well-formed looked like correct.
A supervisor fanned forty invoices out to worker agents for field extraction. Thirty-nine came back with sensible data. One came back with every field present, correctly typed, and null — a perfectly well-formed record of nothing.
The supervisor aggregated all forty into the summary. The report understated the quarter's total by one invoice, and nothing in the system noticed.
The supervisor's blind spot
A supervisor receives a result and, by default, treats well-formed as correct. That's a reasonable instinct in ordinary software, where a malformed response throws and a well-formed one is data.
Agent workers break the instinct. They rarely fail loudly. Given a document they can't read, a page that didn't load, or a schema they can't populate, they return the shape they were asked for — populated with whatever they can manage. Nulls, empty strings, plausible-looking guesses, or a confident extraction of the wrong document.
→ The result is syntactically indistinguishable from success. Which means the check has to be semantic, and it has to happen at the boundary, in code the supervisor owns.
Three layers of check
Schema validation. The floor, not the ceiling. It catches a worker that returned prose instead of JSON and nothing else.
Semantic validation — the layer that actually catches things:
validate_extract(result, source_doc):
require(result.total is not None) # required fields populated
require(result.currency in ALLOWED) # values in range
require(result.line_items) # not structurally empty
require(sum(i.amount for i in result.line_items)
== result.total) # internal consistency
require(result.invoice_no in source_doc) # grounded in the source
That last one is the strongest single check available for extraction work: every extracted value must appear verbatim in the source document. It's a substring test, it costs nothing, and it catches hallucinated fields and wrong-document extractions in one move.
Cross-item validation. Available only to the supervisor, and almost nobody writes it. See below.
🔍 The outlier check nobody writes
When you have forty results for the same kind of work, they're comparable — and comparison is information no individual validator has.
Compute, across the batch:
- Null or empty rate per field. One result with 90% nulls when the batch median is 5% is the failure, even though it validated.
- Output length. An extract a fraction of the size of its peers usually means the worker gave up early.
- Confidence distribution, if workers report it.
- Processing time. An item that finished suspiciously fast often means it failed fast.
Any item more than a modest distance from the pack gets re-run before aggregation. ⚠️ This catches the exact case the schema check misses — a result that is individually plausible and collectively anomalous.
The four responses, chosen deliberately
When a result fails validation, there are four things a supervisor can do, and the failure type should pick:
- Retry the item as-is — for transient causes: a timeout, a rate limit, a tool error. Cheap, and cap it.
- Retry with more context — for ambiguity: add the constraint the worker was missing, or a clearer instruction. Different from a bare retry and worth distinguishing, because repeating an identical call rarely produces a different answer.
- Escalate the item — the worker can't do it and neither can another attempt. A human queue, with the reason attached.
- Proceed without it, and say so — the item is genuinely unavailable. The final output must name it as missing rather than quietly containing thirty-nine of forty.
✅ The fourth is the one that matters most, because it's the difference between a partial result and a silently wrong one. Never let an unvalidated item into the aggregate, and never let a dropped item vanish from the report.
What the worker should return to make this possible
Validation is easier when the worker cooperates. Ask every worker to return, alongside its data:
{ ...extracted fields...,
confidence: "high" | "low",
unsure_about: ["line 4 total is smudged"],
evidence: {"invoice_no": "INV-2231 (page 1, header)"},
source_ref: "doc_8821"
}
unsure_about costs one field and points the supervisor's attention directly at the risky part. evidence makes the grounding check mechanical. source_ref catches the wrong-document case, which otherwise validates perfectly.
The rule
Nothing enters an aggregate until it has been validated at the boundary it crossed.
Not because workers are unreliable in general — most results are fine — but because the one bad result in forty is invisible by construction, and the aggregate is exactly where it stops being traceable.
The takeaway
A supervisor's default assumption that well-formed means correct is what lets a null-filled extract into a financial summary. Validate semantically at the handoff: required fields populated, internal consistency, and every value grounded verbatim in the source. Then run the cross-item comparison only a supervisor can — null rates, lengths, timings — and re-run the outliers. Pick deliberately between retry, retry-with-context, escalate, and proceed-while-naming-the-gap. And have workers report what they were unsure about, because that's the cheapest validation signal you will ever get.