Three Ways to Get Valid JSON, and What Each One Costs

A ticket-triage pipeline extracted six fields from each support message. Validity was reported at a comfortable number, achieved by a retry loop: parse, and if it fails, ask again.

The loop was working. It was also quietly biased, because the tickets that needed two or three attempts were not a random sample — they were the long ones, the multi-issue ones, the ones written in a hurry. On those, the eventual "valid" result was frequently valid and thinner than it should have been.

The three strategies, mechanically

1. Ask and parse, with retry. The prompt requests JSON. The model samples freely. Validity is a property of the sample, not a constraint on it — nothing prevents an unterminated string or a trailing comma, you just find out afterwards.

2. Constrained decoding. The schema is compiled into a grammar, and at each step the sampler is restricted to tokens that keep the output parseable. Invalid output isn't rejected; it's unreachable. Where a provider offers schema-enforced output, this is generally what's happening underneath.

3. Validate and repair. Parse, validate against the schema, and on failure send the broken output back with the error and ask for a corrected version. A second model call that operates on the first one's output.

Where each one actually fails

Ask-and-parse fails at a rate that rises with schema complexity and output length — more fields, deeper nesting, and longer values all mean more opportunities to drift. ⚠️ Critically, the failures are not uniformly distributed across your inputs. Difficult inputs produce difficult outputs, and difficult outputs are where format slips.

Constrained decoding guarantees syntax and guarantees nothing else. A schema-valid object can hold a fabricated invoice number, a mis-parsed date, or a category chosen because it was the first enum value. It also imposes a real constraint on generation: the model must commit to the structure as it goes, and forcing a shape can cost content quality — particularly with deep nesting, long enum lists, or many required fields.

Validate-and-repair doubles the cost on exactly the hard cases, since those are the ones that fail. And repair has a characteristic failure of its own: asked to make something valid, a model will often achieve that by dropping the part it couldn't express. The repaired output parses, and it's shorter than the truth.

💡 The bias nobody measures

All three strategies share one property that matters more than the choice between them: validity failures concentrate on hard inputs.

Which means a pipeline that retries until valid, or repairs until valid, systematically produces its weakest results on precisely the cases where accuracy mattered most — and reports a healthy aggregate validity number the whole time.

The measurement that exposes it is simple and almost never done. Don't report validity overall. Bucket your inputs by difficulty — length, number of distinct issues, whether the text is well-formed — and report validity and content completeness per bucket:

bucket            attempts (mean)   valid   fields populated
short, single       1.0             100%    5.8 / 6
long, single        1.2              99%    5.5 / 6
long, multi-issue   2.1              97%    3.9 / 6   ← the problem

Aggregate validity looks fine. The last row is where the pipeline is failing, and only the per-bucket view shows it.

The schema is a lever, not a given

An advanced detail worth acting on: schema shape affects content quality, not just parse success.

• Flat beats nested. Every level of nesting is more structure to hold while generating.
• Short enums beat long ones. A twenty-value enum invites a plausible-adjacent pick.
• Nullable beats required. ✅ A required field forces a value, and a model with nothing to put there will invent one. Making a field nullable and adding an explicit "unknown" option converts fabrication into an honest gap.
• Fewer fields per call beats one large object. Two focused extractions frequently beat one that tries to capture everything.

Redesigning a schema along those lines often improves accuracy more than switching enforcement strategies.

What to actually do

Use constrained decoding where the provider offers it. It removes an entire failure class for free, and it removes the retry loop that was creating the bias.

Keep semantic validation regardless. Enforcement gives you syntax. Whether the invoice number is real, whether the total reconciles, whether the quoted span appears in the source — all still yours to check, and all still where the damaging errors live.

Prefer an explicit unknown over a repair loop. A field the model can mark as not-found is worth more than a field it was forced to fill and you later had to repair.

Cap repairs at one. If a single corrective pass doesn't produce something valid and complete, route the item to a human. Repeated repair is how content quietly disappears.

🔍 The test

Take your twenty hardest real inputs — longest, messiest, most ambiguous — and measure two numbers, not one: how many produce valid output, and how many produce correct and complete output.

The gap between those two numbers is what your validity metric has been hiding.

The takeaway

Ask-and-parse leaves validity to chance and biases against hard inputs. Constrained decoding removes the syntax problem entirely and touches nothing about truth. Repair loops fix format by shedding content, on exactly the cases you care about. Use enforcement where you can, keep semantic checks either way, shape the schema for the model rather than for your database, and measure validity per difficulty bucket — because the aggregate number is designed to look good.