The Travel Agent Reviewed Its Itinerary Three Times and Approved the Wrong Date Every Time. Reflection Needs Something to Check Against.

A personal travel agent is asked to book a trip: "Lisbon, flying out the Friday after next, back Monday, aisle seat." It drafts an itinerary, then runs a reflection step: review your plan for errors and improve it. Three rounds later the itinerary is tidier, the hotel is closer to the old town, the layover is shorter, and the outbound flight is still on the wrong Friday.

Nothing crashed. Every round of reflection made a real improvement. The one thing that mattered was never examined, because the critic was looking at the same thing the generator had already looked at, and agreeing with it.

This is the most common way reflection gets built, and it is worth taking apart.

🔍 The autopsy

Here is the loop as it was written:

draft = llm(f"{system}\n{history}\nPlan the trip.")
for _ in range(3):
    critique = llm(f"{system}\n{history}\nDraft:\n{draft}\nList any problems.")
    draft = llm(f"{system}\n{history}\nDraft:\n{draft}\nCritique:\n{critique}\nRevise.")

Three things went wrong, and they compound.

1. The critic inherited the generator's misreading. Early in the conversation the agent had resolved "the Friday after next" to a specific date and written it into its working notes. That resolution sits in history. When the critic reads the history, the date is already there, stated confidently, in the agent's own voice. The critic has no reason to recompute it. The mistake is no longer an interpretation; it has become context.

2. "List any problems" invites easy problems. An open-ended critique prompt gets answered with whatever is most visible in the draft: formatting, a long layover, a hotel rating. Those are real findings, so the critique looks productive. But the critic is ranking issues by how noticeable they are in the text, not by how much they would cost the traveller. A wrong date reads perfectly fine on the page.

3. Nothing in the loop could disagree with the model. Every input to the critic was either the model's own output or a transcript the model had already interpreted. There was no calendar, no restatement of the original words, no tool result. A loop with no external signal can only converge on what the model already believes, and it converges with rising confidence, because each round adds another pass of "reviewed and revised" to the record.

The result is the worst kind of failure: a confidently wrong answer carrying evidence that it was checked.

Why this is a design problem, not a model problem

It is tempting to say a stronger model would have caught the date. Sometimes it would. But the structure of the loop is what made the error invisible, and that structure stays the same no matter which model sits inside it. If the critic sees exactly what the generator saw, framed exactly the way the generator framed it, you have built a second opinion from the same person in the same mood.

Reflection earns its cost only when the critic has access to something the generator lacked or ignored.

✅ Three ways to give the critic something to check against

1. Re-anchor on the original request, verbatim

Do not pass the critic the full working history. Pass it the user's original words and the draft, and ask it to verify each requirement independently.

requirements = llm(f"Extract every explicit requirement from this request as a checklist:\n{user_request}")
critique = llm(
    f"Request (verbatim): {user_request}\n"
    f"Today's date: {today}\n"
    f"Checklist:\n{requirements}\n"
    f"Draft:\n{draft}\n"
    "For each checklist item, recompute it from the request and today's date, "
    "then state PASS or FAIL with the value you computed and the value in the draft."
)

The key word is recompute. The critic is told to derive the departure date itself from the raw request and today's date, then compare. Asking "is the date correct?" lets it glance and agree; asking "what should the date be?" forces it to do the work.

2. Run the critic in a fresh context

Even with a checklist, a critic that can see the agent's earlier reasoning will tend to defer to it. Give the critic a clean context: the original request, the draft, and the checklist, nothing else. No scratchpad, no tool chatter, no earlier "I'll interpret this as the 17th." Stripping context feels like throwing away information. Here, the thrown-away information is exactly the anchoring you are trying to escape.

3. Let a tool say no

Some requirements should not be judged by a model at all. Dates, seat assignments, totals, and anything with a source of truth should be checked deterministically.

• ✅ Resolve "the Friday after next" with a date function and assert the booking matches.
• ✅ Read the seat back from the booking API rather than from the draft.
• ✅ Sum the prices in code and compare against the stated budget.
• ❌ Ask the model whether the date "looks right."

A tool check that fails is the most useful thing a reflection loop can produce, because it is the only signal in the loop that does not come from the model's own beliefs.

Making the critique rank by cost, not visibility

Once the critic is checking against something real, change what it reports. Instead of "list any problems," ask it to sort findings into two bins:

• Blocking: violates an explicit requirement, or would cost money or time to undo (wrong date, wrong city, wrong passenger name).
• Polish: everything else.

Then make the revise step act on blocking findings first, and stop the loop once there are none, rather than after a fixed number of rounds. A loop that ends when the checklist passes has a meaning. A loop that ends after three rounds just ends.

A quick test for your own reflection step

Look at the inputs to your critic and ask:

1. Does it see the user's original words, or only the agent's paraphrase of them?
2. Does it see anything the generator did not produce: today's date, a tool result, a spec?
3. Is it asked to recompute requirements, or only to judge the draft?
4. Can any part of the check fail without the model's agreement?

If every answer points back to the model's own output, the reflection step is producing confidence, not correctness. Keep it if the polish is worth the tokens, but do not count it as verification.

Reflection is a good pattern with a narrow precondition: the critic must stand somewhere the generator did not. Give it the original request, a clean context, and at least one check it cannot talk its way past, and the same three rounds that polished a wrong date will catch it on the first pass.