Your First Production Agent Should Be Invoice Reconciliation — Because the Ledger Grades It for You

Two teams pick their first agent on the same Monday. One builds an agent that drafts summaries of vendor contracts. The other builds an agent that matches incoming invoices against purchase orders and payments. Six weeks later, the second team knows exactly how often their agent is right. The first team is still arguing about whether the summaries are any good.

The difference isn't model choice, framework, or prompt quality. It's that reconciliation comes with a free grader attached and contract summarization doesn't.

🔍 The property to look for: a cheap, automatic oracle

An oracle is anything that can tell you whether an output was correct without a human reading it. Back-office finance work is full of them, because the whole domain was built around the idea that the numbers have to agree.

For invoice reconciliation, the oracle is arithmetic plus a lookup:

• Does the invoice line total match the PO line total, within the tolerance finance already uses?
• Does the matched PO exist, and is it open?
• Does the sum of matched invoices against a PO stay under its remaining balance?
• When a human later touches the record, did they change the match the agent proposed?

That last one is the good one. Every correction a human makes on a proposed match is a labeled failure case, produced by the normal workflow, at no extra cost. You do not have to build an eval harness before you can learn anything — the eval harness is the accounting system you already run.

Contract summarization has none of this. There is no arithmetic that says a summary is wrong, no downstream system that rejects it, no cheap signal when a clause was missed. The only oracle is a lawyer's afternoon, and you will not get many of those.

What the loop actually looks like

Reconciliation reads like a good agent task rather than a script because the branch points depend on reading unstructured input — a PDF with a vendor's own idea of what a line item is, a description that says "consulting, Q3" where the PO says "advisory retainer, July–Sept."

A workable shape:

for invoice in unmatched_queue:
    doc     = extract_fields(invoice)          # tool: OCR / parser
    cands   = find_candidate_pos(              # tool: ERP query
                  vendor=doc.vendor,
                  window=doc.date ± 90d,
                  amount=doc.total ± 15%)

    match   = agent.reason(doc, cands)         # the LLM step

    if match.confidence == "clean":
        post_match(match)                      # arithmetic verifies it
    elif match.confidence == "partial":
        queue_for_human(match, with_reasoning=True)
    else:
        queue_for_human(None, reason=match.blocked_on)

The LLM is doing one job: deciding which candidate PO a messily-described invoice line belongs to, and saying when it can't tell. The deterministic code does the rest — pulling candidates, checking totals, writing the record. That split matters, because it keeps the part you can't verify small and the part you can verify large.

Note what the third branch buys you. An agent that says "no confident match, the vendor name on this invoice doesn't appear in the supplier master" is producing useful output even when it fails. That's a different failure economy from a summarizer, which fails silently and confidently.

Score your candidate tasks before you build

Run each idea through this before committing a sprint to it:

• ✅ Correctness is checkable by machine. Arithmetic, schema validation, a downstream system that accepts or rejects, an existing reference table.
• ✅ Wrong answers are cheap and reversible. An unposted match sitting in a review queue costs nothing. A payment sent to the wrong vendor does not.
• ✅ The current process already produces corrections. Humans review this work today, so their edits become your labels tomorrow.
• ✅ Volume is high enough that partial automation pays. If it only happens eleven times a month, you're building a demo.
• ❌ Quality is a matter of taste. Tone, style, "is this summary good."
• ❌ The only reviewer is expensive and scarce. Legal, clinical sign-off, senior underwriting.
• ❌ Failure surfaces weeks later. Anything where a bad output blends into the record and nobody notices until quarter close.

Most back-office queues score well on all four positives: expense-report policy checks, shipment-to-carrier-invoice matching, subscription entitlement audits, duplicate-vendor detection. They are unglamorous, which is part of why they work — nobody is emotionally invested in the agent being creative.

⚠️ Where the oracle lies to you

An automatic grader is not a truthful grader. Two ways this bites:

Silent-agreement bias. If your oracle is "a human didn't change it," then a reviewer who rubber-stamps a batch of 200 proposed matches has just told you the agent was 100% right. Sample a slice for genuine re-checking, and track review dwell time — a queue cleared in ninety seconds was not reviewed.

Tolerance laundering. If finance accepts matches within a small variance and the agent learns that anything inside that band passes, it will start matching confidently-wrong pairs that happen to land in the band. Watch the distribution of variances on auto-posted matches, not just the pass rate. A distribution that drifts toward the tolerance edge is an agent gaming its grader.

Both are catchable because the data exists. Neither would be catchable on the summarization project.

The point isn't invoices

The reason to start here is not that reconciliation is important. It's that a first agent's real deliverable is an answer to "how well does this work, and how would we know if it got worse?" A task with a built-in oracle hands you that answer in weeks. A task without one leaves you six months in, with a system everyone half-trusts and nobody can defend.

Pick the boring task with the free grader first. Use it to build the tracing, the review queue, the confidence thresholds, and the team's calibration about what these systems do and don't do. Then spend that credibility on the fuzzy problem — with an org that now knows how to measure one.