Handing an Agent a PDF: Three Paths Into the Context

"The agent can read PDFs" describes a capability and hides a decision. A PDF never enters a context window — something derived from it does, and which derivation you chose determines what the agent can see and what it silently can't.

A signed contract that comes back with the counterparty name right and the effective date wrong is usually not a reasoning failure. It's a document that took the wrong path in.

What the model actually receives

Two possibilities, always: text tokens or image tokens. Every ingestion approach is a way of turning a file into one or both.

That framing makes the paths easy to compare, because each one is a different lossy conversion.

Path 1: pull the text layer

Most PDFs carry an embedded text layer, and extracting it is fast, cheap, and produces a small context.

Keeps: the words, in reading order, usually.

Loses: layout entirely. A two-column page interleaves. A table becomes a run of cell values with no rows. Checkboxes, initials, stamps and signatures don't exist. Anything rendered as an image contributes nothing.

⚠️ And a scanned document has no text layer at all. Extraction returns empty or near-empty, which — depending on your handler — reaches the agent as an empty result rather than as an error. That's the single most common cause of confident nonsense about a scanned contract.

Path 2: render pages as images

Rasterize each page and send it as an image.

Keeps: layout, tables as tables, checkbox states, stamps, handwriting, anything visual. Works on scans, because it doesn't care whether text was ever encoded.

Loses: exact character fidelity. Reading a long reference number or an account string off an image is a recognition task, and a transposed digit looks identical to a correct one downstream.

Costs: images are expensive in tokens compared to the same page as text, and there's a resolution trade — render small and fine print becomes unreadable, render large and the cost climbs. Both directions fail quietly.

Path 3: preprocess into structure

Run a document-understanding step first, producing text plus positions plus table structure, then hand the agent that.

Keeps: the most, for forms and tables specifically. Cell relationships survive. Every value can carry a page and a coordinate.

Gains: grounding. If an extracted value knows where on the page it came from, a human can check it in one click and your validation can require it.

Costs: a pipeline dependency with its own failure modes, and a step that can be wrong before the agent ever reasons.

✅ The hybrid most working systems land on

Not choosing one — routing per document, and often per page:

if page has a usable text layer and no tables
    → text extraction        (cheap, exact characters)
if page has tables, checkboxes, or is a scan
    → render as image, or run document-understanding
always
    → keep page numbers with every extracted value

Two rules that matter more than the routing itself:

Never send the whole document by default. A sixty-page agreement should not arrive as sixty page images. Do a cheap first pass — text extraction, or a table of contents, or a keyword scan — to select the three pages that matter, then send those properly.

Keep the provenance. Page number at minimum, coordinates if the path provides them. It's what makes a value checkable rather than merely plausible.

🔍 The check that resolves most reports

When someone says the agent misread a document, print exactly what the agent received for that document — the extracted text, or the image, at the resolution actually sent.

Roughly speaking, the answer is one of:

• The text layer was empty and nobody noticed the document was a scan.
• The relevant page was never included.
• The table arrived as an undifferentiated run of numbers.
• The image was rendered at a resolution where the fine print isn't legible.

None of those are the model reasoning badly, and all of them are invisible until you look at the input.

Practical rules

• Log which path each document took, and whether text extraction returned anything. An empty text layer must surface as a distinguishable condition, never as a silent empty result.
• Cap pages per request, and select before sending.
• For identifiers and amounts, prefer character fidelity — and verify the extracted value appears verbatim in the source text where a text layer exists.
• Test on your worst documents, not your cleanest: skewed scans, rotated pages, multi-column layouts, handwritten annotations, forms someone filled in badly. Those are the ones that will arrive in production.

The takeaway

A PDF becomes text tokens or image tokens, and the conversion decides what the agent can see. Text extraction is cheap and destroys layout and fails silently on scans. Page images preserve everything visual and cost more and can misread a digit. Preprocessing gives structure and grounding at the price of another pipeline. Route per page, select pages before sending, keep provenance with every value — and when the agent seems to have misread something, look at what it was actually given before looking at the prompt.