Evaluation Engineer: The Job Description Writing Itself

Of the roles forming around agent deployment, this one is furthest along — the work is well-defined, consistently needed, and consistently unstaffed. Worth writing down what it is, because organizations keep discovering they need it after the point where having had it would have helped.

What the job is

Owning the eval set as a specification. The cases define what the agent is supposed to do, which for many teams is the only place that's written down at all. Maintaining it is maintaining the requirements.

Building oracles. Finding or constructing the mechanical check for a domain that appears to lack one — quoted-span verification, reconciliation, round-trip identity, constraint satisfaction. This is the highest-leverage technical work in the role, because it converts an unmeasurable step into a measurable one.

Turning production failures into cases. Every real failure becomes a regression test. This is what makes the suite representative rather than imagined.

Designing measurement that isn't misleading. Knowing that a small suite can't detect a small difference, that paired comparison beats independent rates, that aggregate scores hide stratum-level regressions, that a judge needs calibrating against human labels.

Running model and prompt comparisons. The frozen suite, the four-dimension report, reading the diverging cases.

Maintaining the safety suites. Injection resistance, forbidden actions, boundary tests — run as gates rather than as metrics.

Why it's a distinct role

It requires domain knowledge and measurement skill together. Knowing what correct means for your domain, and knowing that 18/20 versus 19/20 is noise. Neither alone is sufficient, and the combination is unusual.

Its output is infrastructure, not features. Nobody ships an eval suite to a customer, which is why it loses prioritization arguments to feature work indefinitely unless someone owns it.

It's cross-cutting. It touches prompts, tools, the loop, and deployment, and sits with none of them.

⚠️ Which is exactly why it goes unstaffed: it's everyone's dependency and nobody's job.

What good looks like

• Cases in version control, alongside the code, reviewed like code.
• Deterministic replay, so a comparison measures the model rather than the world.
• Stratified reporting — per category, not just aggregate, so a hard-case regression can't hide behind easy-case stability.
• Safety invariants as gates rather than scores.
• Cost and turn count reported alongside quality, always.
• A cadence for re-sampling against current traffic.
• Should-decline cases paired with should-proceed ones, so calibration is measured rather than caution.

💡 Why it's a good position to take

Similar to agent operations, and for similar reasons:

• Unclaimed. Available without a title change.
• Structurally necessary. Every serious deployment needs it, and most discover this late.
• Built from durable skills — domain knowledge, measurement, verification judgment.
• Visible output. The regression you caught before it shipped is a concrete, attributable contribution, which is rare in infrastructure work.
• It makes you the person who knows what the system is supposed to do, which is the durable position generally.

🔍 Signs your organization needs one now

• Nobody can say what your agent's completion rate was last quarter.
• Prompt changes ship without a before/after comparison.
• The eval set hasn't been updated since it was created.
• Model upgrades happen by trying it and seeing how it feels.
• There's no test where the correct behavior is declining to answer.
• Nobody has run an injection suite.

Two or more of those and the work is already needed, and its absence is currently invisible — which is the characteristic problem with this role.

The takeaway

Evaluation engineering is owning the eval set as a specification, building oracles where none exist, converting production failures into cases, and designing measurement that doesn't mislead. It requires domain knowledge and statistical judgment together, it produces infrastructure rather than features, and it belongs to no existing team — which is why it's consistently needed and consistently unstaffed. It's also unclaimed, made of appreciating skills, and produces visible evidence of judgment.