Job search
AI Evaluation Engineer Jobs: Evaluation Frameworks, Quality Gates, and Project-Based Task Work
Four current employer-authored roles, a completed fictional shortlist and an executed synthetic quality-gate check separate ongoing ownership from project deliverables.
An AI evaluation job can involve maintaining a quality platform, deciding how release evidence should be interpreted, or creating the benchmark tasks other people will run. Those activities overlap, but they offer different kinds of engineering ownership. Record the deliverable, acceptance decision and engagement terms together before treating similar titles as interchangeable opportunities.
This guide checked public employer-authored pages on October 21, 2026. The sample is deliberately small, and we did not apply, take an assessment, run a provider's models or obtain a contract. LandOffer publishes this guide and offers job-search tools. Use recent tech jobs to build a broader search, then verify each promising role's actual responsibilities and location requirements.

Identify the decision the evaluation supports
Start with the output of the job. A reusable evaluation system needs someone to maintain datasets, execution environments, scoring implementations and interpretable reports as the product changes. A benchmark-authoring assignment may instead ask for a difficult task, a reference solution and checks that distinguish valid solutions from plausible failures.
A quality gate adds another question: who acts on the evidence? Building the check, proposing a threshold and authorizing a release are different responsibilities. A JD that mentions deployment decisions does not automatically give one engineer unilateral authority to ship or block every product. Identify the team that reviews exceptions, records unresolved failures and owns the resulting release decision.
Keep three fields in a role note: what you build, how your work is accepted, and who maintains it afterward. If the description only says “improve AI quality,” leave those fields unanswered and identify the missing responsibility. Ask what happens when the score improves but a critical failure appears, or when the scoring method itself changes.
The same questions expose overlap. A project contractor can build durable infrastructure. A full-time employee can spend a substantial period authoring evaluation cases. Employment type tells you about the proposed arrangement; it does not completely describe the engineering problem.
What four current public descriptions actually ask for
Titan AI's Lead QA Engineer is a US remote, full-time example of ongoing quality ownership. The description combines building an AI evaluation framework, regression baselines and CI/CD quality gates with broader application testing and production issue reproduction. It asks for seven or more years in software QA, including at least two personally testing AI or ML systems, and notes occasional travel. This is broader than scoring model responses.
At Elicit, the current Model Behavior Engineer role is full-time and listed for Oakland or remote within US time zones. It combines defining meaningful evaluations with implementing improvements in the product. Inspecting ambiguous failures, experimenting and maintaining software are central. Its location discussion also includes in-person retreats; a remote label should not erase that requirement.
Turing's AI Evaluation Engineer listing focuses on authoring software-engineering benchmark tasks, reference solutions, instructions and validation criteria across Python, Java or web work. Although its header says Full-time, the offer section explicitly describes a contractor/freelancer assignment, no medical or paid leave, forty hours weekly, four hours overlapping PST and a two-month duration. Its relative “next week” start wording does not establish a confirmed calendar start date.
TESTQ's AI Evaluation Engineer, reference 6003, supplies the useful counterexample: a remote contract role asking for gold sets with subject experts, evaluator calibration and regression suites in CI/CD. It is not merely a stream of isolated rating tasks. The page does not establish US eligibility, a pay rate or contract length; a link labeled Career in UK does not answer those questions either.
These are public descriptions, not verified offers or evidence that work will be assigned. Two other search-indexed evaluation vacancies returned “Job not found” on their actual pages during this review. We excluded their old responsibilities from the current-role comparison. A searchable excerpt can remain available after the destination changes.
Worked asset: complete a role-and-evidence shortlist
The following comparison records the four observed roles and makes a completed decision for an invented candidate. It does not rank companies or estimate your chances. Dara is fictional: eight years in software QA, two years testing an AI application, Python regression tooling and experience maintaining a release suite. Her record does not include scientific-domain research, model training or independent release authorization.
| Actual public role | Work and arrangement recorded | Dara's completed routing decision |
|---|---|---|
| Titan Lead QA Engineer | Maintained framework and application quality; US full-time; travel noted | Investigate first: her record supports the stated QA and AI experience categories; confirm travel and release-review responsibilities |
| Elicit Model Behavior Engineer | Evaluation design plus product implementation; full-time; US-time-zone remote scope | Secondary inquiry: show experimental reasoning evidence, but preserve the gap in domain-specific decision research |
| Turing AI Evaluation Engineer | Benchmark tasks and reference solutions; two-month contractor assignment; forty-hour commitment | Defer: Dara's stipulated availability is twenty hours weekly, despite the task-authoring fit |
| TESTQ AI Evaluation Engineer | Expert-calibrated framework and regression delivery; remote contract; geography and duration unresolved | Hold: verify location eligibility and engagement details before preparing domain-specific evidence |
Dara's finished evidence paragraph for the first route reads: “I maintained a Python regression suite for an AI-enabled application and documented failures for the release review. My contribution covered test implementation, case maintenance and reproduction records. The release approver made the shipping decision. I can explain how a scoring change was versioned and how an unresolved critical case remained visible.” This is stipulated fictional experience, not a statement about an actual reader or an employer's systems.
She would attach a permitted, sanitized example only if she actually had one. A tool list alone would not substantiate the paragraph: the reader needs a case record, the implemented check and the review boundary Dara actually owned. Her missing evidence is equally concrete: no claim of having built a statistical evaluator from scratch, no implied clinical expertise and no invented percentage improvement. If the real record lacks a maintained suite, replace the paragraph with a truthful smaller project rather than borrowing Dara's experience.
The shortlist has an outcome: one priority investigation, one secondary route, one availability-based deferral and one unresolved engagement hold. None is marked applied, qualified or hired. This makes the comparison useful even before any application is sent.
Show that a quality gate can reject the wrong improvement
To discuss evaluation engineering, bring an example where the acceptance rule changes the decision. An overall score can hide a failure in a necessary capability. Conversely, a single failed case does not establish a population-wide failure rate. The engineer needs to preserve both the decision rule and the limits of the evidence.
Our second original worked asset is a fully synthetic evaluation ledger for an invented document-tagging feature. There is no real model, customer document, production release or provider assessment behind it. The six cases are fabricated records; the pass/fail outcomes are stipulated inputs, not measurements of an AI system. We executed only the local consistency and gate logic described below.
The fixture fixes case IDs C1–C6, a dataset version and an evaluator version. C1 and C2 are designated critical by the fictional project brief because they concern whether an output retains its source reference. C3–C6 cover ordinary classification behavior. Those labels express the invented brief's priorities, not a universal AI safety standard.
| Fixed case | Baseline stipulated result | Candidate stipulated result | Decision relevance |
|---|---|---|---|
| C1, critical | Pass | Pass | Required case remains present |
| C2, critical | Pass | Fail | Candidate introduces a critical regression |
| C3 | Pass | Pass | Ordinary case retained |
| C4 | Pass | Pass | Ordinary case retained |
| C5 | Pass | Pass | Ordinary case retained |
| C6 | Fail | Pass | Candidate fixes one ordinary case |
Both versions pass five of six cases, approximately 83.3%. A rule that merely accepts an unchanged aggregate would pass the candidate. The completed gate decision is HOLD because C2 regresses, even though C6 improves. The evidence note records the C2 regression and C6 improvement separately. Neither the unchanged total nor the fixed ordinary case erases the stipulated critical failure.
Our original local Python check verified the equal case sets, unique IDs, version agreement, totals and the critical-case condition. It rejected three deliberately flawed interpretations: accepting the candidate because its aggregate did not decline, accepting a record with C2 missing, and allowing a duplicate passing case to inflate the denominator. These checks validate the bookkeeping and stipulated acceptance rule for this finite fixture only; the case outcomes were supplied as synthetic inputs. They do not establish model quality, production readiness or the optimal threshold for an employer.

The finished handoff note reads: “Candidate held under the stipulated rule: C2 changed from pass to fail; C6 changed from fail to pass. Overall passes remain 5/6. Dataset and evaluator versions match. Reproduce the C2 evidence and have the designated reviewer decide the next investigation; do not remove C2 or lower its importance to obtain a passing report.” No actual reviewer has approved this fictional release.
If a later candidate passed all six, that would resolve this particular gate condition, not prove all possible outputs correct. You would still need to understand coverage, repeated-run behavior, evaluator calibration and the consequences of unseen failures. We did not run repeated model trials or calculate confidence intervals from invented pass labels.
Match your evidence to the work being purchased
For a maintained platform role, describe the lifecycle of your evaluation artifacts. Explain how a new case enters the suite, who reviews the expected result, how a scoring change is versioned and how someone diagnoses a failed run. Preserve ownership boundaries: writing a CI check is different from owning the whole deployment process.
For task-authoring work, show the task's constraints, a reference solution and a justification for its checks. A solution that passes your own tests may still leave an ambiguity the instructions never resolved. Explain how you would identify that ambiguity before treating another solution as wrong. Use original examples you are permitted to share, not copied paid assessment material or confidential benchmark cases.
For either route, separate completed engineering evidence from proposed future work. A local fixture can demonstrate careful reasoning about case identity and acceptance rules. It cannot substitute for a claimed history operating a quality platform, working with domain experts or handling production incidents. State what you ran and what remains a design proposal.
An evaluation role can also require deep domain judgment. Recognizing an unsupported source reference in a toy document exercise does not qualify someone to judge clinical or regulatory content. The TESTQ description explicitly involves subject experts; do not erase those collaborators when describing the engineer's contribution.
Resolve the engagement before budgeting the opportunity
Ask for the actual employing or contracting entity, permitted location, required overlap, duration, compensation basis and how additional work is allocated. Distinguish a stated weekly commitment from guaranteed paid hours. A public listing, completed screening or proposed project period does not establish a signed offer, an available task queue or continuing work after the stated term.
Also ask what happens when a task is rejected or the benchmark specification changes. Who accepts the deliverable? Are revisions included, and how is a disagreement escalated? These are unresolved contract questions until the actual terms answer them; this guide supplies no universal payment or employment rule.
Choose the role whose deliverable, decision authority and arrangement match the work you want to own. Keep the source page, observed work arrangement and unresolved location or acceptance questions beside your evidence note. Recheck them before committing to an assessment or reserving project time. LandOffer can support a broader employed-role search while you evaluate a finite contractor opportunity separately.
Sources and Further Reading
- Titan AI Lead QA Engineer: current full-time framework and quality-gate responsibilities.
- Elicit Model Behavior Engineer: current evaluation and implementation role reached from the official index.
- Turing AI Evaluation Engineer: task authoring and explicit contractor offer details.
- TESTQ role 6003: contract framework work and unresolved geographic scope.



