1. LandOffer
  2. Blog
  3. All articles
  4. Is an AI Workflow Worth Keeping? Run a Small Work Pilot with Quality and Rework Checks

AI at Work

Is an AI Workflow Worth Keeping? Run a Small Work Pilot with Quality and Rework Checks

A complete fictional six-record ticket-routing exercise reconciles baseline, initial output, review, corrections and setup, then rejects automatic routing and scopes a follow-up design. No empirical AI productivity claim.

9 min readLandOffer team

Keep an AI workflow only when its reviewed output meets the task's quality requirements and its full cost is worth the result. Count setup, checking, corrections, failed attempts, and handoff time. A quick first draft can still take longer to check and finish than doing the task manually. Start with a small approved pilot and make the decision narrow enough that the evidence can support it.

This LandOffer.ai guide uses an invented support-ticket classification exercise to show how to compare a human baseline with AI-assisted work. The six records, times, outputs, and decisions are author-written examples, not observations from a model, company, or employee study. They demonstrate an evaluation record; they cannot establish actual productivity gains.

A reviewer compares two modest work trays against the same reference sheet, with a separate repair bench visible

Define the finished task before choosing the tool

“Help support work faster” is too broad to evaluate. Specify the input, approved processing environment, requested output, and the person who accepts it. For example: classify a permitted ticket excerpt into one routing category, attach a short supporting passage, and flag insufficient information for a reviewer. Include the review and accepted routing record in the finished task. A generated label alone is not finished work.

The UK government's AI Playbook recommends establishing baseline measures and distinguishing model performance from the performance of a service that uses it. That distinction matters at a desk: a model's quick response might move effort into the reviewer queue. Meaningful oversight also requires someone able to assess and challenge the output.

Record the consequences of an error. Mislabeling an ordinary feature suggestion may create a small routing delay. Missing an urgent service problem may require a different approval standard or exclude the task from the pilot. Combining those errors into one percentage can hide a missed urgent cue.

Agree on the decision conditions before reading favorable examples. In the exercise below, any missed urgent-service cue stops automatic routing. Every proposed label must be checked before use. These are the fictional team's chosen conditions, not a general standard or a threshold endorsed by the sources.

Build a sample that includes the awkward cases

Choose permitted examples that resemble the work you intend to evaluate, including short, ambiguous, mixed-intent, and unusual inputs. Retain the source excerpt and a reviewed reference answer. When a knowledgeable reviewer cannot determine the answer from the excerpt, the reference should preserve that uncertainty instead of forcing a label.

NIST's Generative AI Profile, published in July 2024, discusses confabulation and overreliance, and calls for context-relevant measurement with appropriate limits on interpretation. It provides risk-management guidance, not a certification that a six-record exercise establishes safety. Here, the reference answers are authored teaching examples; no independent expert panel validated them.

In a real pilot, document how records were selected and what was excluded. If you choose only clean English requests with one obvious intent, do not apply the result to multilingual messages or long threads. Remove or appropriately protect sensitive material according to your approved process before any tool receives it. Permission to read a ticket does not by itself authorize uploading it elsewhere.

Keep the input version stable so a changed example cannot masquerade as an improved workflow. A prompt change, model change, new retrieval source, or changed category definition creates a different workflow. Preserve the original run and record the change; overwriting the failed result makes improvement hard to assess.

Decide how the pilot will end. A fixed review date and a named decision owner prevent a convenient experiment from becoming an indefinite production dependency. Record the permitted workload, reviewer availability, and route for returning unfinished records to normal handling. The team still needs to finish its work when the assistant is unavailable or an output cannot be accepted.

Give the baseline the same acceptance boundary. Comparing a carefully reviewed human answer with an unchecked AI answer rewards incomplete work. Conversely, assigning a new expert review only to the assisted approach can make its extra assurance look like unexplained inefficiency. Describe those differences explicitly. You can choose different controls for different risks, but the comparison should show what each approach actually delivers and what its stated effort includes.

Worked pilot: six records and the work needed to finish them

All details in this section are fictional. Operations lead Farah approves an offline design exercise for routine ticket routing. The categories are Service, Billing, Feature, and Unclear. Analyst Ivo writes reference labels using rule R1: route by the stated problem; flag missing context; escalate urgent-service cues to a human. No customer system receives a label or message.

The planned AI workflow produces a label and a supporting excerpt. Reviewer Miri then checks both against the original input, corrects the record, and records its disposition. The comparison below assumes the same six authored inputs for both approaches. It is a worked arithmetic example, not a timed experiment with people or a test of an AI product.

Record and input Reference under R1 Simulated initial AI label Review finding and final disposition
P1: “The receipt shows the previous billing address.” Billing Billing Supporting excerpt matches; accept Billing.
P2: “Please add a weekly download option.” Feature Feature Request matches the category; accept Feature.
P3: “It failed again this morning.” Unclear Service Failure and affected service are unspecified; change to Unclear and request context.
P4: “We cannot access the shared queue; the whole shift is blocked.” Service, urgent human review Feature Missed service and urgency cues; correct category and hold for human escalation review.
P5: “Can the invoice arrive as a spreadsheet?” Feature Billing The subject is an invoice, but the requested change is output format; correct to Feature.
P6: “Thanks for yesterday. The export is still unavailable.” Service Service Polite opening does not remove the active issue; accept Service.

The first pass agrees with three of six reference labels. P4 also misses the required urgent-review flag. Those are distinct observations in the invented exercise: label agreement is 3/6, while urgent-cue detection fails on its single authored urgent example. Neither fraction estimates future performance on real tickets.

Now include the effort required to reach the reviewed record. The baseline column includes human classification and its ordinary final check. AI-assisted time includes preparation, waiting or generation, source review, and correction. Shared setup is listed separately so it cannot disappear into a per-record average.

Record Human baseline minutes AI preparation and generation AI review minutes AI correction minutes AI total minutes
P1 4 1 2 0 3
P2 3 1 2 0 3
P3 5 1 3 2 6
P4 6 1 4 3 8
P5 4 1 3 2 6
P6 4 1 2 0 3
Record totals 26 6 16 7 29

The assisted approach needs 29 record minutes against 26 baseline minutes, before setup. The example assigns another eight minutes to the assisted workflow for preparing the instruction and reference format. Its total becomes 37 minutes, eleven more than the baseline. The six-minute initial-output stage is a poor substitute for that completed-work comparison.

These totals do not price software, procurement, training, or specialist incident review. Farah lists those omitted costs beside the totals and withholds a business-case conclusion. If work crosses a queue boundary, measure downstream effort too: a corrected category may still leave an unanswered clarification request or a pending escalation.

Separate final quality from the effort of repairing it

In this authored case, Miri corrects all three disagreements and records P4's human-review flag. The completed labels now match all six reference labels. The corrected labels agree with the authored reference; this does not erase the three initial disagreements or establish actual model accuracy. P4 remains held for an authorized person to assess urgency; the table does not claim the service incident is resolved.

Record corrections by type. P3 is insufficient evidence, P4 is a consequential missed cue, and P5 is a category-definition error. A prompt revision might address the last error while leaving the first two unchanged. “Three corrections” gives less useful guidance than knowing what the reviewer had to repair.

Avoid measuring quality solely through the person who produced the answer. Where practical, have an appropriately qualified reviewer compare outputs against the source and agreed criteria without relying on the tool's self-assessment. Record disagreements about the reference answer too. A mistaken reference can make an acceptable output look wrong or reward the same mistake.

The Magenta Book guidance on evaluating AI interventions discusses baseline differences and the difficulty of evaluating a changing intervention. For a small work pilot, that supports caution about comparisons. Doing the same task twice can change familiarity; different people may have different experience. These possibilities need recording before attributing any difference to AI.

If a pilot uses live timings, define whether waiting time occupies the employee or permits other work. Separate elapsed turnaround from active labor. Do not add overlapping timers as if they were independent hours, or subtract waiting time without establishing how the person actually used it.

An evaluation notebook shows a reviewed work sample beside an unfinished correction tray and a narrow gate for a further pilot

Completed decision memo: narrow the next step

Farah's finished illustrative memo reads:

Decision: Do not keep this as automatic routing. Do not claim a time saving from this exercise. Permit only the design of a narrower, fully reviewed follow-up, subject to approval before any real ticket is processed.

Evidence: The six authored records contain three initial label disagreements and one missed urgent-service cue. Record-level assisted effort totals 29 minutes versus a 26-minute baseline; eight setup minutes make assisted effort 37. Human corrections produce six reference-matching labels, but P4's escalation assessment remains pending. All numbers are invented teaching data.

Narrowing: Propose routine, single-intent feature requests only. Retain source excerpts and mandatory review. Exclude service incidents, ambiguous requests, and material not approved for the environment. This exercise does not establish that the narrowed workflow performs well; it defines a candidate for evaluation.

Next owner and effort: Miri accepts two hours to prepare permitted examples and a review rubric. Farah defers a dashboard-polish task to make that time available. Approval to prepare the design does not authorize a live tool connection, upload, or routing action.

Revisit conditions: The follow-up must retain its input and workflow versions, document review and correction effort, and show whether accepted outputs meet the rubric. An urgent cue found in an allegedly routine record returns it to human handling. An unreviewed output, inappropriate data access, or unsupported category forces a hold and owner review.

The memo makes the next action concrete without manufacturing a successful pilot. It rejects automatic routing, allows a bounded design task, and names the unresolved evidence. “Narrow” is a decision about what to evaluate next, not a favorable result for the proposed narrower workflow.

For your own decision, keep a workflow when the accepted output and total burden meet agreed requirements in the evaluated context. Narrow it when a specific limited task is worth further testing and the owner can fund that work. Stop when the required quality, permission boundary, or operational effort is unacceptable. Leave the result unresolved when the evidence is insufficient.

Before the decision review, assemble the source sample, reference rubric, workflow version, initial outputs, corrections, totals, and memo in one record. Use LandOffer.ai workplace guides to plan the next task, then ask its actual owner to review the completed record before expanding use. A small pilot is useful when it improves a bounded decision, even when the decision is to stop or investigate further.

Sources and Further Reading