1. LandOffer
  2. Blog
  3. All articles
  4. Use Canary Releases to Compare a Small Rollout with the Stable Version

Engineering Practice

Use Canary Releases to Compare a Small Rollout with the Stable Version

Define comparable traffic, observation windows and stop conditions before evaluating a canary. A complete fictional packet shows how to interpret results and record expansion, withdrawal or missing evidence.

9 min readLandOffer team

Use a canary release to compare a small, time-limited deployment with the stable version serving comparable traffic at the same time. Before exposing users, specify which requests can reach the candidate, what you will measure, how long you will observe and what will stop the rollout. Advance only when the evidence supports the next bounded step; missing traffic or missing telemetry leaves the decision unresolved.

A small rollout with no complaints offers little evidence about a rare request path. A useful evaluation can end in expansion, withdrawal or a pause for better information. This engineering guide from LandOffer.ai develops a complete fictional evaluation packet and two alternative decision records. The arithmetic is checked locally; no service, deployment controller or production traffic was tested.

A ceramicist compares two glazed test tiles beside a kiln and shelves of ceramic vessels.

Compare the change with a credible control

Google's SRE workbook describes canarying as a partial deployment evaluated within a limited period. Its discussion emphasizes representative traffic, meaningful health signals and differences that can be attributed to the change. A service-wide average alone can hide a regression affecting only the new version.

Label observations with the actual version and relevant request class. Confirm that the stable population really runs the intended baseline. A previous day's baseline may differ in load, dependencies or customer mix. Simultaneous observations help, but do not automatically eliminate those differences. A canary confined to one unusually fast machine is a poor comparison with a control spanning several overloaded machines.

Separate this release question from feature exposure. A feature flag can control who sees a behavior within an already deployed version. This article instead evaluates a candidate version against the stable deployment. Flags may be part of an implementation, but changing a flag does not establish that a binary rollout was evaluated.

Before using a canary, identify what cannot be safely reversed. A schema migration, external notification or irreversible write may survive a routing rollback. The worked example deliberately uses a read-only endpoint and retains the stable artifact. That narrower situation does not establish a recovery procedure for a payment or data-migration service.

Complete the evaluation packet before the first request

Everything about PageKiln, its people and its measurements below is authored fiction. The service renders book-cover thumbnails. Candidate thumb-r18 changes image-decoder settings; stable thumb-r17 remains available. Ordinary previews and signed previews use different request paths, so a large ordinary-preview sample cannot stand in for the signed path.

Release owner Mira and on-call engineer Jalen agree on this completed packet. Their thresholds are teaching assumptions, not recommended values for another service.

Packet field Completed agreement
Initial exposure Five percent of eligible requests to r18, 95 percent to r17, using a fictional request-level split that includes both preview paths
Comparable scope Same region, image-size eligibility, dependency configuration and observation clock; confirm actual version labels before evaluation
Exclusions Oversized originals remain outside this release; their traffic is tracked separately and cannot be used to justify expansion to that population
Initial settling period Five minutes identified separately, followed by two complete, consecutive 15-minute evaluation windows; settling-period harm still counts for operational response
Required coverage At least 1,000 canary requests and 100 signed-preview requests within each evaluation window; signed requests are a subset of the total
Error guardrails Canary render-error rate no greater than one percent, and no more than 0.30 percentage points above the simultaneous control rate in either window
Latency guardrail Canary p95 no more than 1.20 times the control p95 for the same window and measurement definition
Immediate stop Confirmed incorrect-image exposure or another severe user-impact signal triggers the incident route without waiting for a completed window
Information failure Missing version labels, missing measurements or unrepresented required paths mean no advancement; inspect whether continued exposure is appropriate
Decision authority Mira approves advancement; Jalen can stop exposure and invoke the approved recovery procedure when a stop condition occurs
Recovery preparation Stable artifact retained; routing procedure, capacity and decoder compatibility checked in the fictional preflight; no schema or persistent-data change in scope
Next step Ten percent only, followed by a new evaluation under an explicitly reviewed plan; this packet does not approve full rollout

This packet gives the next operator a concrete comparison to review. Someone arriving during the evaluation can identify the decision owner and tell which result would stop the exercise. They can apply the recorded criteria instead of guessing whether a dashboard color authorizes more exposure.

Counts and elapsed time answer different questions. Fifteen minutes may include thousands of ordinary previews yet almost no signed requests. If a window has only 40 signed requests, its coverage requirement fails even when the aggregate error rate is low. The team must decide whether to wait within an approved exposure limit, change the design or withdraw. It cannot silently drop the missing path from the criteria.

The numbers also need a stable meaning. Here, a render error means a failed rendering request under one fixed classification; retries remain separate request attempts in both populations. User-level success would require another denominator. The p95 values are authored summaries, not percentiles calculated from raw durations in this article, and the two population percentiles must not be averaged into a supposed overall p95.

Read a filled comparison, including its limitations

These are alternative fictional outcomes under the same packet, not successive stages of a real rollout. Outcome A has enough relevant traffic in both windows. Outcome B shows an adverse first-window result and stops there.

Outcome and window Canary requests / errors Control requests / errors Signed canary / control requests Canary / control p95
A, 10:05–10:20 1,000 / 4 19,000 / 76 120 / 2,280 220 / 210 ms
A, 10:20–10:35 1,200 / 6 22,800 / 91 140 / 2,660 235 / 215 ms
B, 10:05–10:20 1,000 / 20 19,000 / 76 120 / 2,280 320 / 210 ms

In A's first window, both populations have a 0.40 percent error rate. Canary latency is about 1.05 times control latency. In its second window, canary errors are 0.50 percent against approximately 0.40 percent for control: about 0.10 percentage points higher, within the assumed 0.30-point limit. The latency ratio is approximately 1.09. Both windows meet the stated counts and these numerical guardrails.

The authored counts preserve a 5:95 traffic split in each row, but matching totals cannot verify a real router or prove that requests were assigned comparably. The signed-path shares match between populations in each authored window; that coarse match does not show that every important signed-preview variant appeared. Mira's review must still inspect whether the observed mix covers the intended scope and whether correctness signals are usable.

B's canary error rate is 2.00 percent against 0.40 percent for control, an excess of 1.60 percentage points. Its latency ratio is approximately 1.52. Those values fail both error guardrails and the latency guardrail. There is no reason to wait for a second completed window to make this packet's withdrawal decision.

A small Python calculation verifies these rates, ratios, coverage checks and alternative decisions using the authored numbers. It sends no requests, runs no controller and establishes no statistical confidence. Passing that calculation means the article's record is internally consistent; it does not establish that an actual release is healthy.

A cracked tile labeled HOLD sits beside an intact CONTROL tile, while a hand rests on the wooden gate in front of more tiles.

Record expansion and withdrawal as different decisions

Outcome A: advance one step. At 10:38, Mira records: “Approve r18 at ten percent for the same eligible population. Two evaluation windows met the agreed numerical and coverage conditions. No correctness alert is recorded in this fictional packet. Larger images, other regions and full deployment remain unapproved.” Jalen owns the next routing step; Mira retains the advancement decision.

The handoff names the two window records, version identities and still-excluded population. It also requires review of sample expectations and stable capacity for the next stage. A ten-percent stage changes the exposure and potentially the workload distribution; the earlier comparison cannot be copied forward as proof that the next stage has passed.

Outcome B: withdraw and investigate. At 10:21, Jalen records: “Stop further r18 exposure under the recovery procedure. Error and latency criteria failed in the first completed window. Mira owns the release hold; decoder configuration is a hypothesis, not an established cause.” The recorded decision does not mark the incident resolved or the earlier user impact repaired.

In this alternative scenario, the routing record then shows zero new requests directed to r18 and the retained r17 serving new eligible traffic. That observation answers the routing question only. In-flight requests, any cached incorrect results and user-visible recovery still need their own checks. The record leaves those checks open rather than declaring recovery from a command's successful exit.

Mira assigns Jalen the version-specific error breakdown and a second engineer, Rosa, the failing-image reproduction. Another release candidate requires reviewed findings and a fresh evaluation. Keep the observed failure, evidence owner and next decision in the investigation task, with the decoder explanation still marked as a hypothesis.

Treat missing or shared failures honestly

An empty measurement is not a zero-error observation. It might mean no requests, a label mismatch, a delayed metric or a broken query. Keep the information failure visible and decide whether the existing exposure should remain. Do not extend a partial rollout indefinitely because the planned window never became evaluable.

Argo Rollouts' analysis documentation distinguishes successful, failed and inconclusive runs, with different consequences for rollout progression. It also documents explicitly configurable handling for empty results and nonnumeric values. A controller's success label therefore depends on its configured rules; inspect them before treating it as evidence of healthy traffic.

If both versions degrade together, investigate shared dependencies and load. Similar error rates do not justify advancement while the service is failing an absolute health requirement. Conversely, a canary-only difference still needs examination of request mix and infrastructure before assigning a cause. Stop decisions can be prudent before the root cause is known.

For another service, choose coverage and duration from its risks, traffic patterns and established health objectives. A low-volume business process might need an observation opportunity that this thumbnail example never exercises. Statistical design also deserves separate attention when small differences drive a decision. These illustrative thresholds provide neither a significance test nor a universal safe sample size.

Connect the evidence to the actual release mechanism

A written plan must match the system that applies it. Google Cloud Deploy documents staged canary percentages and optional phase analysis, while traffic-management details depend on the deployment target and strategy. That description is not evidence that PageKiln uses Cloud Deploy, or that a particular reader's pipeline has analysis and safe recovery configured.

Check which identity advances a stage, which failures halt it, whether a manual override exists and where its reason is recorded. Confirm that the retained stable version can handle the returned load. Keep permission to stop, permission to advance and permission to change the evaluation rules explicit; a hurried operator should not have to rewrite a threshold to make a release proceed.

For your next review, complete one packet with the actual version pair, comparable population, missing-data behavior and named decision authority. Then attach a filled comparison and a bounded advance, hold or withdrawal record. LandOffer.ai publishes this guide as practical engineering reading; bring the completed comparison and its unresolved questions to the person authorized to approve the next exposure step.

Sources and Further Reading