1. LandOffer
  2. Blog
  3. All articles
  4. Use an Error Budget to Decide When Reliability Work Takes Priority

Engineering Practice

Use an Error Budget to Decide When Reliability Work Takes Priority

Connect an existing SLO to explicit reliability priorities. A filled consumption example and pause/resume records distinguish rolling budget, recent burn, reviewed exceptions and human approval.

9 min readLandOffer team

Use an error budget to change work priorities through an agreed policy: compare observed bad events with the allowance implied by an existing SLO, inspect recent error rates separately from the rolling balance and apply the recorded release restrictions or recovery conditions. Name who makes the decision, which exceptions require review and what evidence allows normal releases to resume.

The budget is not permission to cause avoidable failures, and a positive balance does not make every change safe. This LandOffer.ai guide develops a completed fictional consumption example and pause/resume record. The arithmetic was checked locally; no service telemetry, release controls, staffing plan or production system was accessed or changed.

A supervisor holds a clipboard beside a closed canal lock gate, a waiting boat and a worker at the water-level staff.

Give an existing SLO a decision policy

Google's example error-budget policy links an exceeded budget to restrictions on ordinary changes while preserving specified exceptions. It is a dated example, approved in February 2018, not a universal rule that every team must freeze the same changes or use the same escalation path.

Connect the measured budget to a specific change in release or engineering work. A budget report that leaves the team debating priorities from scratch every week provides little help during a difficult release decision. Agree on the trigger, decision authority, exceptions, reliability work and recovery conditions before the moment when a deadline creates pressure to ignore the evidence.

This article starts with an already agreed SLO. It does not select a new service-level indicator, promise a contractual availability level or change which events count after an incident. The existing definition and its measurement implementation must be available to the reviewer. If either is unclear, the apparent precision of a budget percentage can conceal a classification problem.

Keep the policy proportionate. A restriction on feature releases need not mean stopping incident response, urgent security work or an approved mitigation. Equally, calling a planned feature a reliability improvement should not bypass review. Describe the change, expected risk reduction, evidence and authority for any exception.

Work through a complete consumption record

The VenuePulse service, people, measurements and decisions below are authored fiction. Its existing request-success SLO is 99.95 percent over a rolling 28-day window. Eligible requests and bad outcomes follow the already agreed definition; this case does not redesign it. Service owner Noor and reliability lead Ellis jointly own the policy.

At the fictional October 27 review, the window contains 2,400,000 eligible requests and 1,560 bad outcomes. The allowance is the eligible count multiplied by 1 − 0.9995: 1,200 bad outcomes. Observed consumption is 1,560 divided by 1,200, or 130 percent. The balance is negative 360 events, and observed success is 99.935 percent.

Snapshot field October 27 fictional hold review November 3 authored future recovery scenario
Rolling 28-day eligible requests 2,400,000 2,600,000
Bad outcomes in that window 1,560 1,040
Allowance at the existing 99.95% SLO 1,200 1,300
Budget consumed 130% 80%
Remaining event allowance −360 260
Observed success 99.935% 99.96%
Recent observation, separately identified One hour: 120 bad / 60,000 eligible Six hours: 20 bad / 100,000 eligible
Recent error-rate burn ratio 4 times the permitted error fraction 0.4 times the permitted error fraction

The November 3 column is explicitly a hypothetical future review, not a prediction or observation available on October 27. It illustrates a later decision under the same policy. All counts in both columns are invented teaching inputs.

For a request-based objective, an event allowance is not automatically a number of downtime minutes. Requests arrive unevenly, and different failures can affect different fractions of traffic. Do not turn negative 360 events into an invented recovery duration or tell users how many minutes of failure the team is allowed to cause.

In the later scenario, 400,000 eligible requests and 700 bad outcomes leave the rolling window; 600,000 eligible requests and 180 bad outcomes enter. The resulting totals are 2,600,000 and 1,040. The allowance changes with the denominator. The calculation changes only because the defined window loses and gains events. It does not erase earlier user impact or establish that a mitigation caused the improved result.

Separate accumulated consumption from current burn

Google's SLO alerting chapter describes burn rate relative to the SLO's permitted error fraction. For VenuePulse, that fraction is 0.05 percent. In the October snapshot, the recent one-hour error rate is 120 divided by 60,000, or 0.20 percent. Dividing 0.20 by 0.05 gives a burn ratio of four.

Those 120 recent bad outcomes also equal ten percent of the current 1,200-event allowance. That is an observed contribution, not a forecast from four times burn. Traffic volume varies, so an extrapolation about when the budget will run out needs assumptions about future requests and errors. The budget is already exceeded here; no forecast is needed to trigger the fictional policy.

A cleared short-window alert does not restore the rolling balance. Conversely, an apparently acceptable rolling balance can hide a rapidly worsening current problem. Review both when deciding what work to prioritize, and keep incident-response triggers separate from the scheduled budget meeting.

In the hypothetical recovery column, the recent six-hour error rate is 0.02 percent and burn is 0.4. It is a different window from the earlier one-hour signal, so record the duration rather than presenting the two as identically measured trend points. This six-hour result contributes to the policy's recovery review; it does not establish that every failure class is fixed.

The companion Python calculation verifies the two snapshots, window-change arithmetic and burn ratios with Decimal values. Its 21 assertions also check a missing-information case without dividing by zero or assuming success. It reads no monitoring system and evaluates no production release. Passing the arithmetic is evidence about the worked example, not service reliability.

A worker inspects the closed lock gate mechanism while another carries a tool case through a separate open service gate.

Record the hold and the work it changes

Noor and Ellis agree on this fictional policy before applying the snapshot. These thresholds are authored choices for the example, not a recommendation to copy them unchanged.

Decision condition Agreed response and owner
Budget exceeds 100% consumption Noor pauses ordinary feature releases and records affected changes; Ellis leads a reliability review
Rapid or severe current user impact Use the incident process immediately; do not wait for budget exhaustion or the weekly meeting
Proposed reliability or urgent security change during the hold Noor and Ellis review the bounded change, risks, verification and recovery plan; no automatic exception by label
Missing or disputed measurement Keep uncertainty visible, assign an evidence owner and escalate disagreement to engineering director Sana; no silent exclusion or reset
Resume ordinary releases Require consumption at or below 80%, a recent six-hour burn ratio at or below one with at least 10,000 eligible requests, trusted measurements and reviewed remediation evidence
Authority to resume Noor records the decision with Ellis's evidence review; unresolved disagreement goes to Sana

At the October 27 review, Noor records: “Hold the planned event-filter feature release. The rolling budget is at 130 percent consumption. Keep the feature in development without expanding its exposure. Reassign two engineer-days from the next feature milestone to the identified reliability work.” That capacity allocation is part of the fictional record, not an actual staffing change performed for this article.

The failure breakdown attributes 960 bad outcomes to the service's queue-timeout category, 360 to a dependency-failure category and 240 to an unresolved category. Those categories sum to the full 1,560; they are not independent incident counts or proven root causes. Under this team's policy, eligible user-impacting dependency failures still count. The team does not remove them merely because another organization owns the dependency.

Ellis owns the timeout investigation and a proposed bounded mitigation. Its review must include the affected timeout cases, the change scope, verification results and a recovery procedure; those results are not produced by the arithmetic script. Engineer Mika owns evidence for the unresolved category. Noor owns communicating the feature hold through the organization's existing process. Each work item identifies the failure it addresses, its owner and the evidence needed for review. This lets the feature owner see which reliability work displaced the milestone and what will close that decision.

No messages are sent here. The completed record illustrates responsibilities and decisions; it does not claim that a real investigation, mitigation, communication or deployment occurred.

Review an exception and a later recovery decision

During the fictional hold, Ellis proposes a queue-timeout mitigation. Noor records conditional approval for that narrowly scoped change, with its normal review, test and recovery requirements still in force. The planned event-filter feature remains held. An exception is a reviewed decision about one change, not permission to restart every paused release.

In the explicitly authored November 3 scenario, Ellis presents the 80-percent consumption snapshot and the six-hour burn result. The fictional review also records trusted measurement coverage, the bounded mitigation's verification and an assigned follow-up for the unresolved failure category. These are authored evidence entries, not outputs from the local arithmetic script.

Noor records: “Resume ordinary releases through the normal release gates for the stated service scope. Numeric recovery conditions and the reviewed remediation conditions are satisfied in this scenario. Keep the unresolved-category follow-up open and review budget consumption at the next agreed checkpoint.” The decision does not mark every incident action complete or guarantee future availability.

This distinction prevents a common shortcut: waiting for old errors to leave the window and then treating the improved percentage as proof that engineering work succeeded. Aging explains part of the arithmetic. The separate remediation evidence must explain what changed and what remains unknown. A healthy-looking denominator alone should not close an investigation.

If the numeric conditions pass but measurement coverage is doubtful, keep the release decision open. If a security change cannot wait for the ordinary review cadence, use the organization's authorized urgent route and record the exception. Keep the urgent decision route available and record who approved the action, its scope and the follow-up review.

Make the policy usable without turning it into punishment

Google's implementing-SLOs guidance connects adopted objectives and an agreed budget policy to engineering trade-offs. That adoption matters: the release owner, product counterpart and reliability team must understand what the restriction changes and who can resolve disagreement. A dashboard cannot supply that authority by itself.

Discuss classification changes separately from the current release decision. Correct a genuine measurement error with evidence and a traceable explanation, but do not quietly raise the SLO allowance, narrow the population or restart the window to make a deadline. Keep the old and corrected calculations available to the reviewer where appropriate.

The policy should also preserve learning. An exhausted budget can justify changing priorities without assigning personal blame, and unused budget is not a target to spend on avoidable harm. A remaining allowance represents accepted risk under the objective; each change still needs its usual safety and operating checks.

For the next review, attach a filled consumption record, a named hold or resume decision and the evidence required for any exception. Keep missing measurements and unfinished actions visible. LandOffer.ai publishes this as practical engineering reading; take the completed consumption, exception and recovery records to the named policy owners. Keep their actual release approval separate from an arithmetic pass.

Sources and Further Reading