1. LandOffer
  2. Blog
  3. All articles
  4. A Data Pipeline Passed but the Dataset Is Stale: Check Freshness Separately

Data & Decisions

A Data Pipeline Passed but the Dataset Is Stale: Check Freshness Separately

A synthetic support-queue timeline shows a recent successful load serving an old snapshot, then separates restored current freshness from a missed delivery deadline and unresolved root cause.

9 min readLandOffer team

When a pipeline passes but the dataset is stale, check execution status and data freshness separately. Verify which source snapshot was available, what the transformation actually read, and when the consumer received data representing the expected period. A successful job may correctly transform the same old input without advancing the dataset's as-of boundary. A recent load timestamp can also describe old business data that was copied again.

This LandOffer.ai guide includes a completed job-versus-freshness comparison and a stale-data recovery record. The support-queue feed, people, jobs, and recovery events are fictional. We actually replayed their synthetic timestamp ledger locally with Python; no dbt, Airflow, warehouse, producer feed, dashboard, or alerting account was accessed or changed. Documentation was checked October 27, 2026. The calculations illustrate separate signals, not an operational test or a guarantee that any dataset is complete.

An engineer examines source folders rather than relying on a recently completed processing task

Separate execution, source currency, and consumer availability

Execution success answers whether the configured work completed under the runner's rules. It does not necessarily answer whether the expected upstream batch arrived, whether every required partition was represented, or whether the consumer received the result. An empty extraction, a reused file, or an incorrect selection window can still lead to a technically successful transformation.

Airflow's current Dag Run documentation describes run status in terms of task states and dependencies. It also distinguishes a run's data interval and logical date from actual execution time. Those are named Airflow concepts, not evidence that the fictional jobs below ran in Airflow. The practical lesson is to inspect what a status and timestamp represent before treating them as dataset currency.

Source currency needs an agreed meaning. It might be a producer-confirmed snapshot as of a particular time, a required partition, or a trustworthy watermark. The newest event timestamp alone does not prove every expected record arrived. A load timestamp identifies ingestion or rewriting activity only if that is what the field actually records.

Consumer availability is another boundary. Data can be current in a staging table while a downstream model, extract, cache, or report still serves an older snapshot. Trace the version or watermark into the consumer's actual served dataset before reporting that the data is current. Do not infer delivery from the upstream job finishing, and do not infer currency from a report page saying it refreshed recently.

Define the freshness contract before setting an alert

Write down the expected data period, producer schedule, availability deadline, time zone, relevant working calendar, and the consumer's need. Identify the authoritative field or manifest that establishes which period the data represents. Also decide what a missing timestamp, missing batch, empty but valid batch, and future-dated value mean. Record an unknown timestamp or missing batch as unresolved instead of treating it as fresh.

A quiet source is not automatically stale. Some datasets change only when a business event occurs; a complete zero-change snapshot can be current if the producer confirms the expected period. Conversely, a single new row can update a maximum timestamp while another required region or partition is missing. Choose checks that fit the contract instead of assuming that a row-count increase proves freshness.

Current dbt platform guidance states that dbt build does not itself include source freshness checks. It also distinguishes job configuration patterns: the source-freshness checkbox and an explicit freshness command step can have different effects on subsequent steps when a check fails. Verify the applicable version and job behavior rather than assuming that enabling a check automatically blocks downstream publication.

The current dbt reference also distinguishes versions: for dbt v2.0 and later it recommends dbt freshness for configured sources and models; the legacy dbt source freshness command remains supported for sources. Its configuration includes age thresholds and fields or queries used to determine loaded timestamps. These are documentation facts, not commands or YAML we tested in dbt. Our independent demonstration below uses a custom Python rule, not a reproduced dbt execution.

Completed asset: a successful job serves the old snapshot

Fictional Cedar Support uses an hourly queue snapshot for staffing discussions. Its agreement is that the snapshot representing each clock hour should be available to consumers by fifteen minutes after that hour, in UTC. For example, the 10:00 snapshot is due by 10:15. This is an illustrative business contract, not a universal deadline for support data.

Data owner Anika and pipeline owner Ravi also use an illustrative age rule for triage: a source age greater than 30 minutes is a warning; greater than 60 minutes is an error. Exactly 30 or 60 is not beyond the corresponding threshold. These chosen limits are separate from the scheduled-snapshot deadline. A warning may still accompany a missed deadline, and a normal age does not prove all expected content is present.

At 10:30 UTC on October 27, the synthetic ledger says job J1020 succeeded and wrote a warehouse load timestamp of 10:20. The consumer publication at 10:22 still represents source snapshot S0900, as of 09:00. The expected S1000 snapshot has not arrived at the consumer. The job's recent completion therefore does not satisfy the 10:15 availability commitment.

Signal at the 10:30 check Recorded value What it supports and what it does not
Job status J1020: success The configured task completed; it does not establish new source input
Warehouse load time 10:20 UTC A write occurred 10 minutes ago; this is not the source snapshot's age
Source identity and as-of time S0900, 09:00 UTC Data represents the 09:00 snapshot, now 90 minutes old
Consumer snapshot As of 09:00, published 10:22 The consumer received the old snapshot recently, not the expected 10:00 snapshot
Contract due S1000 by 10:15 At 10:30 the deadline has passed and the expected snapshot is absent
Separate freshness assessment Error under the illustrative age rule A triage signal requiring investigation, not a diagnosis of the producer's root cause

The source manifest and transformation input reference both identify S0900 in this fictional record. That localizes the immediate issue: the successful transformation read the old snapshot. It does not establish why the producer failed to supply the expected one. Ravi's next investigation is the missing S1000 producer delivery; the manifest mismatch alone does not identify an infrastructure root cause.

The consumer's publication timestamp also explains why a recent “refreshed” label would be misleading without an as-of value. The 10:22 publication does not make 09:00 business data represent 10:00. Preserve both timestamps so a reader can distinguish a fresh delivery of an old snapshot from a genuinely current dataset.

Trace the boundary that did not advance

Start at the consumer and follow the recorded snapshot or watermark backward through the serving layer, transformed dataset, raw landing area, and producer manifest. Compare actual input identifiers, expected periods, load results, and publication receipts. An end-to-end lineage statement is more useful than a collection of unrelated green task badges.

Check whether a no-change result is legitimate under the contract. Was a new manifest published for the expected period? Was the correct file or partition selected? Did a filter exclude new records? Did an upstream job produce an empty but valid snapshot, or did the consumer reuse an old one? These are different cases and should not share an automatic remedy.

In Cedar's fictional triage, Ravi confirms that the transformation selected the latest available source identifier, S0900. Anika confirms that S1000 was expected and has not yet been supplied. They record a stale-data incident and label the staffing view with its 09:00 as-of boundary. Those are completed actions within the fiction, not messages or application changes performed by this publisher.

They do not repeatedly rerun downstream jobs merely to obtain another success. Reprocessing the same old snapshot would not supply the missing period and could have other effects in a real pipeline. Any actual replay or publication needs the appropriate operational checks and authorization; this article does not provide a historical backfill or idempotency procedure.

Synthetic job and source timestamps illustrate why a recent write can still contain old data

Completed asset: recovery updates the consumer and retains the miss

The synthetic recovery record continues with two later observations. At 10:56, the ledger represents job J1050 as successful, with a 10:50 warehouse load and the S1000 source snapshot as of 10:00. The consumer received that snapshot at 10:55. Its source age is now 56 minutes, which is a warning under the illustrative rule, while the expected 10:00 period is finally present.

That publication missed its 10:15 deadline by 40 minutes. The lower age no longer meets the example's error threshold, but the batch still missed its delivery deadline. Preserve the missed commitment in the incident record and explain whether consumers made decisions while the old data was displayed. Do not invent an impact count if no one has established it.

At 11:06, the next recorded job J1104 has succeeded, the warehouse load is 11:04, and source snapshot S1100 represents 11:00. The consumer received S1100 at 11:05, before its 11:15 deadline. The current source and consumer ages are six minutes. That is normal under the example's age rule, and the named expected period is present.

Check time, UTC Job state Load age Source and consumer as-of age Expected snapshot present? Recorded disposition
10:30 Success 10 minutes 90 minutes No: S1000 missing Current freshness error; expected-period deadline missed
10:56 Success 6 minutes 56 minutes Yes: S1000 Age warning; 10:00 snapshot published 40 minutes late
11:06 Success 2 minutes 6 minutes Yes: S1100 Current age normal; 11:00 snapshot delivered before its deadline

We computed those ages and deadline comparisons with Python 3.12.14 over the original three-row CSV. The script retains the input hash, rules, timestamps, and derived results. It does not send an alert, inspect a data platform, verify a real source manifest, or recover a dataset. The script only calculates from the invented ledger; it does not establish that any job, source delivery, or consumer publication occurred.

Anika's completed fictional recovery note says: “The consumer now represents S1100 as of 11:00, available at 11:05; the current expected-period and age conditions are satisfied in the recorded scope. Keep the S1000 forty-minute lateness in the incident history. Ravi owns investigation of the delayed source arrival and will report the cause and preventive action by October 29. No staffing-impact total has been established.”

This separates restored current service from a completed root-cause investigation. It also avoids implying that a later current snapshot proves every historical row, partition, or downstream export is correct. If those checks are required for your actual consumer, keep their status visible and do not close them on the strength of the age calculation alone.

Keep freshness visible after the incident

Monitor the data condition on a cadence that can detect a breach while action is still useful. A periodic check observes a point in time; silence between checks is not evidence that nothing went stale. Route alerts to an owner who can distinguish a producer delay from transformation or serving-layer problems, and record acknowledgment and follow-up rather than assuming an alert was acted on.

Review the threshold and deadline together with the consumer's schedule. A steady hourly feed may have predictable age changes between deliveries; the warning rule should serve an agreed purpose instead of creating routine noise. Add explicit expected-batch or partition checks where a maximum timestamp could hide missing data. Keep job execution status available as one input without making it the only signal.

Adapt the comparison and recovery record to your source and consumer contract. LandOffer.ai publishes this data-work guide; it does not monitor or operate your pipelines. Give consumers the verified as-of boundary and availability time, then assign the unresolved freshness or coverage gaps to accepted owners.

Sources and Further Reading