Data & Decisions
Late-Arriving Events Can Change Yesterday's Dashboard: Plan the Data Window
Separate occurrence time from record availability when daily totals change. A completed inspection-form example and recomputation agreement show provisional counts, duplicate handling, routine closure and a versioned correction after the reporting horizon.
Yesterday's dashboard can change when events arrive after its first calculation. Group the metric by the agreed event time, retain arrival and processing evidence separately, and state how long normal recomputation continues. Give later corrections an explicit route instead of silently dropping them or presenting an early count as final.
This LandOffer.ai guide includes a completed fictional daily-metric example, a recomputation policy and an executed local fixture. No stream processor, watermark, warehouse, dashboard or source account was operated. All events, owners and reporting decisions are authored; the October 29 and October 30 records below are explicitly hypothetical future continuations of the example.

Separate the event's date from the time you learned about it
Name the meaning of each timestamp before using it to assign or qualify a daily count. Event time represents when the underlying occurrence happened. Arrival time records when a particular input source received the record. Processing time describes when a processing stage observed or worked on it. A later dashboard refresh is another timestamp again.
Azure Stream Analytics' time-handling documentation distinguishes these concepts and supports arrival-time or application-time processing. In that product, using the payload's application timestamp requires the documented TIMESTAMP BY choice; an event-time field merely existing in a record does not establish how a job groups it.
For fictional FieldAtlas, a site-inspection tool, the metric is the number of distinct completed inspection forms whose recorded completion time falls in a UTC day. The business-day convention is already agreed and stays unchanged throughout this case. We are examining when records become available, not changing time zones or redefining “completed.”
Every retained record has event_id, form_id, occurred_at and arrived_at. The fixture uses arrival time as the local availability cutoff and performs recomputation directly. It does not simulate a processor's processing clock, streaming watermark or dashboard cache. A real implementation should preserve the relevant timestamps at those additional boundaries.
Suppose a form completed Monday night reaches the collector Tuesday morning after a device reconnects. Counting it under Tuesday because that is when the record arrived changes this metric's question. Dropping it because Monday's screenshot already exists creates a different omission. The reporting policy should say how the Monday count is revised.
Completed example: a provisional count moves from 100 to 105
Analyst Noor owns FieldAtlas' metric interpretation. Engineer Elias owns ingestion and daily recomputation evidence. They agree that the October 26 day is the half-open interval from October 26 at 00:00 UTC up to, but not including, October 27 at 00:00 UTC.
The authored fixture contains 100 forms completed during October 26 and available before the first reporting cutoff. Five additional forms complete between 23:00 and 23:04 that evening but arrive between 10:00 and 10:04 on October 27. Each of those five records has an eleven-hour event-to-arrival delay.
Two other incoming rows repeat already seen event IDs and identical occurrence details. They are delivery copies, not additional forms. A separate malformed-clock record claims an occurrence three hours after its arrival and is held for timestamp review under the example's ten-minute future-time guard. That guard is an authored validation choice, not an Azure or Beam default.
| Published version and cutoff | Available evidence | October 26 count | Completed interpretation |
|---|---|---|---|
| v1, October 27 at 08:00 UTC | First 100 forms | 100 | Provisional; routine late-arrival window remains open |
| v2, October 27 at 12:00 UTC | Five new form IDs; two exact delivery copies excluded; future-clock record quarantined | 105 | Provisional revision of +5 forms, or +5% relative to v1 |
| v3, hypothetical October 29 at 00:00 UTC | Same 105 valid distinct forms at routine close | 105 | Settled under the stated routine policy, not proven exhaustive |
| Exception review, hypothetical October 30 at 12:00 UTC | One more October 26 form arrives outside the routine horizon | 105 pending review | Preserve E106 as an exception; do not silently rewrite the published version |
| v4, hypothetical approved correction | Reviewer accepts that form into a corrected daily snapshot | 106 | Versioned +1 correction to the settled count; original versions retained |
The five late forms belong to October 26 because of their occurrence timestamps. Their Tuesday arrival does not establish a Tuesday business occurrence. Conversely, the quarantined record is not automatically reassigned to Monday to make the total look complete. Its timestamp problem requires separate evidence.
Noor can explain the revision by naming the newly available records and each version's cutoff. v1 accurately describes records included as of its cutoff, provided it was labeled provisional. v2 changes the available evidence, while v4 is a documented correction after routine closure.
Define a reporting window separately from a stream watermark
Apache Beam's programming guide describes a watermark as the system's notion of when window data can be expected to have arrived. Records for that window arriving after the watermark are late. Its source and windowing configuration matter; the watermark is not simply the dashboard's refresh time.
FieldAtlas' 48-hour routine recomputation horizon is a business reporting policy measured from the end of the UTC metric day. It is not an observed watermark, a measured worst-case delay or a guarantee that all devices reconnect within two days. The October 26 horizon closes at October 29 at 00:00 UTC in this authored case.
A real team should choose such a horizon using source-delay evidence and the cost of changing decisions. Examine arrival lag by source, offline-client behavior, known outages and the age of records still appearing. For an operational dashboard, routine revisions may be acceptable; a signed external report may require a separately approved correction.
Longer windows trade faster apparent stability for more routine corrections and additional recomputation or retained state. Shorter windows give earlier closure but require a credible process for records that arrive afterward. Neither choice makes late events cease to exist.
Document stream acceptance rules beside the reporting horizon so a dashboard owner can see whether an upstream policy may already have removed the records a longer lookback would need. If the stream drops a record before retained storage, extending a downstream dashboard's lookback cannot recover it. If raw records remain available, a separately authorized backfill may have options that a state-expired streaming window no longer has.
Use a completed recomputation and closure policy
Noor and Elias adopt the following fictional policy for this one metric. It is a filled agreement with named actions, not a vendor configuration applied by this guide.
| Policy area | Completed FieldAtlas decision |
|---|---|
| Metric and ownership | Noor owns distinct completed forms by recorded UTC completion day; Elias owns source lineage and recomputation outputs. Definition version stays unchanged. |
| Initial publication | Publish the next-day 08:00 UTC snapshot as provisional, with cutoff and version visible. Do not label an early total “final.” |
| Routine recomputation | Replace the affected daily snapshot from retained valid form records at review runs through the inclusive day-end-plus-48-hour cutoff; preserve prior versions. The worked second cutoff is 12:00 UTC. |
| Duplicate policy | Exact repeated event IDs with matching form and occurrence details add no count; count distinct form IDs. Conflicting identity or payload evidence requires investigation. |
| Timestamp exceptions | Hold records more than ten minutes into the future relative to arrival for review. Preserve the original fields; do not silently substitute receipt time. |
| Closure condition | At the horizon, Noor reviews source-health and unresolved-gap evidence before marking a version settled under policy. Time passing alone does not prove completeness. |
| Beyond-horizon records | Retain an exception with affected day, IDs, receipt time and proposed delta. Elias prepares reconciliation; Noor approves or rejects a versioned correction and records the reason. |
| Consumer notice | Show snapshot version, event-time interval, availability cutoff, provisional/settled/corrected status and known gaps beside the metric or in its linked definition. |
For October 30's fictional E106, arrival is 84 hours after the metric day ends. The local policy therefore leaves the settled snapshot at 105 and records an exception. Approval creates v4 at 106; the earlier count does not disappear from the audit trail. This is a demonstrated policy branch over authored inputs, not an approval received from a real manager.

Check what the processor and sink do with revised results
A dashboard can still miscount even when the late event enters the correct window. Beam documents triggers and accumulating versus discarding panes: repeated emissions may contain accumulated results or only newer contributions. The consumer must understand which kind of result it receives.
FieldAtlas' local fixture produces replacement snapshots. v1 is 100 and v2 is 105; adding those two cumulative snapshots would incorrectly display 205. Its policy replaces the selected daily value and retains version history separately. It does not append every new total as additional completed forms.
This example does not implement an actual Beam trigger, output sink, database upsert or transactional publish. A real pipeline needs a tested contract for the window key, version ordering, retry handling and whether outputs are replacement values, deltas or retractions. A label saying “late data supported” cannot settle those implementation details.
Azure's event-ordering policy documentation, updated June 11, 2026, describes product-specific choices to drop or adjust records outside configured tolerances. It also distinguishes late-arrival and out-of-order policies. An adjusted processing timestamp and the original payload timestamp need not mean the same thing for downstream grouping.
Do not transplant those product settings into the 48-hour FieldAtlas policy. Review the actual source, timestamp choice and configured behavior first. In this case, no event timestamp is adjusted, no late record is discarded automatically and no Azure or Beam configuration is changed.
Give consumers the completed explanation, not just a fresh chart
Completed fictional dashboard note: “October 26 completed inspection forms: 105, provisional v2. Event-time interval: October 26 00:00 through October 27 00:00 UTC, end excluded. Includes valid records available by October 27 12:00 UTC. Five previously unavailable forms increased v1's count by five; two delivery copies did not add forms. One future-clock record is quarantined. Routine revisions remain possible through October 29 00:00 UTC, subject to source-health review.”
Completed hypothetical correction note: “October 26 total corrected from settled v3's 105 to v4's 106 after E106's late receipt and approved reconciliation. This changes the reported count, not the metric definition or the recorded occurrence date. Earlier versions remain available. Timestamp quarantine remains unresolved.”
The companion Python fixture passes 23 assertions over the as-of selections, duplicate handling, clock guard, horizon and correction branches. It also checks that adding cumulative snapshots produces the incorrect total of 205, and that v2 has 108 raw available rows before duplicate and timestamp handling. It does not measure real delay percentiles or source completeness. No watermark advancement, stream job, warehouse recomputation or actual dashboard update is claimed.
For more practical work guides, visit LandOffer.ai. Before sharing yesterday's number, provide its event-time interval, availability cutoff, version and revision policy. Name the owner of missing or quarantined records and the person authorized to approve a corrected version. Keep that route available after routine processing closes, with the earlier published snapshot still identifiable.
Sources and Further Reading
- Azure Stream Analytics: Understand time handling — event, arrival and processing time; application-time choice and watermark behavior; freshly reviewed October 27, 2026.
- Apache Beam Programming Guide: Watermarks and late data — source-dependent watermarks, lateness, triggers and pane accumulation semantics.
- Azure Stream Analytics: Event ordering policies — updated June 11, 2026; late and out-of-order processing policies and adjusted timestamp behavior.

