1. LandOffer
  2. Blog
  3. All articles
  4. A New Feature's Early Lift Fades: Check Novelty and Learning Effects

Data & Decisions

A New Feature's Early Lift Fades: Check Novelty and Learning Effects

Compare first-exposure cohorts before interpreting a fading feature gain. A completed cohort record and decision memo distinguish early benefit, later observations and unresolved novelty or learning explanations.

10 min readLandOffer team

When a new feature's early lift fades, compare treatment effects at the same time since first exposure before deciding whether to ship. A calendar-day scorecard can mix people discovering the feature with people who have already adjusted to it. Compare equally mature exposure windows, preserve original assignment, and check whether logging or calendar events could explain the apparent decline.

A downward curve can be consistent with novelty wearing off. It does not prove that explanation, establish a steady-state effect, or show that learning has stopped. This LandOffer.ai guide uses a completed synthetic experiment packet to show how an analyst can turn conflicting short-term and later readings into a bounded decision.

A researcher compares potted plants of different ages at a greenhouse workbench.

Ask what changed: the users, the exposure age, or the world

Begin the review with four questions: Did the same eligible population receive the same assigned experience? How long had each group been exposed? Did the metric or logging change? What happened outside the experiment during those dates? These questions produce different investigations; “novelty” should not become a convenient label for every disappointing second week.

Novelty describes an initial response that fades, such as curiosity about a newly prominent control. Learning can produce the opposite shape: initial difficulty followed by improvement as people adapt. Neither shape identifies its own cause. A launch announcement, a customer deadline, a control-side change, an outage or a changing mix of customers can also move the measured effect.

Microsoft's external-validity analysis documents changing effects and explains why systematic novelty or learning trends cannot account for every time-related surprise. A precise estimate for an observed period does not automatically describe the future. Treat that as a reason to investigate the time dimension, rather than dismissing randomization or assuming that another week must settle everything.

Write the intended decision beside the metric so reviewers can see whether first-use or repeated-use evidence would answer it. A feature meant to improve repeated work needs evidence about repeated work. A temporary onboarding improvement may have value during first use even if it does not raise later activity. Calling both outcomes “retention lift” obscures what the product team actually needs to know.

Define exposure without selecting the people who responded

In the fictional example, North Quay tests a report-saving shortcut. Assignment is by user, with a persistent control or treatment allocation. The first-exposure date is the first variant-independent eligible report session, recorded before rendering either experience. Control users receive the same eligibility timestamp definition even though they never see the shortcut.

That definition matters. Defining a treatment cohort by first shortcut click would select responders after treatment. Comparing those clickers with all control users would answer a different, biased question. An analyst should also check whether earlier exposure on another device, reassignment or shared accounts make the proposed clock unreliable.

The metric is the share of assigned eligible users who complete at least one successful report save within a particular fixed 24-hour window: exposure day 0, 7 or 14. Each cohort contains 1,000 users per arm. All remain in the denominator. An inactive user contributes no success only because this fictional packet assumes complete observation; missing telemetry would remain missing and require a separate investigation.

These are individual daily windows, not cumulative success through day 14 and not total saved reports. One user with ten saves counts once in that day's numerator. The same assigned users recur at days 0, 7 and 14, so those user-window observations cannot be treated as independent samples. The table below is descriptive arithmetic, with no confidence interval, significance claim or sample-size justification.

Completed exposure-cohort record: the later gain disappears

The packet is reviewed on October 28, 2026. Cohorts A and B have reached all three selected windows. Cohort C has reached only day 0. Its later windows are not yet observable, so the analyst leaves them blank rather than treating them as zero or dropping that cohort silently.

First eligible exposure Exposure age and window start Control successes / assigned Treatment successes / assigned Difference
A: October 1 Day 0: October 1 200 / 1,000 = 20% 320 / 1,000 = 32% +12 percentage points
A: October 1 Day 7: October 8 210 / 1,000 = 21% 240 / 1,000 = 24% +3 points
A: October 1 Day 14: October 15 220 / 1,000 = 22% 220 / 1,000 = 22% 0 points
B: October 8 Day 0: October 8 190 / 1,000 = 19% 310 / 1,000 = 31% +12 points
B: October 8 Day 7: October 15 205 / 1,000 = 20.5% 235 / 1,000 = 23.5% +3 points
B: October 8 Day 14: October 22 215 / 1,000 = 21.5% 215 / 1,000 = 21.5% 0 points
C: October 22 Day 0: October 22 180 / 1,000 = 18% 300 / 1,000 = 30% +12 points
C: October 22 Day 7: October 29 Not mature at review Not mature at review Not estimated
C: October 22 Day 14: November 5 Not mature at review Not mature at review Not estimated

An exposure-age plot of these rows places age on the horizontal axis and the treatment-minus-control difference on the vertical axis. A and B each run from +12 to +3 to 0 points; C has a single +12-point marker, with no connecting line into unobserved time. Retaining the cohort labels makes the repeated shape visible without inventing a completed curve for C.

For a matched-cohort summary, use A and B at every age. Their pooled day-0 rates are 390/2,000, or 19.5%, for control and 630/2,000, or 31.5%, for treatment. At day 7 they are 20.75% and 23.75%; at day 14 both are 21.75%. Thus the same cohort set moves from +12 to +3 to 0 percentage points.

Adding C to day 0 gives 19% versus 31%, also +12 points. That happens to match here; it does not make unequal-maturity comparisons generally safe. The packet contains 6,000 distinct assigned users across three cohorts. Its seven observed rows contain 14,000 repeated user-window observations across both arms, not 14,000 independent users.

Check the calendar alongside exposure age

The two clocks help distinguish questions but do not fully separate causes. On October 8, A is at day 7 while B is at day 0. On October 15, A is at day 14 while B is at day 7. Those overlaps permit a useful descriptive check: the younger cohort has the larger measured gain on each shared date in this constructed example.

In this constructed packet, the shared-date comparisons are consistent with an exposure-age pattern, though they cannot isolate its cause. It still does not identify curiosity as the cause. Cohort membership was determined by arrival dates rather than randomized between arrival dates. Customer composition, acquisition source, prior experience or tasks could differ. Repeated patterns across cohorts strengthen a diagnostic account; they do not remove every competing explanation.

Before interpreting the pattern, the analyst records whether the assignment ratio, eligibility event and successful-save event stayed consistent. The fictional packet marks those definitions unchanged and its observation windows complete. It does not claim that anyone inspected a real production system. In a workplace analysis, retain the actual logs or review references supporting each check, and leave an unverified check open.

Also inspect interruptions and feature changes. A treatment correction halfway through creates versions, not one stable treatment simply aging. A save-service outage can alter the numerator without changing the shortcut's usefulness. If telemetry failed, replacing absent events with zero would manufacture a behavioral decline. Repair the measurement account before presenting a psychological explanation.

Amber glass strips with fading color rest on an oak rack beside a brass lens and blue envelope.

Completed decision record: keep the short-term result, narrow the claim

North Quay's product manager, Leila, initially requested general release because first-use success was higher. Analyst Omar presents the matched A/B results; engineering owner Tess confirms that ongoing maintenance would still consume team capacity. The following is an authored fictional review record, not an actual release decision or an experiment conducted on a product.

Decision field Completed October 28 review record
Product question Does the shortcut improve repeated successful report saving enough to justify maintaining it? First-use assistance is a separate possible benefit.
Supported observation A and B show +12 points at day 0, +3 at day 7 and no observed difference at day 14. C supports another first-use reading only.
Unresolved interpretation Novelty is plausible. Learning, task changes and cohort differences have not been causally isolated. Day 14 is a later reading, not established steady state.
Immediate disposition Leila withdraws the proposed broad repeated-use claim. The review does not approve general release or declare the feature worthless.
Measurement responsibility Omar owns checking the cohort construction and observation completeness before proposing any inferential analysis. The analyst must account for repeated users and the original randomization unit.
Product responsibility Leila owns deciding whether a first-use benefit alone warrants a differently scoped product proposal, including its maintenance cost.
Engineering responsibility Tess retains the experiment's existing operating constraints. A longer holdout, exposure change or new version needs the normal authorized experiment decision; none is recorded as executed here.
Revisit condition Return with the missing mature windows and reviewed measurement checks under an agreed analysis and stopping plan. No one promises that collecting more days will produce a favorable result.

The review retains the observed early gain and withdraws the unsupported durable-use claim; neither decision establishes the feature’s eventual value. Omar does not average away the later zero, and Leila does not recast the original metric after seeing the result. A new onboarding-focused proposal would need its own question, rather than retroactively changing what this experiment was supposed to establish.

An actual team would also retain any relevant reliability, accessibility, support-load and cost measures. None was generated for this example. Their absence prevents a complete launch assessment; it does not license an assertion that the feature is safe because the success metric rose initially.

Choose further analysis without chasing a better-looking day

The authors' novelty and primacy research abstract describes estimating longer-term effects while considering other time interactions. That is a methodological lead, not a plug-in conclusion from this table. Applying such an estimator requires its assumptions and full method to be reviewed; this example has not reproduced it.

Similarly, Microsoft's long-term experiment publication flags survivorship, selection and identity stability as possible pitfalls. Keeping only users still active at day 14 can change the comparison. Long duration alone does not fix a population definition that changes with treatment response or loses people whose identifiers reset.

Agree the next decision window and analysis approach before examining additional favorable days. Preserve the original assignment, define which cohorts will be mature, and involve someone who can handle repeated measurements and the chosen stopping rule. Do not repeatedly inspect conventional significance tests until one passes, or count correlated daily rows as independent replication.

If the curve instead starts negative and becomes positive, investigate learning with the same discipline. Users may be adapting, but a repaired bug or changing task mix could yield that shape too. Record the treatment versions and outside events before calling the improvement a training effect.

Before the next launch review, put the eligibility timestamp, mature cohort rows and calendar events in one packet, then ask the decision owner which product claim those observations support. Keep the early result, the later result and the unresolved explanation visible together. For more workplace decision guides, return to LandOffer.ai; a clear decision packet should help its owner act without turning an observed trend into a promise about the future.

Sources and Further Reading