Job search
Product Analyst Portfolio: Design an Experiment and Explain an Inconclusive Result
Define a product experiment before seeing results, calculate uncertainty from fictional inputs and explain why an inconclusive readout can still support a clear recommendation.
A product analyst portfolio can demonstrate sound judgment even when an experiment does not produce a winner. Before inspecting results, record what evidence would justify rollout, how accounts would be assigned and measured, and how uncertainty will affect the decision. An inconclusive result should lead to a bounded recommendation, not an invented success story or a claim that the feature has no effect.
This guide uses entirely fictional inputs for an offline teaching exercise, not real users or a deployed experiment. The calculations are reproducible, but the assignment and logging checks are stated assumptions rather than observed production behavior. LandOffer publishes this guide for candidates comparing recent jobs; no hiring outcome is promised.

Start with a product decision rather than a dashboard
Choose one modest product change and explain how it might change behavior. For example, a personal reading app might replace an ambiguous save control with a clearer instruction. The business question is whether the change helps new accounts save something useful, without disrupting their first session. A chart of clicks alone cannot settle that question.
Define what the analyst recommends and who would approve a rollout. In a portfolio, both can be hypothetical roles, clearly labeled. Your contribution is the design, analysis and recommendation; do not imply that you operated a production experimentation platform or approved a product release if you did neither.
Microsoft Research’s pre-experiment guidance, published July 31, 2020 and reread October 22, 2026, connects hypotheses, success metrics, power and the choice of assignment unit. Use those principles to explain your choices. Its platform practices are context, not a claim that your small portfolio reproduces Microsoft’s infrastructure.
Keep the causal question separate from descriptive analysis. Comparing accounts that chose to click with those that did not is observational: motivation may differ between them. A randomized comparison instead assigns the offered experience before that choice. It still requires credible implementation, measurement and assumptions about interference.
Complete a design record before the fictional readout
The following completed preregistration-style record belongs to an invented project called Shelf. It is a finished portfolio specimen, not a registration submitted to an external registry or an experiment run on people. The designer deliberately chooses a limited pilot and acknowledges its poor precision.
| Design field | Finished decision for Shelf |
|---|---|
| Question | Does clearer save-control wording increase first saves within 24 hours of assignment? |
| Treatment | Change only the instruction beside the save control; preserve placement, functionality and the rest of onboarding. |
| Hypothesis | Clearer wording increases the share of eligible accounts with at least one successful save. |
| Population | Newly created personal accounts entering their first reading session; exclude staff and automated test accounts using rules fixed before assignment. |
| Assignment | Stable account identifier, one allocation per account, nominal 50:50; retain the same variant across sessions. |
| Primary metric | Accounts with a first successful save within 24 hours divided by all eligible assigned accounts with a mature 24-hour observation window. Count each account once. |
| Guardrail | Accounts experiencing at least one session-ending application error within that window, divided by the same assigned-account population. |
| Practical criterion | A three-percentage-point improvement is the hypothetical minimum benefit that would justify implementation and maintenance. |
| Analysis | One fixed final readout; two-sided 95% large-sample normal interval for the primary difference, plus the stated guardrail interval. No repeated significance-based stopping. |
| Decision | Recommend rollout review only if the primary interval is wholly above zero, the observed improvement is at least three points, and the guardrail’s upper interval limit is below a 1.5-point increase. Otherwise hold. |
The decision rule is deliberately conservative and belongs only to this exercise. The three-point business target is not a calculated minimum detectable effect. Likewise, the guardrail tolerance is an invented decision threshold, not a universal safety standard or a promise that any small error increase is acceptable.
Finished timing and sample appendix. “Enroll eligible accounts for seven full days, stopping earlier only if the budget cap of 800 assigned accounts is reached. Close enrollment at whichever comes first, then wait until the final account’s 24-hour window matures. This rule is fixed before outcomes are examined. If fewer accounts arrive, report the achieved sample and wider uncertainty. Do not keep enrolling until a preferred p-value appears.”
The cap does not guarantee precisely 400 accounts per arm under ordinary independent allocation. The later arithmetic uses an invented balanced realization of 400 each. Real assignment counts would be analyzed as observed, and a formal allocation-ratio check would assess compatibility with the configured allocation. Even a perfectly balanced count cannot verify variant persistence, event delivery or correct inclusion of every assigned account.
Finished precision note. “Assuming a 20% save rate in both arms and 400 independent accounts per arm, the approximate 95% interval half-width for their difference is 1.96 × square root of (2 × 0.20 × 0.80 / 400), or 5.54 percentage points. This pilot is too imprecise to reliably resolve a three-point business target. It can demonstrate the proposed measurement and decision process, but is not presented as a sufficiently powered launch test.”
This precision note lets a reviewer see why the pilot cannot settle the proposed rollout decision before any result arrives. For a confirmatory study, specify the baseline, the effect to detect, significance level, desired power and allocation before estimating required accounts and usable traffic. A desired interval width is a different planning calculation; do not label it “80% power” without computing that quantity.
Explain the measurement contract and validity checks
Before interpreting a result, describe the records a real implementation would require: eligibility, assignment time, stable account identifier, variant, successful save and session-ending error. Define duplicate handling and the observation-window boundary. An account with several saves still contributes one primary success; several errors still contribute one guardrail event.
The intended comparison includes assigned eligible accounts, not only accounts that interacted with the new control. Removing nonclickers after assignment can select different kinds of accounts in each arm and change the question. Missing telemetry also needs investigation; treating a lost event as a genuine absence of behavior can bias the comparison.
Microsoft’s during-experiment guidance, published January 25, 2021, distinguishes data-quality checks, overall outcomes, feature diagnostics and guardrails. It also warns that repeated examination requires appropriate statistical methods. Monitoring serious operational problems is different from repeatedly declaring efficacy with an ordinary fixed-horizon test.
For Shelf, a real analyst would check allocation ratios, variant persistence, event joins and mature windows before releasing a readout. The synthetic totals below assume those conditions; they do not prove them. No actual account-level assignment file, instrumentation test or missing-event audit exists in this teaching case.

Work the inconclusive result all the way to a recommendation
The second completed asset is Shelf’s fictional final analysis. It follows the design rather than retrofitting the hypothesis to attractive numbers. All 800 account counts and outcomes are invented; there is no customer dataset or claim of measured product impact.
| Outcome after mature windows | Control | Treatment |
|---|---|---|
| Eligible assigned accounts | 400 | 400 |
| Accounts with at least one successful save | 80 | 84 |
| Accounts without a successful save | 320 | 316 |
| Accounts with a session-ending error | 12 | 16 |
Save success and error are separate binary outcomes. An account could belong to both groups; do not add the two outcome counts to construct a denominator. The primary rates are 80/400 = 20% and 84/400 = 21%. Treatment minus control is one percentage point, equivalent to a 5% relative increase from the control rate. Those descriptions are numerically compatible, but “5% lift” without its denominator can exaggerate the impression.
Here is the complete uncertainty calculation for the primary difference. Let pC = 0.20 and pT = 0.21. The unpooled standard error is square root of [pC × (1 − pC)/400 + pT × (1 − pT)/400] = 0.028544. The approximate 95% normal interval is 0.01 ± 1.96 × 0.028544, or −4.59 to +6.59 percentage points.
The separate pooled test of equal proportions uses (80 + 84)/800 = 0.205 in its standard error. Following the large-sample test described in NIST’s Dataplot reference, z is approximately 0.3503 and the two-sided p-value is approximately 0.7261. This article’s interval calculation is the explicitly stated unpooled normal approximation, not a claim to reproduce every Dataplot confidence-limit option.
These calculations assume independent account-level binary observations and adequate counts for the approximation. They are not appropriate unchanged for clustered organizations, repeated session rows treated as independent accounts, or tiny rare-event samples. A portfolio should explain the selected method’s limits rather than silently applying one formula to every metric.
Completed primary interpretation. “The observed one-point difference is below our three-point practical target. Its interval spans both a decrease and an improvement larger than that target. The prespecified efficacy condition is not met. We have not established benefit, equivalence, or absence of an effect; this small pilot leaves meaningful possibilities unresolved.”
The guardrail also remains unresolved. Its rates are 12/400 = 3% and 16/400 = 4%, a one-point increase. Applying the same unpooled approximation gives an interval of −1.55 to +3.55 points. Its upper limit exceeds the invented 1.5-point tolerance. The point estimate alone does not demonstrate acceptable guardrail performance, and the interval does not prove an actual error increase either.
Finished recommendation memo. “Hold broad rollout. The primary success condition was not met, and the guardrail criterion was not cleared. Keep the existing wording while the hypothetical team reviews whether a larger study is worth its cost. First verify the proposed event definitions and assignment implementation in a legitimate test environment. Then obtain a properly sized confirmatory design and explicit operational monitoring rules before collecting new evidence. Treat this pilot as inconclusive, not as a winning feature or proof that wording never matters.”
The memo supplies a product action without pretending uncertainty has disappeared. “Hold” is a decision under the stated rule; it is not a statistical finding that control is better. Another team could choose a different rule for different costs, but that would require an explicit rationale rather than rewriting this record after seeing its results.
Show what the result cannot support
Do not salvage a positive headline by choosing a favorable device segment after the primary result disappoints. Exploratory slices may suggest a new hypothesis, but searching many groups increases opportunities to find chance patterns. Label them exploratory and design a separate follow-up instead of presenting an unplanned slice as the original test’s winner.
Also avoid generalizing a 24-hour first-save outcome into long-term retention, revenue or satisfaction. The treatment might affect those outcomes differently. Shelf has no measurements for them, so its finished readout leaves them unknown. Leave retention, revenue and satisfaction explicitly unmeasured instead of inventing outcomes to make the case look comprehensive.
Make the final presentation easy to inspect: a short question, the dated design version, metric definitions, input totals, formulas, interpretation and recommendation. Include the fictional-data label beside the results, not only in a distant disclaimer. State which parts you calculated and which production conditions you assumed.
Use the portfolio to demonstrate analytical responsibility
A reviewer should be able to trace the final recommendation back to the original decision rule. Explain why your primary metric represents the intended behavior, why the assignment unit fits the product and why the available sample limits the conclusion. That reasoning is more useful than a polished dashboard without an accountable decision.
If asked what you would improve, identify the next dependency: more precise sizing, a better measurement contract or a revised hypothesis. Do not promise that a longer test must become positive. When comparing roles, browse recent roles on LandOffer.ai and compare the role’s experiment-design, measurement and decision-communication responsibilities with the work you can substantiate in this case.
Sources and Further Reading
- Microsoft Research: Pre-Experiment Stage, published July 31, 2020; checked October 22, 2026. Hypothesis, metric, power and assignment-unit guidance.
- Microsoft Research: During-Experiment Stage, published January 25, 2021; checked October 22, 2026. Metric taxonomy, data trust and repeated-analysis cautions.
- NIST Dataplot: Difference of Proportions Hypothesis Test, page last updated November 9, 2015; checked October 22, 2026. Large-sample pooled test reference; the fictional inputs and stated unpooled interval calculation are supplied here.


