Data & Decisions
Randomize Users, Accounts, or Teams? Choose the Unit Before the Experiment
A filled shared-queue experiment design separates assignment, observations, outcome weighting and launch authority. Actual grouping checks on deliberately authored user, team and account maps expose mixed within-account labels while leaving cross-account interference, power and launch unresolved.
Choose the randomization unit by asking who shares the treatment and who can affect someone else's outcome. A user-level split can suit an isolated personal experience. A shared account setting or cross-team workflow may require assigning the whole account together. Before launch, specify what outcome will be compared, how accounts will be weighted and how the analysis respects assignment; more logged user events do not create more independently assigned accounts.
This LandOffer.ai guide includes a complete fictional collaboration-product design and an actual local check of deliberately authored assignment maps. The maps demonstrate conflicting variants inside shared accounts; they are not random draws, real exposure records or an experiment. No feature flag, customer account, product configuration or live traffic was changed, and no effect, significance or power was estimated.

Find the boundary of the shared experience
Microsoft Research's tenant-randomization article describes enterprise collaboration situations where different variants inside one organization can create an inconsistent experience. It also explains why cluster assignment changes variance, power and balance considerations. The answer is not to select the largest unit automatically; identify the feature's actual sharing boundary.
Start with a concrete treatment. A personal font setting may affect only the reader. A shared approval queue can change which work everyone sees. A team-specific workflow could remain isolated, or it might write into the same account-wide state used by other teams. Check whether teams share affected state rather than assuming that separate team names isolate the experience.
Draw who reads and writes the affected object. Include administrators, guests, shared notifications and background processing. Ask whether a control participant can receive a treatment-created artifact, whether one person's actions change another's choices, and whether an account can use two versions of the setting simultaneously. These questions describe possible interference; answering them requires product knowledge rather than a larger event count.
Microsoft's pre-experiment guidance highlights identifier stability, network effects and enterprise constraints when choosing a unit. A cookie, logged-in identity, team and tenant answer different assignment questions. Changing devices or membership can matter even if a bucketing function is deterministic.
Completed design: a shared approval queue
The fictional product SlateDock offers approval queues to customer accounts. Each account contains several teams, but the proposed treatment changes the account's shared routing state: pending items enter a new priority queue visible to all authorized teams. The control retains the existing queue. This behavior is authored for the example, not observed in a real product.
Research lead Anya and engineer Leon review the proposal before launch. The hypothesis is that the new routing arrangement improves timely handling of eligible approval requests without increasing incorrect approvals. They choose the account as the proposed assignment unit because both teams can read and update the same queue. They do not choose teams merely because team IDs are easier to export.
Their completed design record reads as follows:
| Design field | Recorded choice and boundary |
|---|---|
| Treatment and control | New shared priority routing versus existing account-wide routing; no additional simultaneous notification redesign is included. |
| Proposed assignment unit | Stable account ID, with all sessions in an assigned account receiving its variant for the experiment. |
| Why user assignment is rejected | Different users would act on one shared routing state; one user's treatment action could change what a control user sees. |
| Why team assignment is rejected | Teams within the account share the queue, so team boundaries do not isolate the treatment. |
| Observation unit | Individual eligible approval requests, retaining account ID and assignment context in the proposed event record. Repeated requests are not independent randomization units. |
| Primary outcome | For each account, proportion of eligible requests approved within 24 elapsed hours, including unsuccessful requests in the denominator; average those account proportions with equal account weights. |
| Eligibility and timing | Determine account eligibility before assignment using pre-treatment criteria. Define a common request-accrual window and allow each included request its full 24-hour outcome window. Exact dates remain pending feasibility review. |
| Guardrails | Track incorrect approvals and routing errors with agreed definitions; review missing assignment IDs, inconsistent variants and exposure logging before interpreting outcomes. |
| Analysis and weighting | Analyst Anya must specify an account-level analysis consistent with assignment and the equal-account target. User-event independence is not assumed. |
| Launch authority | Leon accepts preparing identity and shared-state checks. Anya accepts reviewing historical account counts, sizes and outcome variability. Release owner Farah keeps launch pending those checks and a completed analysis plan. |
The primary outcome describes the average eligible account's request-handling proportion. It differs from pooling all requests across all accounts, which gives high-volume accounts more weight. Neither target is automatically correct; select the one that answers the product question and retain it when interpreting the result. An account with no eligible requests has no defined request-success proportion. Anya must resolve that case in the analysis plan before launch, since treatment could itself affect request volume; she cannot silently drop zero-request accounts after seeing the results. This unresolved rule is another launch blocker.
That timing rule prevents late-arriving requests from being counted as failures before they have had the promised opportunity to complete. A request entering one minute before the accrual window closes cannot immediately be labeled unsuccessful. The plan must state how it handles observation time and unresolved requests before the team sees treatment results.
Demonstrate mixed assignment before exposing customers
The local fixture has six synthetic accounts, A1–A6, with two users and two scoped team IDs per account. Each toy team contains one listed user; this simplified structure makes the sharing conflict visible, rather than modeling the size distribution of a real customer base. All accounts share state internally under the fictional product definition.
Three deliberately authored maps contrast possible assignments. T and C are treatment and control labels only. The user map gives the first listed user T and the second C. The team map gives each account's Red team T and Blue team C; actual fixture keys include the account prefix, such as A1-Red. The account map assigns A1–A3 to T and A4–A6 to C. None of these maps was generated by a randomization procedure.
| Account | User-map variants inside its shared queue | Team-map variants inside its shared queue | Account-map variant for both users |
|---|---|---|---|
| A1 | T and C | T and C | T |
| A2 | T and C | T and C | T |
| A3 | T and C | T and C | T |
| A4 | T and C | T and C | C |
| A5 | T and C | T and C | C |
| A6 | T and C | T and C | C |
The saved grouping script was actually executed in Python 3.12.14 against the authored maps; it checks label consistency within account IDs, not real product exposures. It finds six accounts containing mixed variant labels under the user map, six under the team map, and none under the account map. There are six assigned accounts, three per arm, and twelve user memberships, six per arm. Twelve memberships do not become twelve independently assigned units under the account mapping.
The harness completes 16 checks covering membership counts, scoped team IDs, grouping, arm counts, stable account lookup and rejected missing identifiers. It does not execute a product experience or measure spillover. “Zero mixed accounts” means zero within this authored mapping, not proof that a real account-level experiment is uncontaminated or well powered.
Check identity transitions and interference outside the account
Account assignment resolves the illustrated within-account conflict, but additional product behavior could cross that boundary. A consultant may work in two accounts. An administrator may copy a treatment-generated template into another customer workspace. A shared support or notification process may affect both arms. Examine those paths before declaring account boundaries sufficient.
The fixture performs one limited lookup check: an account context of A2 returns T, while A5 returns C. It illustrates how one hypothetical multi-account guest could encounter different variants in different contexts. It does not claim to have created a guest session or verified login, navigation or persistence behavior.
Leon records three unresolved design questions: can a guest carry approval state from A2 into A5, can a shared template transfer treatment behavior, and does the assignment service use the active account ID on every evaluation? He will investigate them using the team's approved test process. The design cannot assume these answers merely because every row has an account ID.
The local lookup rejects a missing, empty or unknown account ID rather than falling back to a user-based variant. That is a deliberately chosen fixture behavior, not a universal production error-handling recommendation. For a real product, define the safe experience and logging for missing identity with its owners; avoid silently introducing a second assignment scheme.
Record membership changes, account merges and new-account entry rules as part of the design. Freezing a mapping does not freeze the real organization. Whether to retain, exclude or otherwise handle a changed account requires an explicit pre-specified rule consistent with the question, not a convenient choice made after examining outcomes.

Match analysis to assignment and budget the sample cost
Statsig's legacy Warehouse Native documentation gives a named platform example of organization-level assignment with user-level outcomes, and describes handling differing assignment and analysis units. Its specific instructions and statistical backend are documentation for that product context, not verified settings in SlateDock or a promise that every experimentation platform supports the same method.
For this design, Anya keeps the chosen account-level outcome and its weighting explicit. If the team instead wants a user-weighted or request-weighted outcome, she must revise the estimand and variance approach deliberately. Simply dividing total successes by total requests and applying an independent-request standard error would ignore the shared assignment structure.
The six-account fixture is not a launch sample recommendation. A large request log may still contain only a few independently assigned accounts; report both counts rather than advertising the event total as the experimental sample size. Unequal account sizes, baseline variation, within-account dependence and the effect worth detecting influence feasibility. Adding more requests from the same accounts does not automatically provide the information of adding independent accounts.
Anya has not calculated power or a required run length. She needs the eligible account inventory and appropriate historical information before recommending a feasible design. There is no promised detectable effect, significance level outcome or claim that matching account pairs will solve a shortage of independent units. If the available population cannot support the intended decision, the team must reconsider the question or study design rather than relabel observations as independent samples.
Read back the design decision before launch
Farah's completed fictional review says: “Account assignment is the proposed boundary for this shared queue, and equal-account weighting is the current target. Leon may prepare the identity checks, and Anya may assess feasibility. Launch is not approved: cross-account paths, missing-ID behavior, actual eligible account counts and the analysis plan remain open.”
This is a concrete design decision with accepted preparation work, not an experiment result. No feature reached customers, no exposure record was collected, and no treatment difference was estimated. A later launch requires its own approvals and implementation verification.
Further workplace guides are available from LandOffer.ai. Choose the smallest unit that respects the treatment's shared experience and a defensible interference boundary, document the target outcome and weighting, and retain the sample and identity limitations. Before launch, obtain the actual identity, interference and feasibility decisions; after implementation, check assignment and exposure evidence before interpreting outcomes.
Sources and Further Reading
- Microsoft Research: Why Tenant-Randomized A/B Test Is Challenging and Tenant-Pairing May Not Work — enterprise sharing, cluster variance and power considerations; November 4, 2020.
- Microsoft Research: Patterns of Trustworthy Experimentation, Pre-Experiment Stage — identity stability, network effects and enterprise constraints; July 31, 2020.
- Statsig Legacy Docs: Analysis When Assignment and Analysis Units Differ — named Warehouse Native example and analysis context; September 18, 2025; all sources checked October 28, 2026.


