1. LandOffer
  2. Blog
  3. All articles
  4. A Product Survey Has Few Responses: Check Who Is Missing Before Drawing Conclusions

Data & Decisions

A Product Survey Has Few Responses: Check Who Is Missing Before Drawing Conclusions

Compare your intended users, invitation frame and survey respondents before reporting a product finding. Work through a complete distribution audit, explicit item denominators and an owned follow-up plan that preserves missing evidence and limits population claims.

10 min readLandOffer team

When a product survey has few responses, compare the people you intended to hear from, the people you invited and the people who answered before drawing conclusions. A low completion percentage raises questions; it does not measure the direction or size of bias. Preserve the answers, identify gaps relevant to the decision and narrow the claim while you investigate those gaps.

This LandOffer.ai guide uses an original fictional workplace survey to build two completed materials: a population-and-response audit and a decision record with a targeted follow-up plan. All users, answers, owners and follow-up outcomes are invented. The accompanying offline arithmetic checks verify the example's counts, not survey validity or real customer behavior. No survey was sent and no participants were contacted.

Three adults examine cream envelopes and wooden tokens at a bicycle workshop table while another adult stands in the open doorway.

Separate a small count from a biased picture

Small samples and nonresponse create different problems. A small count can make an estimate unstable. Nonresponse can distort it when participation relates to the experience you are trying to understand. Collecting more replies from the same easy-to-reach group may leave that distortion intact.

Pew Research Center's 2017 telephone-survey study found that response rates alone were unreliable indicators of bias, and that problems differed across topics. That finding concerns the surveys it studied. It does not establish that a voluntary in-product survey with a similar percentage is representative.

Ask whether the missing group could plausibly experience the product differently. People using a shared device may never see an email invitation. Users who stopped using the product may be absent from an active-user list. People with a difficult experience may be especially motivated to answer, or too frustrated to return. These are competing hypotheses until investigated. A roster comparison can reveal an observed gap; explaining that gap requires additional evidence.

Do not infer a motive from silence. An unanswered invitation cannot tell you whether someone was satisfied, unavailable, unable to access the form or unwilling to participate. Record what happened to the invitation separately from your explanation for it.

Reconstruct the denominator before calculating a rate

Write the intended population and time window first. Then describe the invitation frame: the actual list or mechanism through which people could be selected. Deduplicate at the unit relevant to the question. A person with three accounts should not quietly become three independent users.

AAPOR's Standard Definitions guidance distinguishes coverage, measurement and nonresponse effects, and emphasizes recording case dispositions. Its outcome-rate framework requires more care than dividing arbitrary dashboard totals. For a simple internal audit, give every numerator and denominator a plain-language name; do not label a homegrown completion fraction a standardized AAPOR response rate.

Retain eligible invitations that bounced in the invitation count. Show bounces separately rather than making the result look better by silently deleting them. Separate complete submissions, partial submissions and invitations with no recorded start. Define completion, too: here it means submitting the required questions, even if an optional item is skipped. A partial response can still contain an item answer; say whether the analysis includes it.

Open-link surveys need different treatment. If you do not know how many eligible people actually received the opportunity, you cannot produce a defensible eligible-invitation completion rate from page views. Report the available counts and that missing denominator.

Worked asset: a filled invitation and response audit

The fictional Juniper shift-handover app team wants to understand finding handover notes. Researcher Maren defines the target as 400 distinct eligible users active during the preceding 30 days. An authorized fictional roster assigns each user one primary work setting: desk, mobile or supervisor. These categories are mutually exclusive for this audit, not claims about every task a person performs.

Only 320 have an approved email contact route. All 320 are invited once during the same fictional November 2–8, 2026 window; the roster is frozen at the start. The remaining 80 are outside this invitation frame; they are not nonrespondents to an invitation they never received. There is no probability selection, and historical or inactive users are outside the stated target.

Primary work setting Target users Invited Complete Partial Bounced No recorded start
Desk 160 144 36 4 8 96
Mobile 160 112 7 5 8 92
Supervisor 80 64 16 3 4 41
Total 400 320 59 12 20 229

Each invitation has exactly one final disposition here. The 59 complete submissions divided by 320 invitations yield 18.4% completed per invitation. The frame covers 80% of the intended population. Showing both prevents an apparently improved completion fraction from hiding people who never had an invitation route.

Compare each setting with its own denominator. Desk users have 90% frame coverage and 25% completion per invitation. Mobile users have 70% coverage and 6.25% completion. Supervisors have 80% coverage and 25% completion. Mobile users are therefore missing disproportionately at both the invitation and completion stages.

Among complete respondents, desk users account for 61.0%, mobile users 11.9% and supervisors 27.1%, rounded to one decimal. In the target roster those shares are 40%, 40% and 20%. This demonstrates an imbalance on a known attribute, not a numerical estimate of nonresponse bias. It does not establish the size of bias on finding notes, or prove that work setting explains participation.

Maren preserves the invitation extract, disposition rules, question version and analysis cutoff in a restricted fictional research folder. The decision document uses only these aggregate counts. It does not publish names or individual nonresponse records to encourage managers to chase employees.

Keep the item denominator visible

The optional question reads: “During the last seven days, did you need another person's help to find a handover note?” Answers are Yes, No, and I did not look for a handover note. This asks about reported experience in a defined period, not objective search success or how a proposed feature would perform.

The fictional analysis includes complete submissions only, a rule stated before tabulation. Twelve partial submissions remain in the audit but contribute no item answers to this example. Of the 59 complete submissions, three skip this optional question: two desk users and one supervisor.

Setting Answered item Yes No Did not look Skipped item among complete
Desk 34 8 20 6 2
Mobile 7 6 1 0 0
Supervisor 15 3 8 4 1
Total 56 17 29 10 3

Seventeen of 56 item respondents, or 30.4%, report needing help. Excluding the ten who did not look gives 17 of 46, or 37.0%, among item respondents who looked. Neither percentage describes all 400 eligible users. Choose the denominator that answers the stated question and retain the other categories so a reader can reconstruct it.

Six of the seven mobile respondents report needing help. That is a reason to investigate their circumstances, not a stable mobile-population estimate. Avoid a precise subgroup ranking or a conventional probability-sampling margin of error for this voluntary sample. Also avoid counting “did not look” or skipped answers as successful independent searches.

Check the collection mechanism before increasing recruitment

Review invitation delivery, access to the form, device compatibility, language, length and question interpretation. These are checks to perform, not explanations already demonstrated by the counts. Compare known participation attributes that bear on the product decision; do not collect sensitive attributes merely to fill a comparison table.

Pew's question-writing guidance describes pretesting and explains how wording, answer options and question order can change responses. More participants cannot repair an ambiguous question. If you change the question or collection mode during follow-up, retain the new version and analyze its results separately before considering any combination.

Known role totals may suggest weighting, but matching those totals does not prove that respondents resemble nonrespondents within each role. A large weight on seven mobile respondents can amplify their particular experiences. In this example the team declines to produce a weighted population estimate. A defensible adjustment would require suitable benchmarks, assumptions, expertise and a clear account of what remains uncorrected.

Do not manufacture missing answers through AI, fill them with the observed average or equate product telemetry with an opinion. Approved telemetry might help investigate exposure to a control; it cannot tell you an absent user's survey response. Qualitative follow-up can explain a mechanism without estimating its prevalence.

An adult holds a folded street map and cream envelopes beside another adult holding a green notebook near an open courtyard door.

Worked asset: a conclusion and follow-up decision record

Product lead Dorian must decide whether to commit the next release to replacing note search. Maren gives him a scoped conclusion: “Among 56 complete-submission item respondents, 17 reported needing help in the past seven days. Mobile users are underrepresented in the invitation frame and completions. These data identify reported difficulties to investigate; they do not estimate their prevalence across eligible users or justify replacing search.”

The completed planning record assigns finite work rather than an instruction to keep recruiting until a preferred percentage appears:

Question or gap Named action and capacity Decision and evidence boundary
Did the form work on the relevant device? Engineer Ivo reserves 90 minutes on approved test devices to inspect invitation-to-form access Record actual outcomes before attributing low participation to access; no customer-device access assumed
Why are mobile users missing? Researcher Maren reserves four 30-minute voluntary conversations through a reviewed mobile-accessible route Seek varied circumstances, including prior nonrespondents; four conversations are exploratory, not a representative replacement sample
Does the question mean the same thing to users? Researcher Selma reserves two 30-minute pretests before any revised survey Record comprehension and version changes; do not silently pool revised answers
Can people outside the email frame participate safely? Operations owner Otello reviews the proposed route and privacy wording before recruitment Recruitment remains pending until approved; supervisors do not select only their most engaged staff
Should the team replace search now? Dorian retains the existing search and reserves a two-hour problem-review session No implementation commitment; review access checks, follow-up evidence and remaining gaps first

The fictional follow-up record is deliberately mixed. Ivo completes the approved-device check and reproduces a form sign-in loop on one shared test-device configuration. That observation establishes an access issue in that configuration, not its frequency among the missing 80 or 229 people with no recorded start. The original question remains unchanged during this check.

Otello approves a voluntary invitation route, but four planned conversations produce only two completed sessions. In this invented record, both people describe shared-device constraints; one needs help finding notes and the other does not. Two sessions remain unfilled at review. Selma's two pretests are still pending. These are authored scenario outcomes, not research performed for this article.

Dorian's completed decision is hold the search-replacement commitment. Ivo owns investigating the sign-in loop, with a fix and verification still pending. Maren retains the original survey and the two conversations as separate evidence sets. She neither adds those two people to the survey numerator nor treats their accounts as the views of all missing users. Dorian schedules the next decision review for November 16 in this fictional plan. If pretests or the access investigation remain unfinished, the record must say so rather than imply the review date confers approval. No claim of improved participation or search performance is recorded.

Give the next reader an honest stopping point

Before presenting the survey, attach the target definition, frame, invitation window, dispositions, exact wording, item denominators, exclusions and unresolved gaps. Label observations separately from suspected mechanisms. A reader should be able to see which decision the evidence supports and which remains on hold.

Save a bounded next action with an owner and review point. Sometimes that action is repairing access, clarifying a question or acknowledging that the current survey cannot answer the decision. Continue with LandOffer.ai's workplace guides when you need another practical work record; preserve this survey's limits wherever its findings travel.

Sources and Further Reading