Quality · 8 min
Sampling without lying to yourself
Replay tools love WebViews. Native gestures are harder to serialise. If you sample both at the same percentage, your qualitative research quietly becomes an HTML study.
Session replay vendors often instrument the WebView more completely than the native shell. Event sampling in analytics SDKs has the opposite bias: native SDKs may drop high-volume screen views while HTML pixels fire everything. Put those two biases in the same organisation and the product narrative tilts toward whatever the HTML checkout did last Tuesday.
Write the sample rate into the contract
We ask teams to record three numbers: native event sample rate, WebView pixel sample rate, and replay capture rate by runtime. If they are not equal, every qualitative quote and every “users struggle at step four” claim must name the bias. That sentence belongs in the board appendix, not in a footnote nobody reads.
Ethics, briefly
Replay of a payment sheet is not the same as replay of a settings screen. Masking fields is necessary and still insufficient if the WebView loads third-party frames you cannot mask. If you cannot guarantee masking, do not capture that surface. This is not a legal essay; it is a measurement one. Unusable replay is still a sample bias if PMs watch it anyway.
A smaller, truer dashboard
After sampling is named, many teams reduce what they show. That is success. A hybrid measurement strategy that reports fewer metrics, with known coverage, beats a wall of charts that secretly describe two different populations.