Assign users on the server, render the right variant in the HTML, and decide the metrics and duration before you start.
Assignment: put users into buckets on the server or at the edge (a CDN server near the user), using a stable id. Render the chosen variant in the HTML. Assigning on the client after load causes problems:
- a flicker of the control version (the current version), plus layout shift
- blocking scripts that hide the page until the experiment loads, which hurts LCP (Largest Contentful Paint: when the biggest image or text block appears)
Code: variants sit behind the same flag system as feature flags. Large variants are code-split, so users only download their own variant.
Measurement:
- Pick one primary metric and some guardrails before starting. Guardrails are metrics that must not get worse, like errors, performance and revenue.
- Log exposure when the user actually sees the variant, not at assignment. Otherwise the effect gets diluted.
- Run for a pre-planned duration and sample size. Do not stop early because the result looks good. Checking early like this is called peeking.
Hygiene:
- Avoid overlapping experiments on the same surface, unless they are designed for it.
- Exclude bots and internal users.
- Clean up the losing variant afterwards.
- Assign variants on the server or at the edge to avoid flicker.
- Code-split the variants.
- Set the primary metric and guardrails up front.
- Log exposure when the user sees the variant, not at assignment.
- Run for a planned duration, and do not peek at results early.
Read more
- web.dev (https://web.dev/)web.dev