Deterministic A/B Test Buckets

The problem with most A/B frameworks

Bucketing usually relies on a hash of userId mixed with an experiment ID. It's deterministic, but the salt is internal — if you ever change it, every historical bucket assignment becomes unverifiable. Worse, no one outside your team can confirm that a flagged user really was in the treatment group when they saw a bug.

The Provable.io approach

1. Each experiment gets a committed serverHash at launch.
2. Each user's bucket is derived from clientSeed = experiment_id + ":" + user_id.
3. The API returns the same number for the same seed forever — so the assignment is reproducible.

Code

async function bucketFor(experimentId, userId, variants) {
  // variants: [{ name: "control", weight: 50 }, { name: "treatment", weight: 50 }]
  const total = variants.reduce((s, v) => s + v.weight, 0);
  const clientSeed = `${experimentId}:${userId}`;

  const url = new URL("https://api.provable.io/api/ints");
  url.searchParams.set("clientSeed", clientSeed);
  url.searchParams.set("count", "1");
  url.searchParams.set("min", "1");
  url.searchParams.set("max", String(total));

  const res = await fetch(url, {
    headers: { "x-api-key": process.env.PROVABLE_KEY }
  });
  const { outcome, serverHash } = await res.json();

  let cumulative = 0;
  for (const v of variants) {
    cumulative += v.weight;
    if (outcome[0] <= cumulative) {
      return { variant: v.name, clientSeed, serverHash };
    }
  }
}

const { variant } = await bucketFor("exp_2026_checkout_v2", "user_8821", [
  { name: "control",   weight: 50 },
  { name: "treatment", weight: 50 },
]);

Cache it

The API call is deterministic, so cache the first lookup per user/experiment in your DB. After that, you never hit the API for that user again — but if anyone asks "why was user 8821 in treatment?" you can re-run the call and prove it.

Stickiness across rollouts

Want to expand from 10% → 50% treatment without re-shuffling existing users? Use a fixed weight space (e.g. max=100) and grow the treatment band from the same side. Anyone in the original 10% stays in treatment; new users fall into the expanded band.

// Phase 1: treatment = rolls 1..10
// Phase 2: treatment = rolls 1..50   (everyone in phase 1 still qualifies)

Auditing later

If a compliance review asks "did you really only show this UI to 5% of users?", you can:

• Replay the bucket call for any user ID,
• Show the committed serverHash,
• Point them at /verify for independent confirmation.

Next steps

• Use weighted random for prizes
• How provably fair works