Skip to main content
Experiment catalog and interference control: a registry, tagging rules and scheduling matrix for teams

Experiment catalog and interference control: a registry, tagging rules and scheduling matrix for teams

How to stop experiments from quietly corrupting each other's results

Two teams ship an experiment to the same checkout page in the same week. One is testing a new express-pay button. The other is testing urgency copy ("Only 3 left"). Neither knows the other exists. Both report a lift. Three weeks later a PM tries to ship both permanently and the combined effect is negative — the urgency copy pushed people toward the slower standard flow, and the express button cannibalized the exact segment the urgency copy was converting.

Nobody did anything wrong. There was just no shared view of what was running, where, and on whom. That's the gap an experiment catalog closes. Not a fancier stats engine — a registry of what's live, what's overlapping, and what's allowed to touch what. This post is narrowly about one thing: interference control. Not how to design a good test, not how to read significance — but how to keep concurrent experiments from contaminating each other's data and each other's users. If you've never had a result reverse itself in production, you probably have interference you haven't detected yet.

Why interference stays invisible until it hurts

The problem hides so well because each experiment, read in isolation, looks clean. Dashboards are per-experiment. Stand-ups are per-squad. The conflict only exists in the intersection of experiments, and nobody owns the intersection.

  1. Shared surface, separate squads. Pricing pages, onboarding flows, and checkout are magnets. Growth, Monetization, and Lifecycle all want them. Each team sees "their" test; none sees the stack.
  2. Overlapping audiences with different splits. Experiment A buckets 50/50 on all users. Experiment B targets "trial users in week 2." Those populations overlap, the randomization isn't coordinated, and B's control group is partly sitting in A's treatment.
  3. Downstream metric collisions. A notification-frequency test and a re-engagement email test both move 7-day retention. When retention wobbles, two teams claim credit or blame each other, and neither number is trustworthy.
  4. Infra-level side effects. A test that loads a third-party script slows the page by 200ms. Every other experiment on that page now runs on a slower baseline and reads slightly worse than it should.

This usually surfaces as "our experiment results don't replicate." Teams blame the stats. The stats are fine. The environment just wasn't held still.

The three things a registry actually has to track

A catalog that prevents interference needs more than a list of experiment names. It needs enough structure to detect collisions automatically. At minimum, every entry should carry:

FieldWhy it matters for interferenceExample
Surface / routePrimary collision axis/checkout, /onboarding/step-2
Audience definitionReveals population overlap"trial users, days 8–14, US"
Primary metricFlags metric collisionscheckout conversion
Guardrail metricsWhat it must not harmpage load, refund rate
Allocation %How much traffic it's consuming50%
Mutex tagsExplicit "can't run with" markerscheckout-layout, pricing
Owner + squadWho to call when it collidesLifecycle / Priya
Status + datesLive window for overlap checksrunning, 11/4–11/25

The two most underused fields are mutex tags and guardrail metrics. Mutex tags let you declare upfront that anything touching checkout-layout can't co-run with anything else touching checkout-layout. Guardrail metrics catch the second-order damage — the test that "won" on its primary metric while quietly dragging a shared one.

Worth flagging on tagging: tags rot faster than anything else in a registry. If every team invents its own vocabulary ("cart", "basket", "checkout", "co"), the mutex logic becomes useless. Pick a controlled vocabulary for surfaces and audiences, keep it short, and don't allow free-text tags on collision-critical fields.

Priority lanes: deciding who yields

Once you can see collisions, you need a rule for what happens when two important experiments want the same surface. The instinct is "first come, first served," which just rewards whoever filed their ticket fastest — not whoever has the higher-value test.

  1. Lane 1 — Protected. Revenue-critical or compliance-driven tests. These hold the surface. Everything else schedules around them.
  2. Lane 2 — Standard. Normal product experiments. They negotiate among themselves and yield to Lane 1.
  3. Lane 3 — Opportunistic. Small copy tweaks, low-risk UI changes. They run in the gaps, on leftover traffic, and pause the moment a higher lane needs the surface.

Assign lanes based on who can tolerate delay, not just perceived test importance.

The useful reframe: lanes aren't about importance in the abstract, they're about who absorbs the cost of waiting. A protected pricing test might block three Lane 3 tweaks for a couple of weeks, and that's the right trade — the tweaks lose little by waiting, the pricing read loses everything if it's contaminated. Tie lane assignment to outcome value and you're already partway to mapping experiments to real objectives instead of to whoever pushed hardest.

The scheduling matrix

This is where the catalog becomes operational. A scheduling matrix is a grid of surfaces (rows) against time windows (columns), with each cell showing what's occupying that surface and how much traffic it holds.

A stripped-down version:

SurfaceWeek 1Week 2Week 3
/checkoutExpress-pay (50%)Express-pay (50%)—
/pricing—Annual-toggle (Lane 1, 100%)Annual-toggle
/onboarding step 2Tooltip test (30%)Tooltip test (30%)Progress-bar (40%)
Email: re-engage—Win-back cadenceWin-back cadence

Reading this grid, the urgency-copy-vs-express-pay disaster from the opening becomes obvious before launch: both want /checkout in Week 1, and the matrix forces a decision instead of letting two teams back into the same lane blind.

The matrix also exposes traffic math, which teams routinely ignore. If three experiments each claim "50% of users" on overlapping populations, you don't have three clean A/B tests — you have a tangled factorial design nobody meant to run. The matrix makes you add up allocation per surface and notice when you've oversold your traffic.

Conflict-resolution workflow

Detecting a collision is step one. You need a repeatable path to resolve it, or every conflict turns into a Slack argument. A workflow that holds up in practice:

  1. Flag at registration. When a new experiment is submitted, the system checks surface + audience + mutex tags against everything live or scheduled in the same window. A hit raises a conflict before anything ships.
  2. Classify the conflict. Is it a hard conflict (same surface, overlapping users, mutex collision) or a soft one (different surfaces but shared downstream metric)?
  3. Apply lane rules. Higher lane holds. Same lane goes to negotiation between owners, with a default tiebreak of "larger expected effect yields later, not never."
  4. Choose a resolution. Options

    sequence them, split traffic into mutually exclusive buckets, reduce allocation, or hold one for the next window.

  5. Record the decision. Who yielded, why, and the new dates — written down, not just remembered.

That last step matters more than it looks. Undocumented yields breed resentment and repeat fights. A one-line decision record ("Win-back held to Week 3 so retention read stays clean for notification test") kills the re-litigation before it starts.

A simple visual helps teams follow the process and ensures the steps don't get skipped.

Process diagram

That flowchart should be visible from the registry UI so owners see the next steps when a conflict is flagged.

A checklist before any experiment goes live

Run this against every test, regardless of lane:

  1. [ ] Surface and route are declared with the controlled vocabulary, not free text
  2. [ ] Audience definition is specific enough to compute overlap with live tests
  3. [ ] Mutex tags added for any shared-surface or shared-metric risk
  4. [ ] Guardrail metrics defined — including shared downstream metrics owned by other teams
  5. [ ] Allocation checked against total traffic committed on that surface for the window
  6. [ ] Conflict check run and cleared (or resolution recorded)
  7. [ ] Lane assigned and justified
  8. [ ] Owner and live window logged

If you can't fill in the audience definition precisely enough to compute overlap, that's usually the actual problem — a test too vaguely scoped to trust. This connects directly to selecting the right lightweight tests in the first place; a well-scoped bet is also an easy-to-catalog one.

A real scenario

A mid-market B2B SaaS company, around 40 people across product and growth, was running somewhere between 15 and 20 experiments a month across three squads. Their stated problem was "experiment velocity is high but trust in results is low." Leadership had started ignoring experiment reports because too many had reversed in production.

When they laid everything onto a scheduling matrix for the first time, the picture wasn't pretty. On their onboarding flow alone, four experiments had overlapped in a single month, two of them targeting the same trial-user segment with uncoordinated splits. Their "activation lift" from one test was almost certainly borrowing effect from another.

The fix wasn't dramatic. They added surface + audience + mutex tags to a shared registry, built the matrix in a spreadsheet to start, and introduced the three-lane model. Within a couple of quarters, concurrent collisions on shared surfaces dropped to near zero. More importantly, the number of results that reversed after rollout fell from several a quarter to roughly one or two edge cases. Experiment volume didn't drop. They just stopped running tests that quietly ate each other.

No tooling overhaul. Mostly discipline plus one place to look.

Meta-analysis cadence: learning across experiments, not just within them

The registry earns its keep a second way: it turns a pile of individual tests into a body of knowledge. Most teams never do this. Each experiment closes, the owner writes a summary nobody reads again, and the organization re-learns the same lesson every six months.

  1. Which treatments keep winning on this surface, and which patterns keep failing?
  2. Where did guardrail metrics move even when the primary metric didn't?
  3. How often did results that looked positive fail to replicate — and did those tend to run during known collision windows?
  4. Are any surfaces so experiment-heavy that baseline behavior has drifted?

Teams that run meta-analysis start designing fewer, better tests because they stop re-testing settled questions. The catalog is what makes this possible — you can't review across experiments if you can't query across them.

When this is overkill

Not every team needs this. If you're running two or three experiments a month on clearly separate surfaces, a shared registry and a scheduling matrix are bureaucracy you'll resent. A single shared doc listing what's live is plenty.

  1. Multiple squads share high-traffic surfaces (checkout, pricing, onboarding, core feed)
  2. You run enough concurrent tests that overlap is likely, not hypothetical
  3. Results have started reversing in production, or leadership has stopped trusting reports
  4. Downstream metrics like retention or revenue are owned by one team but moved by many

Imposing the full lane-and-matrix system on a small team still fighting to get any experimentation culture going is genuinely a bad idea. Structure before volume just kills momentum. Build the habit of testing first; add interference control when collisions become real.

The takeaway

Interference is the failure mode nobody puts on a dashboard, because it lives in the gaps between dashboards. A real experiment catalog — with disciplined tagging, mutex rules, priority lanes, a scheduling matrix, and a resolution workflow — isn't about running more tests. It's about making sure the tests you run are reading what you think they're reading.

Start smaller than you expect. A single matrix of surfaces against weeks, filled in honestly, will surface more contamination in an afternoon than most teams find in a year of chasing "why won't this result replicate."

Start smaller than you expect. A single matrix of surfaces against weeks, filled in honestly, will surface more contamination in an afternoon than most teams find in a year of chasing "why won't this result replicate."

Built for Product Teams Designed specifically for product managers and agile workflows
Save Time Streamline roadmapping, prioritization & release planning
Increase Transparency Keep stakeholders informed with real-time updates
Drive Growth Focus on impact-driven features and customer value