Library · Traffic · How to run this
Ship or kill with a written number
A landing test without a written decision log is decoration. Write the hypothesis. Lock the metric (settled CAC or convert). Pre-commit n per arm. Set a stop date. On that date, ship or kill and record the number. “Keep watching” is not a decision.
Pair: Landing test decision log on Traffic. Sample-size n: test sample size. Kill/keep score for live campaigns is a different job: campaign kill score.
How to use it
Tick each line when it is actually logged. Empty boxes mean you are decorating. Use the sample-size tool for n. Do not change the metric mid-test.
What not to do
Do not ship on vibes. Do not extend n after peeking. Do not leave “keep watching” as the outcome.
Related: Traffic, test sample size, campaign kill score.
Procedure
- Write the hypothesis.
- Lock the metric and n.
- Set a stop date.
- Ship or kill. Record it.
Failure modes
- No hypothesis.
- Metric is ‘feels better.’
- No stop date.
Worked example
None on this page yet. When one lands it will be a real case: what broke, the number, and what changed. Not a composite.
Glossary
- Decision
- Ship or kill, with the number. ‘Keep watching’ is not a decision.
Work with us — We run this job on your accounts. Services
FAQ
- Can the metric be “feels better”?
- No. Lock settled CAC or convert before you start.
- Can I peek and extend n when it looks close?
- No. Pre-commit n per arm. Peeking is how you ship noise.
- Is “keep watching” a decision?
- No. Ship or kill with the number on the stop date.