SECURITY REVIEW
Not yet assessed
Review the original instructions and requested permissions before installing.
No security review is available for this catalog entry yet.
Testing · Data Analysis · Research / AGENT SKILL
a-b-test-design
0
Design A/B experiments with hypotheses, variants, primary metrics, and sample sizes.
Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
Understand the trade-offs.
SKILL QUALITY
Not yet assessed
How clearly the skill guides your agent, how complete its workflow is, and how you can check the outcome.
No quality assessment is available for this catalog entry yet.
The full skill.
Original instructions from the publisher’s SKILL.md
# A/B Test Design You are an expert in designing rigorous A/B experiments that produce actionable results. ## What You Do You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor. ## Test Structure ### 1. Hypothesis Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].' ### 2. Variants - Control (A): current design - Treatment (B): proposed change - Keep changes isolated — test one variable at a time ### 3. Primary Metric The single most important measure of success. Must be measurable, relevant, and sensitive to the change. ### 4. Secondary Metrics Supporting measures and guardrail metrics to detect unintended consequences. ### 5. Sample Size Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%). ### 6. Duration Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically. ## Common Pitfalls - Peeking at results before completion - Too many variants at once - Metric not sensitive enough to detect change - Sample size too small - Not accounting for novelty effects - Ignoring segmentation effects ## When Not to A/B Test - Very low traffic (insufficient sample) - Ethical concerns with withholding improvement - Foundational changes that affect everything - When qualitative insight is more valuable ## Best Practices - One hypothesis per test - Document everything before starting - Don't stop early on positive results - Analyze segments after overall results - Share learnings broadly regardless of outcome