98e74b79d7
Phase 2 refactoring of skills >500 lines and medium-sized skills: - paid-ads: 553 → 297 lines - Extract ad-copy-templates.md, audience-targeting.md, platform-setup-checklists.md - analytics-tracking: 541 → 292 lines - Extract ga4-implementation.md, gtm-implementation.md, event-library.md - ab-test-setup: 510 → 264 lines - Extract test-templates.md, sample-size-guide.md - copywriting: 458 → 248 lines - Extract copy-frameworks.md (headline formulas, section types) - page-cro: 336 → 180 lines - Extract experiments.md (experiment ideas by page type) - onboarding-cro: 435 → 218 lines - Extract experiments.md (onboarding experiment ideas) All skills now use progressive disclosure with references/ folders, keeping SKILL.md files focused on core workflow while detailed content is available when needed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
269 lines
6.1 KiB
Markdown
269 lines
6.1 KiB
Markdown
# A/B Test Templates Reference
|
|
|
|
Templates for planning, documenting, and analyzing experiments.
|
|
|
|
## Test Plan Template
|
|
|
|
```markdown
|
|
# A/B Test: [Name]
|
|
|
|
## Overview
|
|
- **Owner**: [Name]
|
|
- **Test ID**: [ID in testing tool]
|
|
- **Page/Feature**: [What's being tested]
|
|
- **Planned dates**: [Start] - [End]
|
|
|
|
## Hypothesis
|
|
|
|
Because [observation/data],
|
|
we believe [change]
|
|
will cause [expected outcome]
|
|
for [audience].
|
|
We'll know this is true when [metrics].
|
|
|
|
## Test Design
|
|
|
|
| Element | Details |
|
|
|---------|---------|
|
|
| Test type | A/B / A/B/n / MVT |
|
|
| Duration | X weeks |
|
|
| Sample size | X per variant |
|
|
| Traffic allocation | 50/50 |
|
|
| Tool | [Tool name] |
|
|
| Implementation | Client-side / Server-side |
|
|
|
|
## Variants
|
|
|
|
### Control (A)
|
|
[Screenshot]
|
|
- Current experience
|
|
- [Key details about current state]
|
|
|
|
### Variant (B)
|
|
[Screenshot or mockup]
|
|
- [Specific change #1]
|
|
- [Specific change #2]
|
|
- Rationale: [Why we think this will win]
|
|
|
|
## Metrics
|
|
|
|
### Primary
|
|
- **Metric**: [metric name]
|
|
- **Definition**: [how it's calculated]
|
|
- **Current baseline**: [X%]
|
|
- **Minimum detectable effect**: [X%]
|
|
|
|
### Secondary
|
|
- [Metric 1]: [what it tells us]
|
|
- [Metric 2]: [what it tells us]
|
|
- [Metric 3]: [what it tells us]
|
|
|
|
### Guardrails
|
|
- [Metric that shouldn't get worse]
|
|
- [Another safety metric]
|
|
|
|
## Segment Analysis Plan
|
|
- Mobile vs. desktop
|
|
- New vs. returning visitors
|
|
- Traffic source
|
|
- [Other relevant segments]
|
|
|
|
## Success Criteria
|
|
- Winner: [Primary metric improves by X% with 95% confidence]
|
|
- Loser: [Primary metric decreases significantly]
|
|
- Inconclusive: [What we'll do if no significant result]
|
|
|
|
## Pre-Launch Checklist
|
|
- [ ] Hypothesis documented and reviewed
|
|
- [ ] Primary metric defined and trackable
|
|
- [ ] Sample size calculated
|
|
- [ ] Test duration estimated
|
|
- [ ] Variants implemented correctly
|
|
- [ ] Tracking verified in all variants
|
|
- [ ] QA completed on all variants
|
|
- [ ] Stakeholders informed
|
|
- [ ] Calendar hold for analysis date
|
|
```
|
|
|
|
---
|
|
|
|
## Results Documentation Template
|
|
|
|
```markdown
|
|
# A/B Test Results: [Name]
|
|
|
|
## Summary
|
|
| Element | Value |
|
|
|---------|-------|
|
|
| Test ID | [ID] |
|
|
| Dates | [Start] - [End] |
|
|
| Duration | X days |
|
|
| Result | Winner / Loser / Inconclusive |
|
|
| Decision | [What we're doing] |
|
|
|
|
## Hypothesis (Reminder)
|
|
[Copy from test plan]
|
|
|
|
## Results
|
|
|
|
### Sample Size
|
|
| Variant | Target | Actual | % of target |
|
|
|---------|--------|--------|-------------|
|
|
| Control | X | Y | Z% |
|
|
| Variant | X | Y | Z% |
|
|
|
|
### Primary Metric: [Metric Name]
|
|
| Variant | Value | 95% CI | vs. Control |
|
|
|---------|-------|--------|-------------|
|
|
| Control | X% | [X%, Y%] | — |
|
|
| Variant | X% | [X%, Y%] | +X% |
|
|
|
|
**Statistical significance**: p = X.XX (95% = sig / not sig)
|
|
**Practical significance**: [Is this lift meaningful for the business?]
|
|
|
|
### Secondary Metrics
|
|
|
|
| Metric | Control | Variant | Change | Significant? |
|
|
|--------|---------|---------|--------|--------------|
|
|
| [Metric 1] | X | Y | +Z% | Yes/No |
|
|
| [Metric 2] | X | Y | +Z% | Yes/No |
|
|
|
|
### Guardrail Metrics
|
|
|
|
| Metric | Control | Variant | Change | Concern? |
|
|
|--------|---------|---------|--------|----------|
|
|
| [Metric 1] | X | Y | +Z% | Yes/No |
|
|
|
|
### Segment Analysis
|
|
|
|
**Mobile vs. Desktop**
|
|
| Segment | Control | Variant | Lift |
|
|
|---------|---------|---------|------|
|
|
| Mobile | X% | Y% | +Z% |
|
|
| Desktop | X% | Y% | +Z% |
|
|
|
|
**New vs. Returning**
|
|
| Segment | Control | Variant | Lift |
|
|
|---------|---------|---------|------|
|
|
| New | X% | Y% | +Z% |
|
|
| Returning | X% | Y% | +Z% |
|
|
|
|
## Interpretation
|
|
|
|
### What happened?
|
|
[Explanation of results in plain language]
|
|
|
|
### Why do we think this happened?
|
|
[Analysis and reasoning]
|
|
|
|
### Caveats
|
|
[Any limitations, external factors, or concerns]
|
|
|
|
## Decision
|
|
|
|
**Winner**: [Control / Variant]
|
|
|
|
**Action**: [Implement variant / Keep control / Re-test]
|
|
|
|
**Timeline**: [When changes will be implemented]
|
|
|
|
## Learnings
|
|
|
|
### What we learned
|
|
- [Key insight 1]
|
|
- [Key insight 2]
|
|
|
|
### What to test next
|
|
- [Follow-up test idea 1]
|
|
- [Follow-up test idea 2]
|
|
|
|
### Impact
|
|
- **Projected lift**: [X% improvement in Y metric]
|
|
- **Business impact**: [Revenue, conversions, etc.]
|
|
```
|
|
|
|
---
|
|
|
|
## Test Repository Entry Template
|
|
|
|
For tracking all tests in a central location:
|
|
|
|
```markdown
|
|
| Test ID | Name | Page | Dates | Primary Metric | Result | Lift | Link |
|
|
|---------|------|------|-------|----------------|--------|------|------|
|
|
| 001 | Hero headline test | Homepage | 1/1-1/15 | CTR | Winner | +12% | [Link] |
|
|
| 002 | Pricing table layout | Pricing | 1/10-1/31 | Plan selection | Loser | -5% | [Link] |
|
|
| 003 | Signup form fields | Signup | 2/1-2/14 | Completion | Inconclusive | +2% | [Link] |
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Test Brief Template
|
|
|
|
For simple tests that don't need full documentation:
|
|
|
|
```markdown
|
|
## [Test Name]
|
|
|
|
**What**: [One sentence description]
|
|
**Why**: [One sentence hypothesis]
|
|
**Metric**: [Primary metric]
|
|
**Duration**: [X weeks]
|
|
**Result**: [TBD / Winner / Loser / Inconclusive]
|
|
**Learnings**: [Key takeaway]
|
|
```
|
|
|
|
---
|
|
|
|
## Stakeholder Update Template
|
|
|
|
```markdown
|
|
## A/B Test Update: [Name]
|
|
|
|
**Status**: Running / Complete
|
|
**Days remaining**: X (or complete)
|
|
**Current sample**: X% of target
|
|
|
|
### Preliminary observations
|
|
[What we're seeing - without making decisions yet]
|
|
|
|
### Next steps
|
|
[What happens next]
|
|
|
|
### Timeline
|
|
- [Date]: Analysis complete
|
|
- [Date]: Decision and recommendation
|
|
- [Date]: Implementation (if winner)
|
|
```
|
|
|
|
---
|
|
|
|
## Experiment Prioritization Scorecard
|
|
|
|
For deciding which tests to run:
|
|
|
|
| Factor | Weight | Test A | Test B | Test C |
|
|
|--------|--------|--------|--------|--------|
|
|
| Potential impact | 30% | | | |
|
|
| Confidence in hypothesis | 25% | | | |
|
|
| Ease of implementation | 20% | | | |
|
|
| Risk if wrong | 15% | | | |
|
|
| Strategic alignment | 10% | | | |
|
|
| **Total** | | | | |
|
|
|
|
Scoring: 1-5 (5 = best)
|
|
|
|
---
|
|
|
|
## Hypothesis Bank Template
|
|
|
|
For collecting test ideas:
|
|
|
|
```markdown
|
|
| ID | Page/Area | Observation | Hypothesis | Potential Impact | Status |
|
|
|----|-----------|-------------|------------|------------------|--------|
|
|
| H1 | Homepage | Low scroll depth | Shorter hero will increase scroll | High | Testing |
|
|
| H2 | Pricing | Users compare plans | Comparison table will help | Medium | Backlog |
|
|
| H3 | Signup | Drop-off at email | Social login will increase completion | Medium | Backlog |
|
|
```
|