A/B Testing

A/B testing measures the causal effect of product changes by randomly dividing users into experimental and control groups, and is the cornerstone of data-driven decision-making at internet companies.

A/B测试通常需要至少数百次访问才能得出统计上可信的结论

The standard practice for A/B testing is to split online traffic proportionally and evaluate the effectiveness of new models by comparing key metrics across different groups.

The industry typically employs three evaluation frameworks for AI model A/B testing: rule-based automated evaluation, the LLM-as-Judge method, and comparison against manually annotated golden datasets.

A/B testing itself does not mean a feature will eventually ship, and many experiments are rolled back after validation.

A/B tests are usually conducted during the staged rollout phase, with coverage ranging from one in a thousand to several tens of percent, so the same app may look different in different users' hands.

Using feature flags' percentage rollout directly to run experiments makes it easy to encounter problems such as non-random traffic splitting, imbalanced sample proportions, and the same user being assigned to different groups on different devices.

Feature flags and A/B testing share the "grouping + traffic splitting" capability in engineering implementation, but their goals differ: flags focus on release control, while experiments focus on causal inference.

The selection strategy should shift from "comparing unit prices" to "comparing total cost of ownership," and it is recommended to run small-scale A/B tests under real business loads, recording success rates, retry counts and average cost per output.

A/A testing is used to measure the noise level of the test framework itself, and is a prerequisite for A/B testing.

Incremental prompt iteration (changing only one variable per week) is consistent with the A/B testing approach and avoids the inability to attribute results caused by changing multiple variables at once.

The value of increasing sample size depends on the independence between samples, which is a basic principle in A/B testing and portfolio investing.

For teams using A/B testing or gradual rollout strategies, any switch at the feature level should be explicitly disclosed in the review submission notes, whereas parameters used only for UI fine-tuning or performance optimization are not.

In Agent scenarios, directly comparing the average success rates of two groups of runs is easily misleading, and a more robust approach is to use a paired experimental design.

A/B testing in model evaluation refers to running the same batch of prompts through a fast model and a reasoning model respectively, and comparing key metrics to decide the routing strategy.

When data accumulation is insufficient, one can first use a simple rule engine plus A/B testing to gradually iterate the reminder strategy, avoiding the maintenance cost and unexplainability brought by introducing a model too early.

A/B testing of Agents must fix the input set and tool versions; otherwise, traffic distribution or interface changes will contaminate the conclusions.

In a production line environment, there is a delay between error-correction signals and model updates, requiring engineering mechanisms such as shadow deployment, A/B testing, and hot model switching to ensure the safety and rollback capability of updates.

In A/B testing, accuracy differences of less than 2% are usually hard to distinguish, and a larger sample size is needed to reach reliable conclusions.

The author admits that the CTA copy was modified at the same time as the itch data was being checked, so it cannot be attributed to a single variable—this was not a clean A/B test.