Why Spotify Is Not Using Bayesian A/B Testing

In recent years, amid growing interest in Bayesian interference for A/B testing, many commercial experimentation platforms have added a Bayesian mode, including GrowthBook, LaunchDarkly, PostHog, Amplitude Experiment, Optimizely, VWO, Statsig, and Eppo.

The main argument is that Bayesian A/B testing is modern, flexible, and easier to interpret than frequentist statistics. In addition, it’s often claimed that many of the frequentist complexities, like multiple testing and sequential testing, fall out naturally in the Bayesian way of thinking.

This blog post aims to interrogate these claims and to posit a counterargument: in Bayesian A/B testing, there is a theme of oversimplification that frequently leads to bad inference practice that is no better than peeking in frequentist statistics.

This post is derived from a paper we recently wrote that found that Bayesian and frequentist frameworks are much closer than the debate suggests.

We’ll break down the paper’s findings in five steps:

• First, we examine some generic claims about Bayesian A/B testing and show exactly when and how they apply.
• Second, we organize Bayesian configurations into tiers that provide different guarantees for the experiment program.
• Third, we show how Bayesian decision theoretic formulations can be translated into equivalent frequentist formulations.
• Fourth, we make the broader point that Bayesian and frequentist experimentation overlap.

We see many companies debating whether they need both Bayesian and frequentist modes while not yet nailing the basics of either. Many are drawn to tools with flexibility without accounting for their complexity. Our goal in this post is to articulate this complexity, to explain why Spotify currently sees no need to add Bayesian inference alongside existing frequentist tooling, and to help companies seeking to run experiments understand how to choose the best mode for their needs.

Clarifying what Bayesian inference for A/B testing is

The question of whether to use Bayesian A/B testing arises when a company wants to launch or enhance an experiment program, a series of experiments over time. Many companies have been led by the online discourse to believe that Bayesian inference is by default the best approach.

The problem is that Bayesian inference for A/B testing is not one thing. It is a family of configurations, each defined by a stopping rule, a prior, and a likelihood. Rather than starting with the tool (Bayesian A/B testing), a company should start by asking: what goals are important for my experiment program? For example:

• The program should limit the number of shipped features with no effect
• Impact should be estimated with a certain precision
• A certain cost function should be minimized over time

The statistical configuration that is best for the company then depends on the goals and constraints of the experimentation program.

Online discussion about Bayesian A/B testing is confusing because it implicitly mixes goals and configurations. Non-stats-savvy readers are left thinking that Bayes is an almost mystical form of inference that avoids the problems associated with frequentism. A claim that sounds like a fact about Bayesian A/B testing is often a fact about one goal paired with one configuration.

These are not variations on a theme. They are different configurations with different guarantees to meet different goals. In fact, the Bayesian inference framework covers everything from state-of-the-art sequential methods to procedures that are numerically identical to bad frequentist practice. We think understanding the Bayesian experimentation discourse lies in understanding these intricacies.

Further, we think it is important to realize that frequentistic error rates are not only for frequentists: Any experimentation program produces false positive and false negative results over time, regardless of whether its inference is based on Bayesian or frequentist philosophy, it is simply a classification of errors. The relevant question is whether the goal of the program is to control those rates, and whether the chosen configuration has parameters that control them.

Dissecting common claims about Bayesian A/B testing

In each of the four examples below, we take a claim, dissect it, and show under what conditions it holds.

1. Peeking

Common claim: “Bayesian A/B testing does not require peeking correction.”

This can mean two very different things. You simply may not care about controlling the false positive rate under peeking. Or you may be using a Bayesian configuration, such as Bayes factor stopping, that controls it automatically.

Those are not the same argument. The first implies the goal. The second implies the configuration. Treating both as a generic property of “Bayes” is where the confusion starts.

2. Multiple metrics

Common claim: “Bayesian testing handles multiple metrics automatically.”

This claim mainly hides how specific and demanding the relevant Bayesian configuration is. It’s like saying “peeking doesn’t inflate the false positive rate” for frequentist experiments. It’s true for some configurations of frequentist inference, and false for others: it’s not a statement about the framework itself.

There are two aspects of this statement that makes it muddle the frequentist vs Bayes discourse: First, the error rate that can be bounded without an explicit multiple-testing correction is the false discovery rate, not the false positive rate that many readers will assume. Second, the configuration that delivers false discovery rate control is very specific and far more demanding than the claim suggests.

3. The winner’s curse

Common claim: “Bayes fixes the winner’s curse.”

This is the least problematic claim. Any informative prior shrinks the effect estimate toward the prior mean, and that shrinkage works against the winner’s curse. With a reasonable informative prior, shipped effect estimates will be less inflated than estimates from a flat prior or a frequentist point estimate.

The problem is that most teams using Bayesian A/B testing run the platform default, which usually means a flat prior. A flat prior provides no shrinkage. Its posterior mean equals the maximum likelihood estimate, so the winner’s curse is just as severe as in the frequentist case. The claim “Bayes fixes the winner’s curse” is only true for people using an informative prior, and the degree of correction depends entirely on the quality of that prior.

4. Decision theory often connects to error-rate control

Common claim: “We need Bayes because we want to take a Decision theory-theoretic approach.”

Decision theory lets you specify costs instead of error rates. It’s strictly speaking not a Bayesian branch of math, but in A/B testing, the optimal policies turn out to be Bayes rules. For a simple and natural cost function that sums the costs of false positives, false negatives, and sampling, the optimal or near-optimal policy uses Bayes-factor thresholds. Because those thresholds also provide error-rate guarantees, the decision-theoretic and error-rate formulations become two parametrizations of similar configurations.

Bayes and frequentism are closer than you think

For the configurations most commonly deployed in practice, Bayesian and frequentist A/B testing are far more similar than the debate suggests.

Under a flat prior and the two-group normal model, Bayesian and frequentist procedures produce numerically identical outputs:

• The posterior mean equals the maximum likelihood estimate
• The one-sided posterior probability that B beats A equals one minus the p-value
• A 95% posterior-probability threshold is algebraically equivalent to the one-sided rejection region

An experimenter using flat-prior Bayesian inference with posterior-probability thresholds is running frequentist-equivalent inference under a different vocabulary.

Isn’t the interpretation nicer under Bayes?

Yes. Saying “there is a 95% probability that the effect lies between 0.2% and 1.1%” is more natural than explaining what “95% confidence” means. That said, we think the value of the probabilistic interpretation is oversold.

Under a flat prior, every probabilistic statement maps one-to-one to a frequentist quantity. The point estimates and interval bounds are the same. The posterior probability that B beats A is one minus the p-value. The probabilistic interpretation only becomes meaningfully different when informed by a good prior, and a good prior is not free: specifying, justifying, maintaining, and explaining it costs complexity. We gain interpretation but we pay in other kinds of complexity.

Why Spotify has not added Bayesian modes

Historically, all of Spotify’s production experiments are analyzed using frequentist statistics. We have long debated whether Spotify should allow experimenters to choose the mode of inference, or even only support Bayesian statistics in the platform.

To help us decide, we revisited the goals of the Spotify experimentation program:

• We want to minimize the number of experiments that lead to poor business decisions:
• Shipping changes that harm our product
• Shipping and maintaining changes that don’t improve the user experience
• Evidence from experiments should be trusted
• Experiment results should be hard to interpret incorrectly, especially across teams and departments
• Experiments should be easy to plan and configure, but hard to misconfigure

Because people at Spotify constantly collaborate in experimentation, both in setting up experiments and consuming results across the company, having two modes has a high cost. Supporting both modes of inference would require different planning, different monitoring, different interpretation, and possibly different-looking outputs. So far, the benefit of a second framework doesn’t outweigh those costs.

The advice we want to give is: Start with the experimentation program you want. Decide which guarantees you need and can maintain consistently. Then ask whether a specific Bayesian configuration meaningfully improves on what you already have. For Spotify today, it does not.