Here at Surface AI, our philosophy is
In practice, that means we evaluate a lot of the science behind web optimization, weigh the pros and cons of implementing it into our product, and ultimately, make decisions that usually take a whole army of analysts, machine learning engineers, and data science professionals to make.
One of those is using multivariate bandit testing instead of traditional (and much more well-known) A/B testing. Many of our customers, when we share our methods with them, weren’t aware there were other types of web testing other than A/B.
Multivariate bandit testing is simple to explain. Imagine you have three variations of a landing page, each with different headline copy: Versions A, B, and C. When you launch, you see that Version A is the best performer: it’s converting the most users. The multivariate algorithm will slowly increase the traffic being sent to Version A, and evaluate the conversion as it goes. Version C is performing poorly, and users are getting confused, so traffic is being diverted to the better performing version A. Version B is just ok – not great, but not terrible – so it maintains a steady amount of traffic. This can be done with a theoretically infinite number of variations, as long as your traffic is high enough to support the division and remain statistically significant.
The benefits of this type of testing are threefold.
- More users see the better-performing variant immediately.
In traditional A/B testing, the audience division is static, meaning that if one of your variants is performing terribly, you have to wait until the test is over to shuffle all your traffic to the winner. With multivariate bandit testing, you don’t lose this time – we send users to the better performing variant right away. What does this mean? You don’t lose valuable conversions during your testing time.
- Testing is continuous.
With traditional A/B testing, you generally launch a test, wait a few weeks, and evaluate the outcome. If more optimization is needed, you launch another separate test. With multivariate bandit testing, you can simply add or remove variants instead. With our tooling this is a simple task requiring zero technical resources, and it’s easy to add information about the best performing variants into your generation prompts for new optimization.
- You don’t actually have to decide between your variants.
The members of team Surface have spent far too many hours of their short valuable lives sitting in meetings where various team members were trying to claw apart A/B testing results (often with biases and ulterior motives), because only one could launch! With Surface, you can continue to run all your variants for as long as serves your audience. In addition, this often helps you more easily capture changes that might occur due to factors such as seasonality or new ingress points. It’s not uncommon for one variant to perform best for a period of time, and then for it to swap with another.
Ultimately, we will use the methods that work the best: we’re agnostic, and if we find a better methodology than multivariate bandit testing, we’ll implement it. But for now, we believe deeply that this process gives our customers the best experience, and we can’t wait for you to give it a try.
Book a Demo Today
