March 14, 2017

Why you should move beyond A/B testing for email campaign optimization

Email AB Testing

How do you know that your email marketing campaigns are performing as well as they should be? You’ve probably heard of A/B testing — one of the foundational metrics of marketing — and wondered if it can improve your campaigns. As trained scientists and statisticians, we understand the importance of controlled experiments and valid statistical techniques, and we know that there’s a better way.

If you’ve ever run some A/B tests in the past, you know the drill: select the email you want to test, make some slight variations, set up the audience split (50-50, 60-40, 1-99…?) and let it run for a short period of time. At the end of the experiment, one version has a higher click or read rate, so you select it. If you’re using A/B testing like this, you may be doing yourself a disservice.

Here are 5 reasons why you should move beyond A/B testing:

1. Context matters

The problem with standard A/B test is that they only allow you to compare a single message — completely ignoring the entire context of the customer experience. Intuitively, we know that context matters. A customer is much more likely to use a coupon when they’re shopping for a product compared to a random coupon for an unwanted item. Likewise, a customer might respond better to a product image when it’s sent first, but later respond to more personal messaging.

Comparing two or more versions of a single email might tell you which one is most effective in the context of the current campaign configuration, but it won’t tell you which one is more effective in general. That is, you might learn that a particular message performs best in an A/B test but, when you later change some other aspect of the campaign, you invalidate the previous results. Perhaps there are no differences now, or the email you didn’t pick is actually better!

2. Both your customers and business are dynamic

Neither your business nor your customers are static. Your email strategy shouldn’t be either. One-off A/B tests of emails ignore the fact that your customer base and your product change over time, meaning that the most effective email marketing strategy might change as well. How do you know that the ‘winner’ will still be working well down the line?

3. Managing complexity is time consuming and expensive

Let’s imagine that you’re super eager to know about your customer preferences using A/B testing and you begin segmenting your customer base to run experiments! Awesome, now you can learn about what works for particular groups of customers. The problem is that, with each new segment, you’re decreasing the statistical power of the test and increasing your workload with the complexity of running multiple experiments.

Even for the super-productive, there are limits to the number of experiments you can run and interpret. There are infinite ways to segment, based on activity, demographics, interests, purchasing history, geography, and on and on. Even if you hire a new person to manage all of your email optimization, they too will soon be overburdened and overwhelmed with the sheer amount of data and possibilities.

Even more difficult is understanding what to do with the results. Humans’ inferential abilities and decision-making skills are nothing to scoff at, yet selecting the optimal balance between exploration of new strategies with exploitation of current-best strategies is simply not a good fit for the way we think. To be frank, even with AI, it’s a tough problem to solve in theory (though the ability to perform computations millions of times per second helps with the number-crunching...).

4. You care about more than open and click rates

The standard comparison between A/B tests is often open rates and click rates. They are easy to calculate and nearly-instant with the speed at which most people check their inboxes.

But when you measure success internally, you’re not just looking at open and click rates (unless you sell clicks and opens). You care about the behaviors your customers take and how they interact with your product — whether a customer shares a product with their friend, makes a purchase, or engages with your application. Reading your emails is great, but it’s more important that the messaging you’re using drives people towards the KPIs that matter: your business goals.

5. Customer preferences differ

Each of your customers is an individual with different experiences, behaviors, and preferences. Running an A/B test will tell you which email performs better, but understanding individual preferences is nearly impossible. Perhaps some customers respond better to one message, or one sequence of messages, while others tend to prefer others. A static A/B comparison can never allow you to understand the nuances of your customer preferences at this level of detail.

Move beyond A/B testing with Optimail

Optimail uses adaptive personalization to continually adjust your email campaigns in real-time based on your customers’ behaviors and responses. It learns to drive your customers towards your business goals, whether that’s purchasing, sharing, engagement, or any other metrics that are important to you. Optimail manages the complexity of continuous optimization automatically, freeing you up to focus on other pressing business business tasks. It’s like hiring a team of superhuman data scientists and experienced marketers to work around the clock, at a tiny fraction of the price.

Enter your email to receive updates from the Optimail team, thoughts on AI and machine learning, and email marketing optimization tips.
Thank you for subscribing!
Please double-check your email address.


Ready to get started? Optimail is free while you build and integrate.