6.13 A/B Testing
The purpose of A/B testing is to gather data to drive decision making, specifically in marketing. When marketers need to determine how something should be presented to consumers, such as the layout of the navigation options on a website, the tone of an email greeting, or the background in a mail advertisement, they might start with a hunch. In the absence of any data at the outset, there is nothing wrong with an intuition-driven approach, as it might be the best available option. As more and more data points are collected, though, marketers can use analytical tools to inform their decisions. A/B testing, which is used many thousands of times per day by marketers across the world, is one of the most common such tools. Considered one of the most basic forms of randomized controlled experiments, A/B test essentially involves comparing two versions of something and seeing which one performs better. Its purpose is to identify a cause-and-effect relationship.
So how does A/B testing work? Companies such as Netflix experiment on many things from user interface layout, personalized homepage, the images associated with video titles1. Before such a test is conducted, the experimenter must determine the variable to be altered in different product versions, as well as the outcome variable to be used for measuring results. Then the audience needs to be randomly assigned to see Version A and Version B because the inferential statistics that will be applied to the A/B test assume the observations are independent.
At this point you might wonder what sampling techniques are used to separate groups. It depends. At Netflix, the careful process of ensuring both audience groups are as homogenous as possible is done using stratified sampling.2 On the other hand, an email marketer utilizing marketing automation software could rely on the software to randomly split 10,00 customers into two groups of 5,000. To the first group, the marketer could use a serious-sounding subject line, such as “The current Lobster Land ticket promotion includes a discount that is 10 percentage points lower than last year’s admission rate.” To the second group, the marketer could try something more succinct and catchy, such as “Looking for a Quick Family Getaway that Won’t Break the Bank? Come on up to Lobster Land!” Then, the marketer could measure and track the results of some particular outcome variable, such as whether the recipient clicked on a link in the message.
Companies are not required to notify the subjects of an A/B test that such a test is occurring. If they were, then all A/B test results would potentially be compromised by the aforementioned Hawthorne Effect. Since an A/B test is designed to track and measure a user’s preferences, it is better that the user is not aware that the test is being conducted. Awareness of the test could impact consumers’ behavior, which could bring the validity of the results into question.
But wait! The A/B testing does not have to stop there. Every other aspect of the e-mail, from the introductory sentence, to the introductory paragraph, to the entire body of the message, all the way to the closing signature line (“Is ‘cheers’ or ‘best’ more effective as a sign-off?”) can be separately tested.3
With enough experimentation, and enough data, marketers could theoretically come up with the optimal email for a given situation. In fact, in 2012, President Barack Obama’s re-election staffers did exactly this — they extensively A/B tested variants of their website and e-mail messages in order to best steer viewers into making campaign donations.
Other common subjects for A/B testing include:
- Video games (“Will a motivational message at the end of Level 3 cause more users to attempt Level 4?”)
- Website design (“Will a blue ‘Buy it Now’ button attract more clicks than a red ‘Buy it Now’ button?”)
Marketing message content (“Was this message in the corner of our brochure too sale-sy? Or was it not sale-sy enough?”)
1 https://netflixtechblog.com/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15
2 https://netflixtechblog.com/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15