Online A/B Testing
Target Marketing
by Ken Burke
May 1, 2005
A new world of possibilities for a reliable, old tool.
Sometimes old tools are the best tools. Recently, online merchants have been turning to A/B testing to refine their online messaging, increase the effectiveness of their offers, improve usability, and boost sales.
Long a staple in the direct marketing industry, A/B testing has become a powerful ally of the online marketer now that it can function in concert with e-commerce marketing and analytics applications. Its simple, logical approach to testing and refining even highly subjective variables is perfectly suited to the Web environment, though it predates the Web and even the computer by several decades.
What Is Online A/B Testing?
As in the print marketing world, online A/B testing enables you to find out which of two offers (A or B) is more effective, as measured by the actual sales it generates. You retain and continue refining whichever of the two performs better. A/B testing can be applied to an extremely broad range of variables online, from the design of an entire Web page, to the subtle wording of a promotion, to the structure of your discounts.
Let's assume you want to test a promotion for your homepage. Does a bold "Summer Blowout" approach generate more sales, or is a more sedate "Affordable Summer Specials" better? You create one homepage for each approach, and set your e-commerce engine to randomly direct 50 percent of your customers to one page and 50 percent to the other during a prescribed test period. You measure the sales each page generates, discard the poor performer, and use the better approach for the duration of the promotion. You then can adjust the winner, test it again, and continue refining it until you have the optimal combination of offer, messaging, page design and so forth.
Key elements in online A/B testing are a flexible e-commerce platform and a powerful analytics engine. Your e-commerce platform must allow you to change major aspects of your site quickly and efficiently on the fly. If your design, promotion and messaging changes take too long to put in place, the cost of each test increases, and its effectiveness decreases. Also, your analytics tool must track sales generated by individual page elements such as buttons or linked images. If you can assign source codes to these links and track their usage, you probably have a system that can be used for A/B testing.
The big advantage to online A/B testing is that it can be done in virtually real time. Whereas a cataloger might have to wait weeks before the results of an A/B test come in, an online merchant can accomplish the same thing in a few hours. You could create and deploy two different promotions before your morning coffee break, track link usage as sales roll in, and confidently launch the best promotion before you go to lunch. It also allows you to isolate and improve key factors that determine how and when customers buy.
A/B Testing in the Real Virtual World
A/B testing can be applied to any customer-facing element of your online marketing. Here is a list of commonly tested items, and some of the questions that adhere to each.
Offers and messaging: These are the most commonly tested items in both print and online. Different permutations of these appear on every page of a merchandising Web site. As you go through the rest of this list, remember that every item can incorporate the testing of offers or messaging.
Landing and homepages: What design layouts and top-level messaging stimulate the customer to actually enter the site?
Directory pages: What is the optimal number of product thumbnails you can display? Which layout encourages the largest number of shoppers to click through to product pages?
Product page layouts: What stimulates a customer to click "Add to Cart"?
Navigation scheme, button placement and button design: What encourages the customer to drill deeper into the site?
Pieces of functionality such as More Info, Tell a Friend, View Larger and Zoom.
Checkout pages: Do upsells or cross-sells early in the checkout process generate more sales, or do they distract buyers?
Product category names: What terms and shades of meaning do customers respond to best?
Category types: Do customers respond best to category types that present products of distinct types (pants vs. shirts, furniture vs. appliances), or do they prefer composite categories that group diverse but related items, e.g., ensembles, clearance, new items, brands or themed items?
Kickers: Which product kickers (small "ads" for other products, usually placed at the bottom of product pages) generate the most sales? How many should be used per page? How big should they be? Should they be used at all?
Shopping cart: Do kickers in the shopping cart enhance or detract from sales?
Colors and designs: What design elements best promote your brand and your products online?
Multivariable Variant Testing
More complex multivariable variant testing can string one test after another. Using the summer promotion example again, you could randomly direct 50 percent of the "Summer Blowout" clickers to a page that highlights poolside furniture, and 50 percent to a page for camping gear. The swimmers and the campers could then be split again, with half being offered a fixed percentage off everything, and the other half being offered a smaller discount plus free shipping. Each of these individual tests can be run simultaneously without muddling the results, since each A/B test can be measured and tracked independently.
A/B Testing and Customer Base Segmentation
Naturally, a change that elicits a positive result among one customer group could easily have a negative result among another. The solution is to A/B test within each of your customer groups. This requires an e-commerce platform that can identify which group each shopper belongs to and load customized pages for each group.
Start with your most profitable customer group, such as women between the ages of 35 and 50. Find a key element that has a big impact on how they buy, such as the structure of a promotion. Send half of these customers to pages that feature your baseline promotion, and half of them to pages with a test promotion heavily weighted for them. Male customers can be tested simultaneously with their own customized pages, or not at all. Either way, they won't see the female-oriented test promotion, and therefore will not be affected by it.
How Do I Gear Up for A/B Testing?
If you think A/B testing has something to offer, start by talking to your e-commerce platform developer. Some developers actually have integrated A/B testing capabilities into their technology, so you may be able to put an A/B testing program in place fairly quickly.
Third-party applications and services also are available for integration into your site. Leading providers include Offermatica, Optimost, and SiteSpect.
Let Customers Tell You What They Like
It's important that you only alter one factor at a time for any given test. If you change more than one thing on a page, you won't be able to tell what causes a change in customer behavior. Are they responding to the new colors, the new fonts, the fabulous new offer, or the new buttons? Give them a simple A or B choice, and their preferences will be made clear. If you must run simultaneous tests, make sure you develop a proper multivariable variant testing plan that will isolate the results of each test.
Test the most important things first. Typically these include promotions, messaging and usability issues. Pay attention to how your sales move in relation to these changes, and apply any lessons learned to other parts of your site.
The real advantage to A/B testing is that your customers tell you precisely what they like and what they don't like. They vote with their dollars, and their opinions manifest themselves shamelessly in your sales reports, unencumbered by your own assumptions or biases. For many online merchants, A/B testing is the ideal way to test multiple approaches to promotions, messaging, page designs, navigation paths and many other key elements. By testing and then acting on the test results, you can gain efficiencies that will translate to boosted sales, increased conversions, and reduced costs in production and deployment. Results may be incremental, but even a small increase in sales or efficiency can have a big impact on your bottom line.
Ken Burke is president and CEO of Multimedia Live, an e-commerce technology and development company based in Petaluma, Calif. He can be reached at (707) 773-3434 or ken@mmlive.com.
This article was originally published at http://www.targetonline.com/columns/283219125942313.bsp
