A/B Testing

A/B testing is a very old technique that originated long before the advent of the computer and certainly the Internet. A/B testing (also referred to as Split Run Testing) has been used by marketers for years in direct mail and other direct response marketing efforts.

An A/B test is simple to conduct. In such tests, the "A" option is the control, or current champion. The "B" option is the challenger being tested in an attempt to provide better results than "A." Visitors or respondents are randomly shown or offered the "A" or the "B" option. The difference between the two response rates is then evaluated for being statistically significant or not.

As it relates to website testing, generally there are two techniques for A/B testing: (a) change one page element at a time, or (b) change multiple page elements simultaneously.

In the case of (a), the challenger has one difference from the control (maybe the headline or the price listed has changed). Although this method can give the tester information on the cause of any difference in response rates between the pages, to test every possible factor that can affect conversion would take an extremely long time. Unlike this form of "serial A/B testing," multivariable testing provides much quicker and more reliable results.

In the case of technique (b), the challenger may have many different variables changed on the page. However, regardless of the result (which may show the challenger losing to the control), there is no information why one page performed better than another. This is not a "controlled" experiment. For example, even though "B" may have performed better than "A," perhaps the change in one variable was actually a detriment that reduced the otherwise strong improvement from a change in another variable.

Although there are limited occasions when A/B testing is appropriate (such as if the marketer wants to test two completely distinct concepts of a page or a funnel), generally there is no upside to choosing A/B testing over multivariable testing in terms of time, efficiency, ease of implementation, or likelihood of superior results.