Methodology FAQs
How do you define the success of an experiment?
I hear a lot of people talk about Taguchi – what is this?
What is Optimal Design and how does it differ from other testing methods?
How does Optimost determine the sample size for a test?
What is the purpose of a testing "wave"?
Answers
How do you define the success metrics for an experiment?
Optimization succeeds when a web page achieves its objectives, and thus success is different for every page. The goal of one web page may be simply to get visitors to delve deeper into your site. The goal of another may be to get visitors to register for your service or help them make their way through the shopping cart. You may measure success on a simple count of visitors or you may want to record shopping cart closing values and maximize average order value. Whether it's clickthrus, registrations, add-to-carts, average sale value or cost-per-thousand impressions, Optimost is capable of natively tracking any type of success metric required to facilitate the optimization process. Optimost is able to group these success events together by attribute and assign values to each event. And success metrics can be measured from any page, not just the page(s) you’re testing. Also, offline conversions can be incorporated as well, such as telephone sales or conversions through other offline campaigns.
Top
I hear a lot of people talk about Taguchi—what is this?
Genichi Taguchi was a Japanese engineer who is regarded as one of the pioneers in the field of Design of Experiments. His work enabled experimenters to sample smaller groups within larger populations and get statistically valid information on all members of that population. Taguchi published his first book on experimental design in 1958, and while revolutionary at the time, he did not anticipate today’s computers or the Internet. Taguchi arrays are "pre-planned" fractional factorial designs, which allow experimenters to test a much smaller subset of permutations using a set formula. However, Taguchi’s designs do not take into account possible interactions between variables being tested. This limitation, particularly as it relates to websites (where almost all content has some relationship to other content on the page), severely limits the scope of what can be tested. Therefore, although Optimost is "design agnostic" and can implement Taguchi design if appropriate, it’s usually deemed not an ideal solution. Please see our section on Design of Experiments to learn more.
Top
What is Optimal Design and how does it differ from other testing methods?
Optimal Design is a newer methodology that was developed by the world’s leading experts on Design of Experiments. Optimost, in consultation with leaders in the field of experimental design from institutions including Harvard Business School, Columbia Business School and the Yale School of Management, has further refined the Optimal Design methodology to take advantage of the unique requirements and possibilities of the Internet. Optimal Design allows marketers to take advantage of fractional factorial designs (which allow you to test a small sample of permutations and yet simulate an entire population), while at the same time taking into account interactions and constraints among content elements on a page. Therefore, unlike other designs, with Optimal Design there is no limitation on the type or amount of content that can be included in an optimization experiment.
Top
How does Optimost determine the sample size for a test?
The Optimost system is able to automatically determine when a test is reaching, or has reached, statistical significance. The Optimost reporting in the Campaign Management Console (CMC) automatically displays the confidence interval and level for each creative and variable being tested. However, before an experiment is launched, Optimost can estimate the sample size needed based on a number of factors, including the current conversion rate, the difference in lift among the creatives tested, the number of variables tested, and the "confidence" required by the tester.
Top
What is the purpose of a testing "wave"?
A "wave" is a mini-test that usually runs for about a week. It serves three main purposes: (1) it gives the marketer learnings very quickly as to what is working and not working in any experiment; (2) it creates a "survival of the fittest" paradigm that informs the design of future waves and allows the tester to optimize the site with the smallest number of creatives, using Optimal Design; and (3) it allows the validation and confirmation of results from one wave to the next, reducing the "false positives" that inevitably appear in any experiment.
Top
