Glossary of Terms

The following is a glossary of terms related to Design of Experiments and testing styles.

A/B Testing, Split Testing, or Champion-Challenger Testing

A/B Testing is the most simplistic way of conducting direct marketing and website tests. In such tests, the "A" option is the control, or current champion. The "B" option is the challenger being tested in an attempt to provide better results than "A." During a split run, visitors are randomly shown or offered the "A" or the "B" option. The difference between the two response rates is then evaluated for statistical significance. While simple to conduct and understand, A/B testing is much less informative and much more costly if more than two factors need to be tested, and has a much lower efficiency than multivariable experimental designs.

Bayes’ Theorem

Being, relating to, or involving statistical methods that assign probabilities or distributions to events or parameters based on experience or best guesses before experimentation and data collection. For example, the probability of an event A conditional on another event B is generally different from the probability of B conditional on A. However, there is a definite relationship between the two, and Bayes' theorem is the statement of that relationship.

Blocked Designs

A design that takes into account variables that are not under the direct control of the experimenter but can be expected to impact the results – for example, the day of the week on which a visitor comes to the site, or the search engine from which they come. "Blocking" becomes useful, or even essential, if we know (or strongly suspect) that such "extraneous" factors will impact the results in a way that might mask the effects of our experimental variables.

Conjoint Analysis

Conjoint analysis is a tool that allows a subset of the possible combinations of product features to be used to determine the relative importance of each feature in the purchasing decision. Conjoint analysis is based on the fact that the relative values of attributes considered jointly can better be measured than when considered in isolation.

Covariates

Covariates are extraneous variables that might have an impact on the experimental results. For example, the income or age of the respondent, the time of day, or a telephone rep's rating on selling skills, might be expected to impact the results of some multivariate tests. Such variables are not under the control of the experimenter, but can be "adjusted" by measuring them and then including them in a covariate analysis. Optimal Designs permit the addition of a few additional combinations to allow for this adjustment.

Experimental Variables/Factors

These are the aspects of the web page or any process that you want to run through a multivariate test. For example, the color of the headline could be one factor, the location of the submit button could be another factor, and so on. For a variable to be an experimental factor, it has to be something that can be manipulated or controlled by the experimenter.

Experimental Values/Levels

The variations that you want to test for each experimental factor are called its levels or values. So, if three background colors are being tested, then the experimental factor background color is said to have three levels or values in the multivariate test.

Fractional Factorial Designs

Fractional Factorial Design allows the tester to test a subset of combinations by focusing only on the most important main effects and interactions. Depending on how many interactions are chosen, the factorial design can be much smaller or only slightly smaller when compared to full factorial testing.

Full Factorial Designs

A Full Factorial Design is one where every possible combination of experimental variables is tested. Full factorial design provides the greatest amount of information about the individual and joint impacts of the experimental variables. However, it also is the most demanding of time and resources. For interactive and direct marketers, full factorial designs are rarely used. Rather, such designs are used as benchmarks in choosing smaller more efficient designs for multivariate testing.

Full Page Multivariable Test

An experiment where every single element on a web page can be tested—headlines, price points, navigation schemes, layouts, images, colors, design, pricing, etc.—as well as any relationships among such elements. For a Full Multivariable Test, an experimental design is normally required.

Multivariable or Multivariate Testing

These terms generally represent a test design where the tester is able to test multiple experimental variables simultaneously, generating multiple permutations/combinations of those variables. Many different test designs can be used for a multivariable test, including Full Factorial and Fractional Factorial. Multivariable testing allows for greater flexibility and scalability during the test than A/B Testing does.

Permutations/Combinations/Recipes

Each combination of experimental factors at various levels is called an experimental run, combination or recipe. For example, suppose there are five experimental factors: Price, Commitment, Logo, Copy, and Endorsement. Now suppose there are two levels (or variations) for each. All together there would be 2^5, or 32, possible combinations of these variables.

Optimal Design

Using modern technology and a variety of optimization algorithms, Optimal Design allows the experimenter the greatest flexibility in specifying which main effects and interactions they want to estimate, which combinations may not be included (or must be included), and how many runs or combinations they want to test at most. Given these inputs and the number of levels for each experimental factor, these algorithms provide the most efficient design possible. Generally, the designs are somewhat larger but much more efficient and flexible than Taguchi arrays.

Orthogonality

Orthogonality is a common term that appears when describing the properties of multivariable experiments. Orthogonality is a mathematical term that is often used in algebra to describe the relationship between two sets of numbers. Experimental designs can be described as orthogonal designs because of the mathematical property that enables multiple variable effects to be estimated simultaneously. Among other criteria, an orthogonal design is one where each variable tested is used an equal number of times and also paired an equal number of times with every other variable.

Page Concept Testing

A form of A/B testing where the marketer is testing two radically different page or funnel concepts. Perhaps the most appropriate time to select A/B testing as the technique.

Plackett-Burman Designs

These pre-planned designs were created to make it easy for the practitioner to conduct an experiment by simply referring to a published source. However, in such designs, if there are any significant interactions among the variables, these designs will not produce accurate results.

Simple Multivariable Test

An experiment where one or more elements on a page are tested in an independent fashion. An Experimental Design may or may not be necessary depending on the scope of the experiment.

Taguchi Method

Genichi Taguchi developed and proposed several pre-planned fractional factorial designs. His designs help reduce the size of an experiment considerably. For example, using just 18 combinations, Taguchi design can simultaneously test seven variables each at three levels and one 2-level variable (a full factorial design would require 3^7 * 2 or 4,374 combinations). However, the ability to estimate important interactions and to deal with more complicated designs is severely compromised.