Split testing is great: Test version A against version B, watch the results roll in, and choose the winner. Really simple and the rationale behind the testing is transparent: Customers respond better to one version than the other and the winning creative is used in the future.

But what happens when marketers want to test lots of variables? If simple splits are used to test a lot of variables, the construction of the test pages becomes extremely tedious and the testing itself can be prohibitively time consuming.

To address this when optimizing websites, many companies use multivariate (or multivariable) testing. Multivariate testing uses some clever statistical tricks that allow marketers to test many variables more quickly and in a smaller population than would be possible with traditional split testing.

The advantages of testing multiple variables at once are obvious. If marketers can test a lot of variables all at once, they can get the results faster, respond to the market more quickly, and stay competitive. Also, since so many variables can be tested at once, testing is more inclusive because each individual element is less expensive (in terms of time and energy) to test. This allows marketers to test variables that aren't necessarily at the top of the list, but that might still have an impact.

Set Up

As with any test, the place to start is to decide what success from this test will look like. More sales? longer visitor time on the site? more page views per visitor? Deciding this early on will help direct which variables to test and how to analyze the results.

In multivariate tests, marketers can test multiple variables and multiple versions of each of the variables. Typical examples of variables on a landing page are a headline, a graphic, and the background color of the page. For each of these variables, variations are then constructed. There might be two or three variations for each variable. For instance, headline 1 might be tested against headline 2, graphic 1 against graphic 2, background color 1 against background color 2 against background color 3... well, you get the idea.

In a multivariate test, splits are constructed so they combine the variations in unique ways. In order for the results to be valid, the testing protocol with the different combinations has to be followed exactly.

The set up for multivariate testing is complicated as is the analysis. There are multiple companies that will assist with this type of optimization. The current favorite among many, though, appears to be the Google Website Optimizer on the basis that it is free. (This is available only to AdWords advertisers.)

Results

How customers respond to the splits is tracked, and the relative contribution of each change can be sorted out through clever statistical analysis. When all the results are combined and analyzed, a "best" page is derived. This is then back tested against the control.

As with any other type of testing, multivariate testing is an iterative process. While it is almost guaranteed that multivariate testing will improve results, it does not improve them permanently. Constant testing is needed to make sure the page stays abreast of changes on the web and with customers.

More Links

Landing Page Optimization Using the Taguchi Method: A great case study. This is highly recommended for anyone who wants a better understanding of how these tests work.

Multivariable Testing: Marketing Experiments consistently produce interesting and useful articles. This is a good overview of multivariate tests.

Google Website Optimizer