A/B Testing with Sitecore

With A/B testing, marketers optimize their conversion metrics by presenting different marketing messages and testing their results. Wikipedia defines A/B testing as:

A/B testing is a method of advertising testing by which a
baseline control sample is compared to a variety of single-variable test samples. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.

Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome.

Let's consider the components of A/B testing as they play out in Sitecore. First, we need to define the test. Typically, web tests will consist of at least one of several attributes:

1) Text
2) Graphics
3) Page layout

A text-based test could be testing different marketing messages; a graphical change could include different ad copy, varied product images, etc.; a layout change could place different items on different locations on the page (to determine, for example, whether placement in the center of the page increases sales).

For our purposes, imagine a promotion wherein two different offers are being presented to a customer: 10% off or 20% off. The marketing goal is to determine the price sensitivity of site visitors. Marketers can use different text or graphics to present the offers and then test the effectiveness of each.

There are many approaches to defining tests in Sitecore but the primary driver behind test definition should be usability for marketing authors. You might explore Sitecore's Wizard Framework (requires SDN access) to create a simple wizard for test generation. A simpler approach could be to use content markers to allow authors to create a new test. Consider, for example, the following implementation:

Test variations can be stored as children of a primary content item or can be stored elsewhere in the content tree. Regardless of where the are stored, the implementation should result in a usable experience for test authors.

In addition to determining presentation and content elements to vary with each test, each test will require a duration and a distribution. In other words, how long should the test run for and what percentage of users should get each test (or should the distribution be completely random?). Implementions of distribution rules will vary based on the logic of the rules themselves: Will tests be implemented on separate servers, with distributions handled by the load balancer? Will tests be driven by presentation logic? Will a 3rd-party test engine be integrated with the system?

To evaluate the results of the test you will need to define a conversion event. The conversion will typically involve clicking on a link or a button in a form. You will need to think carefully about what a conversion means for a particular page. Does the conversion occur on this page? Or does it occur on a subsequent page (such as completing a transaction vs. simply adding an item to your shopping cart).

To view conversion metrics, business users can view reports either within the Sitecore Desktop or in a reports module viewable only in WebEdit mode. Conversion metrics should be both clear and actionable. The relevance of conversion metrics will be directly related to the considerations accounted for in the test definition. For help with defining tests, consider working with an experienced Sitecore partner, integrating with a 3rd-party rules engine or reading up on best practices in A/B testing.