Caching

If you are using split on a high traffic website you'll likely be using some variety of caching in your application.

This can interfere with running A/B tests as you may cache the template of an alternative and end up serving that to all users, invalidating your test.

Below are a number of different caching techniques used in Rails (and sinatra) and how best to use Split with them.

Note: These examples and suggestions are for starting experiments rather than completing them, which usually involves writing to a database or some other controller based logic which is unlikely to be cached.

Page Caching

Page caching usually stores the full html contents of the page on disk, often with apache/nginx serving up the html directly, never hitting your application logic.

In this situation you can use two different methods for starting the test:

• Use different urls for alternatives - redirect a user though a controller action which sets the alternatives for a new user and redirects to the correct url for that alternative (a page cached template).

• Alter the page with javascript instead of in the template - this involves creating a (uncached) controller action that returns some json with the alternative a user should be seeing, on page load some javascript will load this json and then running a callback which customises the page for that alternative.
  A split extension is in the works to easily provide this functionality: https://github.com/andrew/split-api

Fragment Caching

Fragment caching stores a segment of a page in cache referenced by a key which is then rendered or created by your application as the template is rendered.

This is the easiest form of caching to integrate with split as you can add your alternative name to the fragment's cache key, rendering the correct fragment for the alternative that the current user should be seeing

<% alternative = ab_test('design_change', 'old', 'new') %>
<% cache("recommendations-#{alternative}") do %>
  <div id='<% alternative %>'>
    <% slow_loading_database_call(alternative) %>
  </div>
<% end %>

Action Caching

Action caching is where you store the result of the controller action of a page but allows before filters such as authentication. The some of the options for starting an ab_test from a cached action are:

• A before filter which redirects to a separate url or controller action based on the alternative.
• Include the ab_test alternative in the cache_path option to effectively create to cache options without extra urls.
• Use the javascript based method as mentioned in Page Caching

Proxy caching

Proxy caching is where you place software such as Varnish between your web server and your users, it then uses the standard http expires and cache control headers to determine if the page should be cached or reloaded from the rails app.

Often this will mean that a user never interacts with the routing or controller logic of your application. The same options as page caching can be used when a/b testing through a proxy cache.

You can also include the ab_test alternative in the generation of etags which will ensure that users on separate alternatives never get an incorrectly cached page, this may be too inefficient though.

response.etag = @etag = [
  User.find(params[:id]),
  ab_test('user_page', 'old', 'new')
]

if request.fresh?(response)
   head :not_modified
else
  render :action => 'index'
end