I have been looking for a lightweight solution to split test static sites. This blog is a static site generated with Jekyll, and I want to run experiments that might help people get more value out of the things that I publish here. For example, I think the current email subscription call to action is pretty weak, so it would be useful to try to run an experiment around it. Also, I want to beef up my experimental design skills and this seems like a low-risk way to do it.

In this post I’ll tell you how I evaluated a few different options for split testing and why I decided to fork one and open source it.

A potential solution

To start, I didn’t want to use something that required me to set up a server, and I’m guessing that the overall amount of testing won’t require a fancy dashboard. If those were the case, Sixpack seems like an interesting approach.

I found a library called ABalytics that uses Google Analytics to set up split tests. At first, it seemed like ABalytics did everything that I wanted:

• you can specify split test experiments to run
• there can be multiple experiments running at a time
• it stores the tests in an analytics service (Google Analytics)
• it runs with vanilla JavaScript on the client
• there is no server component
• it stores which variant the user has seen and presents it to them if they come back

However, after installing and playing around with it, it seems like the mechanism for how it works in Google Analytics (“Custom Variables”) doesn’t seem to work any more. I think Google Analytics changed the way that they handle these custom variables. Also, it was hard to test since Google Analytics doesn’t really work unless it is on your production site and there is high latency between when you send an event and when Google Analytics reports the event.

Changing ABalytics to use Heap

I considered rolling my own static site split testing framework, but was pretty happy with the feature set of ABalytics. So I decided it would be easiest to fork it and change it to work with Heap Analytics instead.

Heap seems to be a good competitor to Google Analytics. One of the most powerful features of Heap that it stores all of the actions that the user took, and you can retroactively set up events and funnels for tracking the events. So you don’t need to know exactly what you want to track, you can figure this out when you actually want to know it. This seems to be really helpful because unnecessary setup is a large form of waste. Also, you won’t lose days or weeks of data by forgetting to track something important.

Heap tracks most of the same things that Google Analytics tracks, like browser type, number of visits, sessions by a particular user, referral information, and so on. Events that you send to Heap show up almost immediately, so it’s easier to experiment with.

Digging in

First, I had to consider the license of the original code. Since ABalytics uses the MIT license, licensing was not an issue. I just added another copyright line with my information since part of the license requires retaining the existing license when distributing. I don’t think that I would need to change the license if I was using the code for just my site, but it seemed like it would be easy to open source my changes so I wanted to do that. This has the benefit of sharing my work with others and also potentially getting outside improvements.

The key change was changing from using Google Analytics to using Heap. I needed to tell Heap what experiments were being run for the user viewing the page. If we send the experiment information as event properties for the user, all events sent to Heap will be associated with this experiment information. To do this, we just set up a hash with the experiment data and send it to Heap:

HeapWrapper.prototype.recordExperiment =
        function (experimentName, variantName) {
    var properties = {}
    properties['experiment_' + experimentName] = variantName;
    this.heap.setEventProperties(properties)
};

I figured since Heap would potentially get a lot of event data, that I would prefix any experiments with experiment_ before sending.

How it went

I successfully resisted the desire to change a bunch of things, since I didn’t know whether my approach would work at all and I wanted the diff to be as small as possible. I reasoned that I could change the code to my liking after I had a good solution locally and pushed up. I tested the changes locally, and it seemed to come together quickly.

Heap sets up a development and production environment by default, and they are switchable by setting a variable. With this, I am able to test locally and won’t impact my production data. I have a client snippet that looks at the location and uses the production environment identifier if it is running on my blog and otherwise uses the development identifier. This keeps complexity down by not needing to know the environment when building the site.

The source is available at the HeapAB repo. The instructions in the README should be enough to get up and running, but let me know if you run into anything that is tough or could be explained better. I’d like to add a few features and clean it up, but it should be helpful for people today.

I’m not currently running any split tests, but am looking forward to using this library for improving this blog.