Overview

A/B testing is essential to validating that changes to your product will help you achieve your business goals. In this guide, we’ll walk through the following examples using Heap for A/B testing:

Example 1: Increase Feature Usage

Your product team is trying to increase the usage of a particular feature. You use an A/B test to try out two different placements of the button that activates that feature to see which one results in increased engagement.

Example 2: Testing Homepage Taglines

Your marketing team is trying to choose between two different taglines for your homepage to see which results in increased trial signups. For your A/B test, you may try out one homepage with tagline A, and another with the tagline B, then compare engagement with these pages.

Step 1: Determine goals

Before setting up your A/B test, it’s important to decide on the desired outcome so you can determine what data to capture. Do you want to increase signups, purchases, or use of a key feature? If so, what user activity represents this (clicks on a button, views of a page)? Take a moment to write this down to reference later in the process.

Once you’ve determined what information you need to have in Heap to analyze this experiment, you’re ready to set up Heap or one of our integrations to capture this data.

Step 2: Capture test data

There are several ways you can get your experiment data into Heap:

Integrations

Heap has direct integrations with Optimizely X, Oracle Maxymizer (enterprise plans only) and Visual Website Optimizer (VWO) to get your experiment data into your account.

Specific recommendations for using Optimizely X and VWO to capture user test data and experiment test data are provided below.

Optimizely Classic & Optimizely X Experiments

If you're running experiments through Optimizely, it's easy to send over experiment information to Heap using the addUserProperties API. Sending experiment data to Heap can help you see more granular information about conversions and usage.

If you're using Segment you can skip this entirely, and instead go to your Optimizely integration and check the box next to Send experiment data to other tools (as an identify call).

This example assumes you've already installed Optimizely and have started running experiments.

You can use Optimizely's API to send the names of all the active experiments, along with their variations to Heap. Optimizely stores all the necessary information in the optimizely object, described in more detail below:

Active Experiment IDs

optimizely.activeExperiments is an array with the list of active experiment IDs, it looks like this: ["8675309", "72680"]

Experiment Names

Optimizely provides an object called optimizely.data.experiments which lets you map an experiment ID to more detailed information about the experiment, though all we need to know is that there is also a property called name. For example, using the experiment ID from above, 8675309, we can find out the name for that experiment with optimizely.data.experiments["8675309"].name.

Variation IDs

Each experiment can have a number of variations. Just like experiments, they are referenced by their variation ID, which can be mapped to a variation name. We're using names for this example because it will make the data easier to understand when it's in Heap. We can read variation names directly with the experiment IDs from above, though in case you're curious, the variation IDs are available in optimizely.variationMap.

Variation Names

The names of the experiments and variations are labeled initially by you in the Optimizely dashboard. This example uses the API to read those names. To get the variation name for one of the active experiments, you can use optimizely.variationNamesMap["8675309"]

Bringing it All Together

If you're using Optimizely Classic, use this code block:

// Create an object to store experiment names and variations
var props = {}

for (idx in optimizely.activeExperiments) {
    props[optimizely.data.experiments[optimizely.activeExperiments[idx]].name] = optimizely.variationNamesMap[optimizely.activeExperiments[idx]];
}

heap.addUserProperties(props)

If you're using Optimizely X, use this code block:

// Create an object to store experiment names and variations
var props = {}

// This covers both A/B Experiments and Peresonalization
var camState = optimizely.get("state").getCampaignStates({"isActive": true});

    for (x in camState) {
        if(camState.isInCampaignHoldback != true){
            props[camState[x].id] = camState[x].variation.name;
        }

    else {
            props[camState[x].id] = "Campaign Hold Back";
        }
    }
heap.addUserProperties(props);

If we have two experiments running, called Experiment #1 and Experiment #2, and the visitor was in Variation A and Variation B respectively, the resulting props object would look like this:

{"Experiment #1": "Variation A", "Experiment #2": "Variation B"}

Since heap.addUserProperties takes an object as its only parameter, this maps directly.

Visual Website Optimizer (VWO)

Visual Website Optimizer (VWO) is another popular A/B testing platform, and integrating it with Heap works similarly to the example given above. The code below converts VWO experiment data into an object which gets passed to heap.addUserProperties. It assumes you have already installed VWO on your site.

Put this code on every page on which you're running experiments, after both the VWO snippet and the Heap snippet. For more information, check out VWO's documentation on Integrating VWO with Heap.