Introduction

There are endless options for A/B testing on your application or website. It’s important to come up with a hypothesis and test it to truly understand if there’s a more optimal way to present information to your users or customers - including shipping methods! In this case, we want to understand the impact of free versus paid shipping tiers on conversion.

Use this guide to improve checkout times and find out what shipping methods have the greatest impact on a successful checkout flow by mitigating cart abandonment.

Step 1: Bring A/B test details into Heap

For teams testing multiple versions of shipping options, it is important that all of your variation details are in Heap to run analysis on. In this guide, we will examine the impact of free shipping options versus paid shipping tiers during checkout via A/B testing.

If you are using a testing tool for your A/B variations, you can check out this list of integrations to see if your testing tool has a native integration. If you are not using a testing tool that has an official integration with Heap, no fear! You can always use Heap’s API capability to bring in custom properties.

Step 2: Create appropriate segments

Next, create behavior-based segments of users who have been exposed to different shipping method variations that can then be used in analysis.

Step 3: Analyze!

Charts 1-3: Usage over time

To start your analysis, set up the following usage over time charts:

1. Conversion rate between the checkout flow and select shipping method
2. Conversion rate between the begin checkout flow and submit order
3. Count unique purchase grouped by A/B variant segment(s)

What do these tell you?

These charts help you establish baselines for what your purchase completion rates are and what the purchase conversion rate is after someone has been exposed to the different shipping variants. This helps you isolate the areas in your checkout flow and the shipping selection you may want to optimize.

Chart 4: Determine which shipping variant is most desirable

Next, you'll set up a funnel chart and add each step of your checkout flow. Group by your a/b test experiment property, and choose a date range for the duration of time during which the experiment was live.

What does this tell you?

This identifies which shipping preference leads to a higher checkout conversion, and helps surface if any shipping methods are overall undesirable.

Chart 5: Verify impact of shipping variations on time to checkout

To hone in on your shipping estimate UX, set up a usage over time chart to measure the time between "Select Shipping" or Equivalent and "Continue to Payment" or "Submit Order". Group by your a/b test experiment property.

What does this tell you?

This tells you if a given shipping method is preferable optimal at moving folks through your funnel. It also indicates if certain shipping methods create friction in a critical step of your checkout funnel.

You could also consider adjusting the cost of the goods and showing free shipping if confirmed it’s more desirable without losing revenue. Especially in competitive markets online, the more competitive your rate for shipping of a similar product the likelihood of increasing checkout goes up. People search for shipping prices and if you offer competitive rates to reduce likelihood of finding the product elsewhere.

Chart 6: Verify impact of shipping variations on retention

Dig in deeper into your shipping UX by setting up a retention chart with the following conditions:

• Start Event: Submit Order
• Return Event: Submit Order
• Compare Users: A/B variant segment(s)

What does this tell you?

Does a particular shipping variant lead to higher retention? Are uses who don't have to pay for shipping more likely to complete additional orders, or are users who pay fo expedited shipping more likely to complete additional orders?

Step 4: Interpret your results and take action

Is conversion higher for users who have a free shipping option, rather than just paid shipping tiers?

If yes, then you will want to continue optimizing the checkout process and include a free shipping method moving forward. Continue digging into analysis to understand if any marketing efforts have had an impact on completion success, or if there are any other actions successful purchasers take before completing the purchase. Even bundling the shipping cost into the total cost of a product and running an additional A/B test could provide additional insight on how to best optimize the checkout flow.

If no, then you will want to continue optimizing your purchase completion by digging deeper into drop off points, analyzing what users are doing instead of completing a purchase and run some A/B tests, if you have not already done so.

Conclusion

Small changes to the checkout flow can have a major impact on customer experience. Ensure your customers have a quality purchase experience by optimizing all aspects of the checkout process, even with something as simple as shipping options!

Additional eCommerce events (optional)

As you continue to optimize your customers' checkout experience, you may want to define additional events so that you can be more specific in your analysis. Below are some common eCommerce events: