Skip to content
  • Home
  • Developers
  • API
  • Releases
  • Community
  • University
  • Status
  • Home
  • Developers
  • API
  • Releases
  • Community
  • University
  • Status
Home Analysis Guides Product Test the Impact of Different Shipping Method Options
Getting Started Session Replay Data Management Data Types Charts Chart Examples Dashboard Templates Analysis Guides Administration Integrations Heap Connect Data Privacy

Table of Contents

Was this article helpful?

Yes No

Thank you for your feedback!

Test the Impact of Different Shipping Method Options

In this article you'll learn:

  • What charts to measure a/b test results for different shipping options
  • Guidance for analyzing these results and optimizing your overall purchase experience
This doc is for: Product Managers eCommerce Businesses

To make the most of this guide, you’ll need a baseline understanding of key concepts like events, charts, and properties. If you are still learning about Heap (meaning those terms don’t mean anything to you), we recommend taking our Hello Heap course or reviewing our Setting Up Heap guide prior to jumping into this guide.

Introduction

There are endless options for A/B testing on your application or website. It’s important to come up with a hypothesis and test it to truly understand if there’s a more optimal way to present information to your users or customers – including shipping methods! In this case, we want to understand the impact of free versus paid shipping tiers on conversion.

Use this guide to improve checkout times and find out what shipping methods have the greatest impact on a successful checkout flow by mitigating cart abandonment.

Step 1: Bring A/B test details into Heap

For teams testing multiple versions of shipping options, it is important that all of your variation details are in Heap to run analysis on. In this guide, we will examine the impact of free shipping options versus paid shipping tiers during checkout via A/B testing.

If you are using a testing tool for your A/B variations, you can check out this list of integrations to see if your testing tool has a native integration. If you are not using a testing tool that has an official integration with Heap, no fear! You can always use Heap’s API capability to bring in custom properties.

Note: If enriching via API

It is vital this information is brought into Heap in a way that allows you to properly run your analysis. Your CSM can discuss with you the difference between user and event properties and the impact it will have on the types of charts you can run, otherwise refer to this Track article.

Step 2: Create appropriate segments

Next, create behavior-based segments of users who have been exposed to different shipping method variations that can then be used in analysis.

Feel free to change the segment’s time range from “at any time” to a time-bounded window. If completing an order permanently changes the way you think about a user’s lifetime value, then it may make sense to define this segment as “at any time.” However, if completing an order, or submitting multiple orders has a time-limited effect on a user (e.g. may increase their likelihood to purchase within a month) then change the time range accordingly.

Step 3: Analyze!

Charts 1-3: Usage over time

To start your analysis, set up the following usage over time charts:

  1. Conversion rate between the checkout flow and select shipping method
  2. Conversion rate between the begin checkout flow and submit order
  3. Count unique purchase grouped by A/B variant segment(s)

What do these tell you?

These charts help you establish baselines for what your purchase completion rates are and what the purchase conversion rate is after someone has been exposed to the different shipping variants. This helps you isolate the areas in your checkout flow and the shipping selection you may want to optimize.

Chart 4: Determine which shipping variant is most desirable

Next, you’ll set up a funnel chart and add each step of your checkout flow. Group by your a/b test experiment property, and choose a date range for the duration of time during which the experiment was live.

What does this tell you?

This identifies which shipping preference leads to a higher checkout conversion, and helps surface if any shipping methods are overall undesirable.

Chart 5: Verify impact of shipping variations on time to checkout

To hone in on your shipping estimate UX, set up a usage over time chart to measure the time between “Select Shipping” or Equivalent and “Continue to Payment” or “Submit Order”. Group by your a/b test experiment property.

What does this tell you?

This tells you if a given shipping method is preferable optimal at moving folks through your funnel. It also indicates if certain shipping methods create friction in a critical step of your checkout funnel.

You could also consider adjusting the cost of the goods and showing free shipping if confirmed it’s more desirable without losing revenue. Especially in competitive markets online, the more competitive your rate for shipping of a similar product the likelihood of increasing checkout goes up. People search for shipping prices and if you offer competitive rates to reduce likelihood of finding the product elsewhere.

Chart 6: Verify impact of shipping variations on retention

Dig in deeper into your shipping UX by setting up a retention chart with the following conditions:

  • Start Event: Submit Order
  • Return Event: Submit Order
  • Compare Users: A/B variant segment(s)

What does this tell you?

Does a particular shipping variant lead to higher retention? Are uses who don’t have to pay for shipping more likely to complete additional orders, or are users who pay fo expedited shipping more likely to complete additional orders?

Step 4: Interpret your results and take action

Is conversion higher for users who have a free shipping option, rather than just paid shipping tiers?

If yes, then you will want to continue optimizing the checkout process and include a free shipping method moving forward. Continue digging into analysis to understand if any marketing efforts have had an impact on completion success, or if there are any other actions successful purchasers take before completing the purchase. Even bundling the shipping cost into the total cost of a product and running an additional A/B test could provide additional insight on how to best optimize the checkout flow.

If no, then you will want to continue optimizing your purchase completion by digging deeper into drop off points, analyzing what users are doing instead of completing a purchase and run some A/B tests, if you have not already done so.

Conclusion

Small changes to the checkout flow can have a major impact on customer experience. Ensure your customers have a quality purchase experience by optimizing all aspects of the checkout process, even with something as simple as shipping options!

Additional eCommerce events (optional)

As you continue to optimize your customers’ checkout experience, you may want to define additional events so that you can be more specific in your analysis. Below are some common eCommerce events:

EventDescriptionCommon Definition Structure
Submit – SearchWhen a user submits a string in a product search barSubmit on .search-form
View – Product Category PageWhen a user views a product category pageView page *
Add filters for all product category paths
View – Product PageWhen a user views a product landing pageView page * 
Add filters for all product page paths
View – Product ReviewsWhen a user views a product review pageView page *
Add filters for all product review page paths
Click – Add to CartWhen a user clicks ‘add to cart’Click on .add-to-cart-button
View – CartWhen a user views the checkout pageView page /cart
Click – Remove From CartWhen a user clicks ‘remove item from cart’Click on .remove-from-cart-button
Click – Apply Promo CodeWhen a user successfully enters a promo code and the promo code is successfully applied to the cartClick on .apply-promo-code-button
Click – Checkout (Start)When a user clicks ‘start checkout’Click on .checkout-button
Click – Place OrderWhen a user clicks the button to place/submit their order (this does NOT include payment confirmation of the order)Click on .place-order
View – Marketing PageWhen a user views a marketing page, like a ‘why [your product]’ pageView page /why-heap/*
View – Blog PostWhen a user views a blog postView page /blog/*
View – Landing PageWhen a user views a landing pageView page /
Optional: filter by domain (ex. ‘Domain equals learn.heap.io’)
Click – Call to ActionWhen a user clicks a marketing CTAClick on .free-trial-signup-button
Submit – Product ScoreWhen a user submits a product score reviewSubmit on .product-score
Completed Order [Custom event via API]When a user completes ‘purchase from cart’ and the payment processor returns a successSource set to ‘custom’ with the custom event defined via a Heap API selected, ex. ‘Complete Purchase’
Account Created [Custom event via API]When a user successfully signs up for an account and gets a unique userIDSource set to ‘custom’ with the custom event defined via a Heap API selected, ex. ‘Account Created’

Was this article helpful?

Yes No

Thank you for your feedback!

Last updated August 3, 2023.

heap playin-app guidekey features
  • Blog
  • Partners
  • Legal
  • Security
  • Terms
  • About
  • Careers
  • Privacy
  • Contact Us

© 2023 Heap, Inc.