Skip to content
  • Home
  • Developers
  • API
  • Releases
  • Community
  • University
  • Status
  • Home
  • Developers
  • API
  • Releases
  • Community
  • University
  • Status
Home Heap Plays Test the Impact of Different Shipping Method Options
Session Replay Getting Started Administration Data Management Definitions Charts Analysis Examples Heap Plays Integrations Heap Connect Data Privacy

Table of Contents

Was this article helpful?

Yes No

Thank you for your feedback!

Test the Impact of Different Shipping Method Options

There are endless options for A/B testing on your application or website. It’s important to come up with a hypothesis and test it to truly understand if there’s a more optimal way to present information to your users or customers – including shipping methods! In this case, we want to understand the impact of free versus paid shipping tiers on conversion.

Use this play to improve checkout times and find out what shipping methods have the greatest impact on a successful checkout flow by mitigating cart abandonment.

Step 1: Bring A/B test details into Heap

For teams testing multiple versions of shipping options, it is important that all of your variation details are in Heap to run analysis on. In this Play, we will examine the impact of free shipping options versus paid shipping tiers during checkout via A/B testing.

If you are using a testing tool for your A/B variations, you can check out this list of integrations to see if your testing tool has a native integration. If you are not using a testing tool that has a native integration with Heap, no fear! You can always use Heap’s API capability to bring in custom properties.

Note: If enriching via API

It is vital this information is brought into Heap in a way that allows you to properly run your analysis. Your CSM can discuss with you the difference between user and event properties and the impact it will have on the types of reports you can run, otherwise refer to this Track article.

Step 2: Create appropriate Segments

Create behavior based segments of users who have been exposed to different shipping method variations that can then be used in analysis. For details on the importance of segments or how to make a segment, check out this article. 

Tip: Feel free to change the segment’s time range from “at any time” to a time-bounded window. If completing an order permanently changes the way you think about a user’s lifetime value, then it may make sense to define this segment as “at any time.” However, if completing an order, or submitting multiple orders has a time-limited effect on a user (e.g. may increase their likelihood to purchase within a month) then change the time range accordingly.

Step 3: Analyze!

Queries #1-3Baseline Reports: Graphs
Graph 1Conversion Rate Between the Begin Checkout Flow and Select Shipping Method
Graph 2Conversion Rate Between the Begin Checkout Flow and Submit Order
Graph 3Count Unique Purchase 
– Group by: A/B Variant segment(s)
What do these tell you?
Establish baselines for what your purchase completion rates are and what the purchase conversion rate is after someone has been exposed to the different shipping variants. This helps you isolate the areas in your checkout flow and the shipping selection you may want to optimize.
Query #3Determine which shipping variant is most desirable
Funnel– Each Checkout Flow Step
– Group By: A/B test experiment property
– Choose a Date Range: while experiment was live
What does this tell you?
This identifies which shipping preference leads to a higher checkout conversion, and helps surface if any shipping methods are overall undesirable.  
Query #4Verify impact of shipping variations on time to checkout
Graph

– Time Between: “Select Shipping” or Equivalent and “Continue to Payment” or “Submit Order”
– Group By: A/B test experiment property
What does this tell you?
This tells you if a given shipping method is preferable optimal at moving folks through your funnel. It also indicates if certain shipping methods create friction in a critical step of your checkout funnel.

You could also consider adjusting the cost of  the goods and showing free shipping if confirmed it’s more desirable  without losing revenue. Especially in competitive markets online, the more competitive your rate for shipping of a similar product the likelihood of increasing checkout goes up. People search for shipping prices and if you offer competitive rates to reduce likelihood of finding the product elsewhere.
Query #5Verify impact of shipping variations on retention
Retention

– Start Event: Submit Order
– Return Event: Submit Order
– Compare Users: A/B variant segment(s)
What does this tell you?
Does a particular shipping variant lead to higher retention? Are uses who don’t have to pay for shipping more likely to complete additional orders, or are users who pay fo expedited shipping more likely to complete additional orders?

Step 4: Interpret your results and take action

Is conversion higher for users who have a free shipping option, rather than just paid tiers?

If yes, then you will want to continue optimizing the checkout process and include a free shipping method moving forward. Continue digging into analysis to understand if any marketing efforts have had an impact on completion success, or if there are any other actions successful purchasers take before completing the purchase. Even bundling the shipping cost into the total cost of a product and running an additional A/B test could provide additional insight on how to best optimize the checkout flow.

If no, then you will want to continue optimizing your purchase completion by digging deeper into drop off points, analyzing what users are doing instead of completing a purchase and run some A/B tests, if you have not already done so. 

Conclusion

Small changes to the checkout flow can have a major impact on customer experience. Ensure your customers have a quality purchase experience by optimizing all aspects of the checkout process, even with something as simple as shipping options!

Have feedback? Let us know!

Was this article helpful?

Yes No

Thank you for your feedback!

Last updated January 29, 2021.

heap playin-app guidekey features
  • Blog
  • Partners
  • Legal
  • Security
  • Terms
  • About
  • Careers
  • Privacy
  • Contact Us

© 2023 Heap, Inc.