To make the most of this guide, you’ll need a baseline understanding of key concepts like events, charts, and properties. If you are still learning about Heap (meaning those terms don’t mean anything to you), we recommend taking our Hello Heap course or reviewing our Setting Up Heap guide prior to jumping into this guide.
Introduction
There are endless options for A/B testing on your application or website. It’s important to come up with a hypothesis and test it to truly understand if there’s a more optimal way to present information to your users or customers – including shipping methods! In this case, we want to understand the impact of free versus paid shipping tiers on conversion.
Use this guide to improve checkout times and find out what shipping methods have the greatest impact on a successful checkout flow by mitigating cart abandonment.
If you’re new to analysis in Heap, we recommend reviewing Create Your First Chart, which covers helpful charts 101 info.
Step 1: Bring A/B test details into Heap
For teams testing multiple versions of shipping options, it is important that all of your variation details are in Heap to run analysis on. In this guide, we will examine the impact of free shipping options versus paid shipping tiers during checkout via A/B testing.
If you are using a testing tool for your A/B variations, you can check out this list of integrations to see if your testing tool has a native integration. If you are not using a testing tool that has an official integration with Heap, no fear! You can always use Heap’s API capability to bring in custom properties.
Note: If enriching via API
It is vital this information is brought into Heap in a way that allows you to properly run your analysis. Your CSM can discuss with you the difference between user and event properties and the impact it will have on the types of charts you can run, otherwise refer to this Track article.
Step 2: Create appropriate segments
Next, create behavior-based segments of users who have been exposed to different shipping method variations that can then be used in analysis.
Feel free to change the segment’s time range from “at any time” to a time-bounded window. If completing an order permanently changes the way you think about a user’s lifetime value, then it may make sense to define this segment as “at any time.” However, if completing an order, or submitting multiple orders has a time-limited effect on a user (e.g. may increase their likelihood to purchase within a month) then change the time range accordingly.
Step 3: Analyze!
Charts 1-3: Usage over time
To start your analysis, set up the following usage over time charts:
- Conversion rate between the checkout flow and select shipping method
- Conversion rate between the begin checkout flow and submit order
- Count unique purchase grouped by A/B variant segment(s)
What do these tell you?
These charts help you establish baselines for what your purchase completion rates are and what the purchase conversion rate is after someone has been exposed to the different shipping variants. This helps you isolate the areas in your checkout flow and the shipping selection you may want to optimize.
Chart 4: Determine which shipping variant is most desirable
Next, you’ll set up a funnel chart and add each step of your checkout flow. Group by your a/b test experiment property, and choose a date range for the duration of time during which the experiment was live.
What does this tell you?
This identifies which shipping preference leads to a higher checkout conversion, and helps surface if any shipping methods are overall undesirable.
Chart 5: Verify impact of shipping variations on time to checkout
To hone in on your shipping estimate UX, set up a usage over time chart to measure the time between “Select Shipping” or Equivalent and “Continue to Payment” or “Submit Order”. Group by your a/b test experiment property.
What does this tell you?
This tells you if a given shipping method is preferable optimal at moving folks through your funnel. It also indicates if certain shipping methods create friction in a critical step of your checkout funnel.
You could also consider adjusting the cost of the goods and showing free shipping if confirmed it’s more desirable without losing revenue. Especially in competitive markets online, the more competitive your rate for shipping of a similar product the likelihood of increasing checkout goes up. People search for shipping prices and if you offer competitive rates to reduce likelihood of finding the product elsewhere.
Chart 6: Verify impact of shipping variations on retention
Dig in deeper into your shipping UX by setting up a retention chart with the following conditions:
- Start Event: Submit Order
- Return Event: Submit Order
- Compare Users: A/B variant segment(s)
What does this tell you?
Does a particular shipping variant lead to higher retention? Are uses who don’t have to pay for shipping more likely to complete additional orders, or are users who pay fo expedited shipping more likely to complete additional orders?
Step 4: Interpret your results and take action
Is conversion higher for users who have a free shipping option, rather than just paid shipping tiers?
If yes, then you will want to continue optimizing the checkout process and include a free shipping method moving forward. Continue digging into analysis to understand if any marketing efforts have had an impact on completion success, or if there are any other actions successful purchasers take before completing the purchase. Even bundling the shipping cost into the total cost of a product and running an additional A/B test could provide additional insight on how to best optimize the checkout flow.
If no, then you will want to continue optimizing your purchase completion by digging deeper into drop off points, analyzing what users are doing instead of completing a purchase and run some A/B tests, if you have not already done so.
Conclusion
Small changes to the checkout flow can have a major impact on customer experience. Ensure your customers have a quality purchase experience by optimizing all aspects of the checkout process, even with something as simple as shipping options!
Additional eCommerce events (optional)
As you continue to optimize your customers’ checkout experience, you may want to define additional events so that you can be more specific in your analysis. Below are some common eCommerce events:
| Event | Description | Common Definition Structure |
|---|---|---|
| Submit – Search | When a user submits a string in a product search bar | Submit on .search-form |
| View – Product Category Page | When a user views a product category page | View page * Add filters for all product category paths |
| View – Product Page | When a user views a product landing page | View page * Add filters for all product page paths |
| View – Product Reviews | When a user views a product review page | View page * Add filters for all product review page paths |
| Click – Add to Cart | When a user clicks ‘add to cart’ | Click on .add-to-cart-button |
| View – Cart | When a user views the checkout page | View page /cart |
| Click – Remove From Cart | When a user clicks ‘remove item from cart’ | Click on .remove-from-cart-button |
| Click – Apply Promo Code | When a user successfully enters a promo code and the promo code is successfully applied to the cart | Click on .apply-promo-code-button |
| Click – Checkout (Start) | When a user clicks ‘start checkout’ | Click on .checkout-button |
| Click – Place Order | When a user clicks the button to place/submit their order (this does NOT include payment confirmation of the order) | Click on .place-order |
| View – Marketing Page | When a user views a marketing page, like a ‘why [your product]’ page | View page /why-heap/* |
| View – Blog Post | When a user views a blog post | View page /blog/* |
| View – Landing Page | When a user views a landing page | View page / Optional: filter by domain (ex. ‘Domain equals learn.heap.io’) |
| Click – Call to Action | When a user clicks a marketing CTA | Click on .free-trial-signup-button |
| Submit – Product Score | When a user submits a product score review | Submit on .product-score |
| Completed Order [Custom event via API] | When a user completes ‘purchase from cart’ and the payment processor returns a success | Source set to ‘custom’ with the custom event defined via a Heap API selected, ex. ‘Complete Purchase’ |
| Account Created [Custom event via API] | When a user successfully signs up for an account and gets a unique userID | Source set to ‘custom’ with the custom event defined via a Heap API selected, ex. ‘Account Created’ |
As a quick way to get started measuring impact, Heap’s impact calculator allows you to measure the potential impact of any change so you can compare the value of different initiatives you’re considering. To learn more about how the calculator works, read our best practice guide.