There are endless options for A/B testing on your application or website. It’s important to come up with a hypothesis and test it to truly understand if there’s a more optimal way to present information to your users or customers – including shipping methods! In this case, we want to understand the impact of free versus paid shipping tiers on conversion.
Use this play to improve checkout times and find out what shipping methods have the greatest impact on a successful checkout flow by mitigating cart abandonment.
Step 1: Bring A/B test details into Heap
For teams testing multiple versions of shipping options, it is important that all of your variation details are in Heap to run analysis on. In this Play, we will examine the impact of free shipping options versus paid shipping tiers during checkout via A/B testing.
If you are using a testing tool for your A/B variations, you can check out this list of integrations to see if your testing tool has a native integration. If you are not using a testing tool that has a native integration with Heap, no fear! You can always use Heap’s API capability to bring in custom properties.
Note: If enriching via API
It is vital this information is brought into Heap in a way that allows you to properly run your analysis. Your CSM can discuss with you the difference between user and event properties and the impact it will have on the types of reports you can run, otherwise refer to this Track article.
Step 2: Create appropriate Segments
Create behavior based segments of users who have been exposed to different shipping method variations that can then be used in analysis. For details on the importance of segments or how to make a segment, check out this article.
Tip: Feel free to change the segment’s time range from “at any time” to a time-bounded window. If completing an order permanently changes the way you think about a user’s lifetime value, then it may make sense to define this segment as “at any time.” However, if completing an order, or submitting multiple orders has a time-limited effect on a user (e.g. may increase their likelihood to purchase within a month) then change the time range accordingly.
Step 3: Analyze!
| Queries #1-3 | Baseline Reports: Graphs |
|---|---|
| Graph 1 | Conversion Rate Between the Begin Checkout Flow and Select Shipping Method |
| Graph 2 | Conversion Rate Between the Begin Checkout Flow and Submit Order |
| Graph 3 | Count Unique Purchase – Group by: A/B Variant segment(s) |
Establish baselines for what your purchase completion rates are and what the purchase conversion rate is after someone has been exposed to the different shipping variants. This helps you isolate the areas in your checkout flow and the shipping selection you may want to optimize.
| Query #3 | Determine which shipping variant is most desirable |
|---|
| Funnel | – Each Checkout Flow Step – Group By: A/B test experiment property – Choose a Date Range: while experiment was live |
This identifies which shipping preference leads to a higher checkout conversion, and helps surface if any shipping methods are overall undesirable.
| Query #4 | Verify impact of shipping variations on time to checkout |
|---|
| Graph | – Time Between: “Select Shipping” or Equivalent and “Continue to Payment” or “Submit Order” – Group By: A/B test experiment property |
This tells you if a given shipping method is preferable optimal at moving folks through your funnel. It also indicates if certain shipping methods create friction in a critical step of your checkout funnel.
You could also consider adjusting the cost of the goods and showing free shipping if confirmed it’s more desirable without losing revenue. Especially in competitive markets online, the more competitive your rate for shipping of a similar product the likelihood of increasing checkout goes up. People search for shipping prices and if you offer competitive rates to reduce likelihood of finding the product elsewhere.
| Query #5 | Verify impact of shipping variations on retention |
|---|
| Retention | – Start Event: Submit Order – Return Event: Submit Order – Compare Users: A/B variant segment(s) |
Does a particular shipping variant lead to higher retention? Are uses who don’t have to pay for shipping more likely to complete additional orders, or are users who pay fo expedited shipping more likely to complete additional orders?
Step 4: Interpret your results and take action
Is conversion higher for users who have a free shipping option, rather than just paid tiers?
If yes, then you will want to continue optimizing the checkout process and include a free shipping method moving forward. Continue digging into analysis to understand if any marketing efforts have had an impact on completion success, or if there are any other actions successful purchasers take before completing the purchase. Even bundling the shipping cost into the total cost of a product and running an additional A/B test could provide additional insight on how to best optimize the checkout flow.
If no, then you will want to continue optimizing your purchase completion by digging deeper into drop off points, analyzing what users are doing instead of completing a purchase and run some A/B tests, if you have not already done so.
Conclusion
Small changes to the checkout flow can have a major impact on customer experience. Ensure your customers have a quality purchase experience by optimizing all aspects of the checkout process, even with something as simple as shipping options!