A/B experiments in reward journey let you run two versions of a journey at the same time, so you can measure which rewards and messaging actually retain subscribers and grow their lifetime value.
Subscribers are split into a control group (A) and a variant group (B), each group runs a different reward strategy, and Loop tracks retention and revenue metrics to determine a winner. This article covers what can be tested, how to set up an experiment, the metrics tracked, how a winner is decided, and how to monitor performance.
If you are new to reward journeys, see Reward journey for how to set one up before adding an experiment.
What can be tested in a reward journey experiment
A reward journey experiment lets you test two things:
the reward on a milestone
the way that reward is communicated
For the action itself, control (A) and variant (B) can each be set to any of the actions in the Add action menu, so any two actions can be compared head to head at the same milestone. The available actions are:
Subscription based: add discount, remove discount, mystery reward
Product based: add product one-time, add product as subscription
Customer based: add customer tag, remove customer tag
Common comparisons include discounts vs. mystery reward, mystery discount vs. mystery gift, discount A vs. discount B, and reward vs no reward.
For communication, you can test:
Customer portal reward banner content (A vs. B, or content vs. none)
Reward notification email content (A vs. B, or content vs. none)
Add reward text in the upcoming order email (A vs B, or content vs. none)
Setting up an A/B experiment in reward journey
Navigate to Loop admin > Retain > Flows, then create or edit a reward journey.
Add an internal name for easier management, then define conditions that may be relevant for setting up the reward journey.
In the experimentation block, click Create experiment.
Enter an Internal name for the experiment.
Under Audience split and tags, set the percentage of eligible subscribers assigned to control group (A) versus variant group (B), for example 50/50 or 70/30. You can also add a customized control group tag and a variant group tag to identify subscribers in each group.
Set the enrollment duration and experiment duration.
Enrollment duration is the window during which new subscriptions join the experiment. A subscription enters the experiment when it is created within this window, and the customer is randomly assigned to control group (A) or variant group (B) at that point.
Experiment duration is the full period the experiment runs and collects data. A subscription remains in its assigned group for the entire experiment duration, even as its eligibility for the reward journey changes across recurring orders.
Because the reward journey triggers on each recurring order, an enrolled subscription may qualify or disqualify for the journey over time based on its eligibility criteria and state. However, its group assignment does not change, so a subscriber assigned to A always sees A, and a subscriber assigned to B always sees B, for the length of the experiment.
Note that subscriptions enroll only when they are created, not from a later order milestone.
Under Metrics to be tracked, select the metrics to measure for each group, and choose one as the Primary metric. The primary metric is used to decide the winner. All the metrics are defined in the next section.
Under Winner calculation, set the minimum percentage difference required in the primary metric to declare a winner. Loop compares the primary metric between control (A) and variant (B) as a relative difference, and declares a winner only when that difference meets or exceeds the minimum set.
For instance, if the primary metric selected here is "Order attempt rate", control (A) is at 50%, and variant (B) is at 55%, then B is performing 10% better on a relative basis (5 ÷ 50), which meets a 10% minimum. If B were at 48%, the relative difference would still be only 4% (2 ÷ 50), and no winner would be declared against a 10% minimum.
Configure rewards for the control and variant by adding one or more actions for each order configured in the flow. If you are new to reward journeys, see Reward journey for how to set one up before adding an experiment.
Configure communications for control and variant. Both can be configured side-by-side to enable comparison between what each group receives. You can also choose to test only communication by keeping the rewards in both groups consistent, but changing the content for communications.
You can try out different reward banners to see which helps increase retention.
You may also choose to set up emails, add reward text in the upcoming order email, and experiment with different types of email content.
Once you are done configuring your journey, set the reward journey status as active, and save it to start running your reward journey experiment.
Metrics tracked in reward journey experiments
Loop tracks a set of retention and revenue metrics for each group so you can compare performance. All rate metrics are measured against total scheduled orders, which is the sum of total successful billed orders and the next upcoming order across all contracts enrolled in the experiment.
Metric | What it means | Example |
Order attempt rate | % of subscription orders scheduled for a specific order number which were attempted for charge, relative to total orders that were scheduled to be attempted for charge of that specific order number in the selected period | 500 subscriptions orders for order #3 were attempted for charge from total 1000 subscriptions orders scheduled for order #3.
Order attempt rate for order #3 is 500/1000 i.e 50%. |
Revenue realization rate | % of subscription orders amount which were successfully charged for a specific order number, relative to total order amount that were scheduled to be attempted for charge of that specific order number in the selected period. | $400 worth subscriptions orders for order #3 were successfully charged from total $1000 worth subscriptions orders scheduled for order #3.
Revenue realization rate for order #3 is $400/$1000 i.e 40%. |
Skip rate | % of specific order number subscription orders that were skipped, relative to total order count that were scheduled to be attempted for charge of that specific order number in the selected period. | 100 subscriptions orders for order #3 were skipped from total 1000 worth subscriptions orders scheduled for order #3.
Skip rate for order #3 is 100/1000 i.e 10%. |
Pause rate | % of specific order number subscription orders that were cancelled, relative to total order count that were scheduled to be attempted for charge of that specific order number in the selected period | 100 subscriptions orders for order #3 were paused from total 1000 worth subscriptions orders scheduled for order #3.
Pause rate for order #3 is 100/1000 i.e 10%. |
Cancellation rate | % of specific order number subscription orders that were cancelled, relative to total order count that were scheduled to be attempted for charge of that specific order number in the selected period. | 100 subscriptions orders for order #3 were cancelled from total 1000 worth subscriptions orders scheduled for order #3.
Cancellation rate for order #3 is 100/1000 i.e 10%. |
Same-day cancellation rate | % of subscribers who cancelled their subscription on the same day their previous order was placed, relative to total churn during the selected period. | 100 subscriptions cancelled on the same day order #2 was placed from total 1000 subscriptions who cancelled between order #2 and #3 .
0-day churn for order #2 placed is 100/1000 i.e 10% |
How reward journey experiments appear to your customers
Subscribers see either the control (A) or the variant (B) version of the reward journey, never both. The banner, rewards, and emails appear as a normal part of your store experience, with no experiment label shown to them.
Assignment is sticky. Once a subscriber is placed in a group, they continue to see that same group for the duration of the experiment, which keeps their experience consistent.
Viewing experiment performance
Navigate to Loop admin > Tools & apps > Experiments to see all your experiments in one place. Each experiment shows its status, type, and high-level metrics by group.
Click an experiment to open its details page, where you can view the primary metric and all tracked metrics for both control (A) and variant (B), along with which group is currently leading based on your winning criteria.
The Metrics section has a dropdown to switch between metrics, showing the overall value for each group and a day-on-day trend so you can see how performance develops over the experiment. The Trends section works the same way for volume measures such as total orders scheduled.
Click Export results to download the underlying data for your own analysis, or Stop experiment to end the experiment before its duration completes.
Experiment completion
An experiment completes automatically once its configured experiment duration ends, and you can stop it manually from its details page at any time.
What customers see after completion?
Once an experiment completes (auto or manual):
Control group (A) content starts getting automatically shown to all eligible subscribers for that step (benefits or offers).
Subscribers no longer see variant B in the storefront experience unless you explicitly choose to continue with B.
What do you see in the admin portal after completion?
If you choose to stop an experiment, you will be able to choose which rewards journey stays live (control or variant.
If the experiment comes to a completion automatically, for 7 days after experiment completion:
You will still see:
Control (A) configuration
Variant (B) configuration
You can select which variant you want to continue with as your long-term experience (A or B).
After those 7 days:
If you haven’t explicitly chosen a variant:
Subscribers will continue seeing control (A) content.
In the reward journey on the admin portal, you will only see the control (A) configuration.
Content from variant (B) will no longer be visible in the reward journey.
You can then either:
Continue using control (A), or
Create a new experiment to test a fresh set of hypotheses.
For the full behavior after completion, see A/B experiments.
Considerations
A/B experiments in reward journey are available on the Pro plan.
Subscriptions enroll only when they are created within the enrollment duration, not from a later order milestone.
A subscription remains in its assigned group for the full experiment duration, even if its eligibility for the journey changes across recurring orders.
Experiments are currently supported for reward journeys. They are not yet available for individual flows.
FAQs
Why should I add a "Type of subscription plan" condition to my experiment?
Adding a "Type of subscription plan" condition lets you enroll one frequency only, which keeps the experiment consistent. Without it, subscriptions on different frequencies reach the same order milestone at different times. At three months, for example, a monthly subscriber has reached order #3 while a quarterly subscriber is still on order #2, so the same milestone would be measured across subscribers at very different points in their lifecycle.
Can a customer have one subscription in control (A) and another in variant (B)?
No, assignment happens at the customer level, not the subscription level. Once a customer is assigned to a group, all of their eligible subscriptions see only the rewards designated to that group, so a single customer is never split across control and variant.
What happens if a subscription becomes ineligible during the experiment?
The subscription keeps its group assignment even if it qualifies or disqualifies for the reward journey during the experiment. Because the reward journey triggers on recurring orders, eligibility can change over time based on the subscription's criteria and state, but a subscriber assigned to A always sees A and a subscriber assigned to B always sees B for the length of the experiment.
Can a subscription join the experiment after it has already started?
Yes, but only if it is created within the enrollment duration. Subscriptions enroll when they are created during that window, and they cannot join from a later order milestone, because reward journeys are designed to show subscribers the full order-by-order reward path from order #1.
Can I run an A/B experiment on an individual flow instead of a reward journey?
No, experiments are currently supported only for reward journeys, but support for individual flows is planned. For experiments on cancellation benefits pages and offers, see A/B experiments.
Need help?
No worries - we're here for you!
If you have any questions or need assistance, feel free to email us at [email protected] or chat with us using the support beacon at the bottom right of your screen.
Regards,
Loop subscriptions team 🙂