A/B experiments in reward journey let you run two versions of a journey at the same time, so you can measure which rewards and messaging actually retain subscribers and grow their lifetime value. Subscribers are split into a control group (A) and a variant group (B), each group runs a different reward strategy, and we track retention and revenue metrics to determine a winner.
This article covers what can be tested, how to set up an experiment, the metrics tracked, how a winner is decided, and how to monitor performance. If you are new to reward journeys, see Reward journey for how to set one up before adding an experiment.
What can be tested in a reward journey experiment
A reward journey experiment lets you test two things: the reward on a milestone, and the way that reward is communicated.
For the action itself, control (A) and variant (B) can each be set to any of the actions in the Add action menu, so any two actions can be compared head to head at the same milestone. The available actions are:
Subscription based: add discount, remove discount, mystery reward.
Product based: add product one-time, add product as subscription.
Customer based: add customer tag, remove customer tag.
Common comparisons include discount vs mystery reward, mystery discount vs mystery gift, discount A vs discount B, and reward vs no reward.
For communication, you can test:
Customer portal reward banner content (A vs B, or content vs none)
Reward notification email content (A vs B, or content vs none)
Add reward text in the upcoming order email (A vs B, or content vs none)
Setting up an A/B experiment in reward journey
Navigate to Loop admin > Retain > Flows, then create or edit a reward journey.
Add an internal name for easier management, then define conditions that may be relevant for setting up the reward journey.
In the experimentation block, click Create experiment.
Enter an Internal name for the experiment.
Under Audience split and tags, set the percentage of eligible subscribers assigned to control group (A) versus variant group (B), for example 50/50 or 70/30. You can also add a customized control group tag and a variant group tag to identify subscribers in each group.
Set the enrollment duration and experiment duration.
Enrollment duration is the window during which new subscriptions join the experiment. A subscription enters the experiment when it is created within this window, and the customer is randomly assigned to control group (A) or variant group (B) at that point.
Experiment duration is the full period the experiment runs and collects data. A subscription remains in its assigned group for the entire experiment duration, even as its eligibility for the reward journey changes across recurring orders.
For example, if the experiment duration is 6 months and the enrollment duration is 1 month, only subscriptions created during that first month are enrolled in the experiment. They then remain in their assigned group for the full 6 months, while subscriptions created after the first month are not part of the experiment at all. Subscriptions that are not a part of the experiment will see the control reward journey by default.
Because the reward journey triggers on each recurring order, an enrolled subscription may qualify or disqualify for the journey over time based on its eligibility criteria and state. However, its group assignment does not change, so a subscriber assigned to A always sees A, and a subscriber assigned to B always sees B, for the length of the experiment.Note that subscriptions enroll only when they are created, not from a later order milestone.
Under Metrics to be tracked, select the metrics to measure for each group, and choose one as the Primary metric. The primary metric is used to decide the winner. All the metrics are defined in the next section.
Under Winner calculation, set the minimum percentage difference required in the primary metric to declare a winner. Loop compares the primary metric between control (A) and variant (B) as a relative difference, and declares a winner only when that difference meets or exceeds the minimum set.
For instance, if the primary metric selected here is "Order attempt rate", control (A) is at 50%, and variant (B) is at 55%, then B is performing 10% better on a relative basis (5 ÷ 50), which meets a 10% minimum. If B were at 48%, the relative difference would still be only 4% (2 ÷ 50), and no winner would be declared against a 10% minimum.
Configure rewards for the control and variant by adding one or more actions for each order configured in the flow. If you are new to reward journeys, see Reward journey for how to set one up before adding an experiment.
Configure communications for control and variant. Both can be configured side-by-side to enable comparison between what each group receives. You can also choose to test only communication by keeping the rewards in both groups consistent, but changing the content for communications.
You can try out different reward banners to see which helps increase retention.
You may also choose to set up emails, add reward text in the upcoming order email, and experiment with different types of email content.
Once you are done configuring your journey, set the reward journey status as active, and save it to start running your reward journey experiment.
Metrics tracked in reward journey experiments
Loop tracks a set of retention and revenue metrics for each group so you can compare performance. All rate metrics are measured against total scheduled orders, which is the sum of total successful billed orders and the next upcoming order across all contracts enrolled in the experiment.
Metric | What it means | Example |
Order attempt rate | % of subscription orders scheduled for a specific order number which were attempted for charge, relative to total orders that were scheduled to be attempted for charge of that specific order number in the selected period | 500 subscriptions orders for order #3 were attempted for charge from total 1000 subscriptions orders scheduled for order #3.
Order attempt rate for order #3 is 500/1000 i.e 50%. |
Revenue realization rate | % of subscription orders amount which were successfully charged for a specific order number, relative to total order amount that were scheduled to be attempted for charge of that specific order number in the selected period. | $400 worth subscriptions orders for order #3 were successfully charged from total $1000 worth subscriptions orders scheduled for order #3.
Revenue realization rate for order #3 is $400/$1000 i.e 40%. |
Skip rate | % of specific order number subscription orders that were skipped, relative to total order count that were scheduled to be attempted for charge of that specific order number in the selected period. | 100 subscriptions orders for order #3 were skipped from total 1000 worth subscriptions orders scheduled for order #3.
Skip rate for order #3 is 100/1000 i.e 10%. |
Pause rate | % of specific order number subscription orders that were cancelled, relative to total order count that were scheduled to be attempted for charge of that specific order number in the selected period | 100 subscriptions orders for order #3 were paused from total 1000 worth subscriptions orders scheduled for order #3.
Pause rate for order #3 is 100/1000 i.e 10%. |
Cancellation rate | % of specific order number subscription orders that were cancelled, relative to total order count that were scheduled to be attempted for charge of that specific order number in the selected period. | 100 subscriptions orders for order #3 were cancelled from total 1000 worth subscriptions orders scheduled for order #3.
Cancellation rate for order #3 is 100/1000 i.e 10%. |
Same-day cancellation rate | % of subscribers who cancelled their subscription on the same day their previous order was placed, relative to total churn during the selected period. | 100 subscriptions cancelled on the same day order #2 was placed from total 1000 subscriptions who cancelled between order #2 and #3 .
0-day churn for order #2 placed is 100/1000 i.e 10% |
How reward journey experiments appear to your customers
Subscribers see either the control (A) or the variant (B) version of the reward journey, never both. The banner, rewards, and emails appear as a normal part of your store experience, with no experiment label shown to them.
Assignment is sticky. Once a subscriber is placed in a group, they continue to see that same group for the duration of the experiment, which keeps their experience consistent.
Experiment completion
An experiment completes automatically once its configured experiment duration ends, and you can stop it manually from its details page at any time. If you choose to stop an experiment manually, you can select which reward journey stays live, control or variant.
For what happens to subscriber-facing content and your admin view once an experiment completes automatically, including the 7-day window to choose a variant, see A/B experiments.
Considerations
A/B experiments in reward journey are available on the Pro plan.
Subscriptions enroll only when they are created within the enrollment duration, not from a later order milestone.
A subscription remains in its assigned group for the full experiment duration, even if its eligibility for the journey changes across recurring orders.
Experiments are currently supported for reward journeys. They are not yet available for individual flows.
FAQs
Why should I add a "Type of subscription plan" condition to my experiment?
Adding a "Type of subscription plan" condition lets you enroll one frequency only, which keeps the experiment consistent. Without it, subscriptions on different frequencies reach the same order milestone at different times. At three months, for example, a monthly subscriber has reached order #3 while a quarterly subscriber is still on order #2, so the same milestone would be measured across subscribers at very different points in their lifecycle.
Can a customer have one subscription in control (A) and another in variant (B)?
No, assignment happens at the customer level, not the subscription level. Once a customer is assigned to a group, all of their eligible subscriptions see only the rewards designated to that group, so a single customer is never split across control and variant.
What happens if a subscription becomes ineligible during the experiment?
The subscription keeps its group assignment even if it qualifies or disqualifies for the reward journey during the experiment. Because the reward journey triggers on recurring orders, eligibility can change over time based on the subscription's criteria and state, but a subscriber assigned to A always sees A and a subscriber assigned to B always sees B for the length of the experiment.
Can a subscription join the experiment after it has already started?
Yes, but only if it is created within the enrollment duration. Subscriptions enroll when they are created during that window, and they cannot join from a later order milestone, because reward journeys are designed to show subscribers the full order-by-order reward path from order #1.
Can I run an A/B experiment on an individual flow instead of a reward journey?
No, experiments are currently supported only for reward journeys, but support for individual flows is planned.
Need help?
No worries - we're here for you!
If you have any questions or need assistance, feel free to email us at [email protected] or chat with us using the support beacon at the bottom right of your screen.
Regards,
Loop subscriptions team 🙂