A/B experiments in flows let you run two versions of the same flow at the same time, so you can measure which actions and notifications actually keep subscribers billing. Subscribers who trigger the flow are split into a control group (A) and a variant group (B), each group receives a different set of actions, and we track order and retention metrics to declare a winner.
This article covers what you can test, popular use cases, how to set up an experiment, the metrics tracked, and how to review results. If you are new to flows, read Flows first to understand triggers, conditions, and actions.
What can be tested in a flow experiment
A flow experiment tests two things: the actions the flow applies, and the way you tell subscribers about them. Triggers and conditions stay the same for both groups, so only the actions and notifications differ.
For actions, control (A) and variant (B) can each use any action from the Add action menu, so you can compare any two actions head to head. The available actions are:
Subscription based: add discount, remove discount, mystery reward.
Product based: add product one-time, add product as subscription.
Customer based: add customer tag, remove customer tag.
For product actions, you can also test pricing. A product can be added at its base price, at the discount configured on its selling plan, or at a specific discount value, such as 100% off for a free gift.
For reward notifications, you can test each of these per group:
Send email to customers from Loop
Publish event to 3rd party apps like Klaviyo
Add reward variables to upcoming order emails (Loop & 3rd party apps)
Display reward banner on customer portal
To test communication only, keep the same action in both groups and change only the notifications.
Popular use cases
Each use case below names the flow setup, what each group gets, and the primary metric that best answers the question.
Discount vs free gift on a milestone order
A coffee brand rewards subscribers when their 3rd recurring order is placed. Control gives $5 off the next order, and variant adds a free bag of single-origin beans at 100% off using Add product(s) one-time.
Gifts often feel more valuable than their cost to you, so if the gift matches or beats the discount, you protect margin on the order total. Primary metric: Order placement rate.
Discount depth
Control gives 10% off the next order and variant gives 20%. Many brands over-discount by default, so if 10% keeps subscribers about as well as 20%, you keep the extra margin.
Primary metric: Order placement rate, with Revenue realization rate to confirm the deeper discount pays for itself.
Plain discount vs mystery reward
Control gives a fixed 15% discount, and variant wraps the same value in a mystery reward that subscribers reveal. This tests whether surprise drives more engagement than a known offer.
Primary metric: Order placement rate or Skip rate.
Welcome reward for new subscribers
Use the A new subscription is created trigger to reward subscribers before their first recurring order, when churn risk is highest. Control adds a one-time protein powder sample, and variant gives 10% off the first recurring order.
Primary metric: Order attempt rate or Same day cancel rate.
Sample vs subscription add-on
Control adds a new flavor once with Add product(s) one-time, and variant adds it with Add product as subscription at a discount. This shows whether subscribers keep a recurring add-on or remove it.
Primary metric: Revenue realization rate.
Reward notifications
Keep the same reward in both groups. Control sends no notification, and variant turns on the reward email and customer portal banner.
This tells you whether subscribers need to know about a reward for it to change their behavior. Primary metric: Order placement rate.
Reward vs no reward
Control applies only a customer tag for tracking, and variant applies a discount. This measures whether the incentive changes behavior at all, or whether subscribers would have ordered anyway.
Primary metric: Order placement rate.
Setting up an A/B experiment in a flow
Navigate to Loop admin > Flows, then create a new flow or open an existing one.
Add a name, select the triggers, and add any conditions. These apply to both groups, so set them before creating the experimen.
In the Experimentation block, click + Create an experiment.
Enter an Internal name for the experiment.
Under Audience split and tags, set the percentage of subscribers assigned to control group (A) and variant group (B), for example 50/50 or 70/30. You can also add a control group tag and a variant group tag to identify subscribers in each group.
Set the Enrollment duration and Experiment duration in days.
Enrollment duration is the window during which subscribers who trigger the flow join the experiment and are assigned to a group. Each subscription enrolls only once, and the order that triggers enrollment is not counted in results.
Experiment duration is the full period the experiment runs and collects data. Enrolled subscribers stay in their group for the entire experiment duration.
Under Metrics to be tracked, select the metrics to measure, then choose one as the Primary metric. The primary metric decides the winner.
Under Winner calculation, set the minimum percentage difference in the primary metric required to declare a winner. The default is 10%.
Click Save. The flow now shows side-by-side Control group (A) and Variant group (B) sections.
Add one or more actions for each group, then configure reward notifications for each group.
Set the flow status to Active, then click Save to start the experiment.
The winner is decided on relative difference. For example, if your primary metric is order placement rate, control (A) is at 50%, and variant (B) is at 55%, then B performs 10% better (5 ÷ 50), which meets a 10% minimum. If B were at 52%, the difference would be 4% (2 ÷ 50), and no winner would be declared.
Metrics tracked in flow experiments
Every metric counts only what happens after a subscription enrolls. Each subscription enrolls once, and the order that triggered enrollment is not counted.
Terms used in these metrics
Enrolled subscriptions: subscriptions that triggered the flow during the enrollment duration and were assigned to a group.
Order number: order numbers count from enrollment, so the first order after enrollment is order 1. The upcoming order's number is the number of successful orders since enrollment plus 1, whatever its status.
Total scheduled orders: successful orders placed after enrollment, plus one upcoming order per enrolled subscription. For example, 100 enrolled subscriptions that have placed 200 orders since enrolling have 300 scheduled orders.
Skipped orders: an upcoming order that is skipped keeps its order number. Skipped orders with a billing date in the past are not counted.
Zero day cancellation: a cancellation made on the same day an order was placed or the subscription enrolled, or the day after, in your store's time zone.
Aggregate metrics (all orders)
When the order filter is set to All orders, rates are measured against total scheduled orders. The examples use 100 enrolled subscriptions with 300 scheduled orders.
Metric | How it is calculated | Example |
Order placement rate | Orders placed after enrollment ÷ total scheduled orders. | 200 orders placed ÷ 300 = 66.67% |
Order attempt rate | Charge attempts after enrollment ÷ (charge attempts + enrolled subscriptions). | 240 attempts ÷ (240 + 100) = 70.59% |
Revenue realization rate | Measured the same way as order placement rate, using order value instead of order count. | $8,000 charged ÷ $12,000 scheduled = 66.67% |
Skip rate | Upcoming orders skipped ÷ total scheduled orders. | 15 skipped ÷ 300 = 5% |
Pause rate | Paused subscriptions ÷ total scheduled orders. | 9 paused ÷ 300 = 3% |
Cancellation rate | Cancelled subscriptions ÷ total scheduled orders. | 12 cancelled ÷ 300 = 4% |
Same day cancel rate | Cancelled subscriptions whose last cancellation was a zero day cancellation ÷ all currently cancelled subscriptions. | 3 zero day ÷ 12 cancelled = 25% |
Upcoming order cancellation | % of cancellations made before the upcoming order was attempted. | 5 ÷ 12 cancelled = 41.67% |
Order-wise metrics (single order)
When you pick an order number in the filter, rates are measured against enrolled subscriptions instead. This shows how each group performs at a specific point after enrollment, such as the 3rd order.
Metric | How it is calculated | Example (order 2) |
Order placement rate | Orders placed at that order number ÷ enrolled subscriptions. | 70 placed ÷ 100 = 70% |
Order attempt rate | Orders at that order number with at least one charge attempt ÷ enrolled subscriptions. | 78 attempted ÷ 100 = 78% |
Revenue realization rate | Measured the same way as order placement rate, using order value instead of order count. | $2,800 ÷ $4,000 = 70% |
Skip rate | Orders skipped at that order number ÷ enrolled subscriptions. | 6 skipped ÷ 100 = 6% |
Cancellation rate | Subscriptions cancelled before that order number ÷ enrolled subscriptions. This is cumulative. | 8 cancelled ÷ 100 = 8% |
Pause rate | Subscriptions paused before that order number ÷ enrolled subscriptions. This is cumulative. | 5 paused ÷ 100 = 5% |
Same day cancel rate | Of the subscriptions counted in cancellation rate for that order, the share whose last cancellation was a zero day cancellation. | 2 ÷ 8 cancelled = 25% |
The results page also shows Orders scheduled, Subscriptions enrolled, and Winning duration, which is the share of days each group has led on the primary metric.
Reviewing experiment results
You can check a flow experiment from two places. The Experimentation block on the flow shows a quick summary: status, the primary metric for each group, the start date, and which group is winning.
For the full results, click Show results on the flow, or navigate to Loop admin > Tools & apps > Experiments, then open the experiment. The Experiment module line shows Flow with a link back to the flow it belongs to.
The results page includes:
Overview: every tracked metric for control (A) and variant (B), with the winning group marked.
Order filter: view results for all orders or a single order number, such as the 3rd order after enrollment, to see where in the lifecycle a difference appears.
Date range: the experiment start date through today, or a range you choose.
Metrics: a bar chart of the overall primary metric and a day-by-day trend.
Trends: totals per group for orders placed, attempted, scheduled, skips, pauses, cancellations, upcoming order cancellations, and revenue.
Click Export results to download raw data, including subscription IDs, the group each subscriber was in, and their outcome. For how the experiments list and details page work across all modules, see A/B experiments.
How flow experiments appear to your customers
Subscribers receive either the control (A) or the variant (B) actions, never both. Rewards, emails, and banners appear as a normal part of your store experience, with no experiment label.
Assignment is sticky. Once a subscriber is placed in a group, every later trigger of the flow applies that same group's actions for the length of the experiment.
Experiment completion
An experiment completes automatically when its experiment duration ends, and you can stop it any time by clicking Stop experiment on the results page. When it completes, control (A) stays live for everyone who triggers the flow.
For 7 days after completion, the flow still shows both control (A) and variant (B), so you can switch to variant (B) manually. After that, only control (A) remains in the admin. For the full behavior, see A/B experiments.
Considerations
While an experiment is running, you can only rename the flow. Stop the experiment to change its triggers, conditions, priority, actions, or rewards.
Triggers and conditions are shared by both groups. Only actions and reward notifications can differ between control (A) and variant (B).
Enrollment and experiment duration are set in days.
Subscriptions on different billing frequencies reach the same order number at different times. Add a Type of subscription plan condition to compare like with like.
Small samples swing easily. A difference across a handful of scheduled orders is not a reliable result, so let the experiment collect enough orders before you pick a variant.
FAQs
Can I edit actions while the experiment is running?
No, a running experiment locks the flow's triggers, conditions, priority, actions, and rewards so results stay comparable, but you can stop the experiment, make your changes, and start a new one.
Can I test notifications without changing the reward?
Yes, keep the same action in both groups and change only the reward notifications, such as turning on the customer portal banner for variant (B).
Can I use different triggers or conditions for each group?
No, triggers and conditions apply to the whole flow, but you can create a separate flow with its own triggers and run an experiment there.
What happens to subscribers when the experiment ends?
Control (A) stays live for everyone who triggers the flow. You have 7 days to switch to variant (B) manually before only control (A) remains in the admin.
Can I see results for a specific order number?
Yes, use the order filter at the top of the results page to view results for a single order, such as the 3rd order after enrollment.
Need help?
No worries - we're here for you!
If you have any questions or need assistance, feel free to email us at [email protected] or chat with us using the support beacon at the bottom right of your screen.
Regards,
Loop subscriptions team 🙂










