A/B experiments in cancellation flows let you test two versions of a step, like a benefits page or a retention offer, to see which one saves more subscribers. You split your audience between a control (A) and a variant (B), track key experiment metrics, and define clear winning criteria so you can confidently roll out the highest-performing content instead of guessing. For what A/B experiments are generally and how to view and export results, see A/B Experiments.
Why use A/B experiments for cancellation flows
A/B experiments help you answer "what actually works?" instead of guessing.
Popular use cases for cancellation benefits pages:
Compare different content formats, such as video against text, or GIF against static image.
Test different founder or brand videos, with different intros, hooks, or tone.
Compare a long-form story against short, punchy copy.
Popular use cases for retention offers:
Gift vs discount: should you offer a gift to protect margins, or does a discount save better?
Discount A vs discount B: do you need a higher discount, or can a smaller one achieve a similar or better save rate?
Gift A vs gift B: which gift actually saves more subscribers?
Running experiments instead of one-off changes lets you prove which approach increases save rate, protect your margins by validating whether richer offers are truly worth it, and continuously optimize your cancellation flows without guesswork.
How A/B experiments work in cancellation flows
When you create an experiment, for either benefits or offers, you configure:
Audience split: decide what percentage of eligible subscribers see group A versus group B, for example 50/50 or 70/30. We randomly assign each subscriber to a group when they enter the cancellation flow, and once assigned, they always see the same variant for that step until the experiment completes or you stop it.
Customer tags: we add different tags to subscribers in the control and variant groups.
Experiment completion rules: choose when the experiment should auto-complete, either by number of days (for example, run for 14 days) or number of attempts (for example, run for 2,000 attempts).
Winning criteria: define how much better one variant's save rate must be for it to be declared the winner. This is a minimum delta between A and B.
Save rate is the primary metric for cancellation flow experiments: the number of cancellation attempts that were "saved," divided by the total attempts that saw that step.
For example, with a winning criteria of "20% better save rate": if control (A) has a 10% save rate and variant (B) has a 12% save rate, B is saving 20% more than A, so B is declared the winner. If variant (B) only reaches 11%, that's a 10% improvement, which falls short of the 20% threshold, so no winner is declared.
Setting up an A/B experiment for cancellation benefits
Use this when you want to test different versions of your benefits page before the exit survey and offers. An experiment for cancellation benefits covers the drawer or page title, body content (text, images, videos, GIFs), and call-to-action buttons, and you configure control (A) and variant (B) side by side.
Navigate to Loop admin > Retain > Cancellation flows > Benefits page.
Create or edit the benefits page where you want to run the experiment.
In the Experimentation block, click Create experiment.
Configure your experiment: experiment name (for example, "Benefits video vs text"), audience split, audience tags, experiment completion (days or cancellation attempts), and winning criteria.
Configure content for both groups side by side: control (A) title, body, and CTAs, and variant (B) alternate title, content, and CTAs.
Use the side-by-side preview to compare how A and B will look to subscribers.
Click Save on the benefits page to start the experiment.
Once live, subscribers who enter the flow and are eligible for this benefits page are split between A and B according to your audience split.
Setting up an A/B experiment for retention offers
Use this when you want to test what kind of offer saves best after the exit survey. An experiment for retention offers covers the offer drawer title, body copy, the type and value of the offer (for example, gift vs discount, 10% off vs 20% off, or gift A vs gift B), and CTAs.
Navigate to Loop admin > Retain > Cancellation flows > Retention offers.
Create or edit the retention offer set where you want to run the experiment.
In the Experimentation block, click Create experiment.
Configure your experiment: experiment name (for example, "Gift vs 15% discount"), audience split, audience tags, experiment completion (days or cancellation attempts), and winning criteria.
Configure control (A) and variant (B) offers side by side: control (A) is your current or baseline offer (for example, 10% discount), and variant (B) is your alternative offer (for example, a free gift, or 15% discount).
Review the side-by-side preview for both offer variants.
Click Save on the offers configuration to start the experiment.
Subscribers who reach the offers step in the cancellation flow are split between control and variant groups and see the offer configured for their group.
How A/B experiments appear to customers in cancellation flows
Subscribers go through the standard cancellation flow (benefits, reasons, offers). Wherever an experiment is configured, they see either version A or version B, never both, and the page or drawer looks like a normal benefits page or offer with no experiment label shown to them. If the same subscriber attempts cancellation again while the experiment is active, they see the same group they were previously assigned to, to keep the experience consistent.
If you run experiments on both the benefits page and retention offers, a subscriber may be in group A or B for the benefits experiment, and independently in group A or B for the offers experiment.
When a cancellation flow experiment completes
An experiment completes automatically once it has run for the configured number of days, or reached the configured number of cancellation attempts, whichever comes first. You can also manually stop an experiment earlier, for example if you already see a clear winner or performance looks poor.
For what happens to subscriber-facing content and your admin view after an experiment completes, including the 7-day window to choose a variant, see A/B experiments.
FAQs
Can I run A/B experiments on both benefits and offers at the same time?
Yes, you can run an experiment on the benefits page and a separate experiment on retention offers at the same time. A subscriber may participate in both, but assignment is independent for each step, so they might see Benefits A and Offer B, depending on random assignment.
What happens if neither variant meets the winning criteria?
If the difference in save rate between A and B is less than your winning criteria, for example less than a 20% improvement, no winner is automatically declared. The experiment still completes once it hits the configured days or attempts, and control (A) continues showing to all subscribers unless you explicitly choose to switch to variant B within the 7-day post-completion window.
Need help?
No worries - we're here for you!
If you have any questions or need assistance, feel free to email us at [email protected] or chat with us using the support beacon at the bottom right of your screen.
Regards,
Loop subscriptions team 🙂