Does AI-written email copy improve retention?

We do not yet have a general answer that AI-written email copy improves customer retention. Three randomized experiments at one retailer found no statistically significant difference in gross profit from orders between AI-written and human-written newsletter content over their short test periods. Another large experiment found more clicks from AI-generated email titles but no significant increase in buyer conversion. Neither result establishes that customers keep buying for longer because the copy was written with AI.

That does not make AI writing useless. It means the decision has two parts: whether the new writing process produces acceptable emails at a worthwhile cost, and whether customers respond better over a period long enough to measure repeat purchasing. A faster draft settles only the first part.

AI email copy and retention, in short. AI can be a viable way to produce email copy, and a particular AI-written variant may win an email test. To say it improves retention, compare it with the existing copy process using the same audience, offer and send schedule, then measure repeat purchases over a window chosen before the test. Opens, clicks and production savings answer different questions.

What have direct tests of AI-written email found?

Dubé and Xu’s study of Wine Access found no statistically significant difference in gross profit from orders between human-written, AI-written and human-edited AI newsletter content across three short, roughly two-week randomized field experiments at one online wine retailer. The teams received the same wines and professional tasting notes. The first experiment also included a group that did not receive the tested newsletter.

That result does not prove the methods are equivalent in every setting. In the first experiment, all three groups receiving the newsletter produced nearly double the gross profit from orders of the no-newsletter group during the test. That is evidence about the tested newsletter versus no tested newsletter; it is not evidence that AI beat human copy.

The researchers also modeled annual net profit after writing labour and software costs. Their decision model selected an AI policy over the human-only writing policy for this retailer in each experiment. That is a meaningful production and profit result for the case studied. The authors explicitly describe the experiments as short term and say those periods cannot verify longer-term effects.

Why can a better email metric fail to become retention lift?

In a one-week Mercari experiment reported in a preprint by Jobson and co-authors, AI-generated email titles produced a statistically significant 24% relative lift in item click-through, but no significant lift in overall buyer conversion among targeted users. The test randomized more than a million users who had recently stopped using the app to fixed-template or generated titles. The recommended items in the email body came from the same algorithm in both groups.

A title that gets more people to click has cleared one step of the journey. It has not, by that result alone, made them buy again.

Open rate is an even weaker basis for a retention claim. Litmus explains that Apple Mail prefetching can register false opens and inflate overall open counts. Opens can help diagnose a message, with that limitation; they cannot tell you whether copy changed repeat purchasing.

What should a retention team test?

First specify the decision. If the question is “Can we produce this email more efficiently?”, measure preparation time, editing time, approval work and cost alongside basic quality checks. If the question is “Will this copy retain more customers?”, make a customer outcome the primary measure before seeing the results.

Choose one job first. A win-back email and a post-purchase flow reach customers at different moments; combining them would make a copy result hard to interpret. Keep the offer, product facts, audience eligibility and send schedule aligned between a human-written version and an AI-assisted version. If testing a sequence, assign each customer once and keep that assignment through every email.

For a single Klaviyo campaign, its A/B instructions say to open Campaigns → Create campaign → Email → Create A/B test, then choose Test content. Name the human and AI-assisted versions, change the copy only, and set the test size to 100% if everyone should remain in the two test arms. At a smaller test size, Klaviyo by default selects a winner after the test period and sends that version to the remaining recipients. Its results can compare opens, clicks and, where the account has a Placed Order metric, placed-order rate. That comparison covers the campaign’s own test window; a longer repeat-purchase test across later sends needs persistent customer assignments and later order data.

For example, suppose a store with a roughly 60-day buying cycle assigns 8,000 eligible customers evenly and chooses repeat purchase within 90 days before the send. If 800 of 4,000 human-copy customers and 920 of 4,000 AI-copy customers buy again, the observed difference is 120 repeat buyers, or three percentage points. These are illustrative numbers, not a result or a sample-size recommendation. Before running the test, check whether the audience can detect the smallest difference worth acting on. To assess the 90-day outcome, preserve each customer’s variant assignment and join it to later orders; Shopify’s default customer cohort report groups by first-order date, not by email variant.

This last distinction matters. The Wine Access no-newsletter group answered a different question from its human-versus-AI groups. Treating the newsletter’s effect as the AI writer’s effect would give the wrong answer.

Small tests may leave the order difference unresolved. In research using customer holdouts across email and catalog campaigns, Zantedeschi and co-authors found that estimates from individual campaigns could be imprecise when holdouts were small. Decide in advance what size of difference would matter and whether the available audience can distinguish it. If the result is too uncertain, say so; a higher click rate does not fill that gap.

What would count as a useful result?

If AI-assisted copy maintains the customer outcomes that matter while reducing production work, that may justify using it even without a retention lift. That is the kind of decision the Wine Access researchers examined for their retailer. If repeat purchases improve in a sound comparison, the team has evidence for that specific audience, campaign type and follow-up window; it still needs to check whether the effect holds elsewhere.

If only opens or clicks rise, call it an engagement result. If the purchase difference is too uncertain, do not call it a retention win or a loss. Keep the copy-quality checks in place either way: a draft can be quick to produce and still need corrections before it is fit to send.

The useful decision is whether the copy method saves production work, changes customer purchases, or both. Report each result under the question it actually answers.