Does AI-written email copy improve retention?

We do not yet have a general answer that AI-written email copy improves customer retention. A randomized test at one retailer found no statistically significant difference in gross profit from orders between AI-written and human-written newsletter content over its short test periods. Another large experiment found more clicks from AI-generated email titles but no significant increase in buyer conversion. Neither result establishes that customers keep buying for longer because the copy was written with AI.

That does not make AI writing useless. It means the decision has two parts: whether the new writing process produces acceptable emails at a worthwhile cost, and whether customers respond better over a period long enough to measure repeat purchasing. A faster draft settles only the first part.

AI email copy and retention, in short. AI can be a viable way to produce email copy, and a particular AI-written variant may win an email test. To say it improves retention, compare it with the existing copy process using the same audience, offer and send schedule, then measure repeat purchases over a window chosen before the test. Opens, clicks and production savings answer different questions.

What have direct tests of AI-written email found?

Dubé and Xu’s study of Wine Access found no statistically significant difference in gross profit from orders between human-written, AI-written and human-edited AI newsletter content across three randomized field experiments at one online wine retailer. The teams received the same wines and professional tasting notes. The first experiment also included a group that did not receive the tested newsletter.

That result was assessed at the researchers’ preregistered levels. It does not prove the methods are equivalent in every setting; the study did not establish a gross-profit difference between them. In the first experiment, all three groups receiving the newsletter produced nearly double the gross profit from orders of the no-newsletter group during the test. That is evidence about the tested newsletter versus no tested newsletter; it is not evidence that AI beat human copy.

The researchers also modeled annual net profit after writing labour and software costs. Their decision model selected an AI policy over the human-only writing policy for this retailer in each experiment. That is a meaningful production and profit result for the case studied. The authors explicitly describe the experiments as short term and say those periods cannot verify longer-term effects.

Why can a better email metric fail to become retention lift?

In a one-week Mercari experiment by Jobson and co-authors, AI-generated email titles produced a statistically significant 24% relative lift in item click-through, but no significant lift in overall buyer conversion among targeted users. The test randomized more than a million users who had recently stopped using the app to fixed-template or generated titles. The recommended items in the email body came from the same algorithm in both groups.

A title that gets more people to click has cleared one step of the journey. It has not, by that result alone, made them buy again.

Open rate is an even weaker basis for a retention claim. Litmus explains that Apple Mail prefetching can register false opens and inflate overall open counts. Opens can help diagnose a message, with that limitation; they cannot tell you whether copy changed repeat purchasing.

What should a retention team test?

First specify the decision. If the question is “Can we produce this email more efficiently?”, measure preparation time, editing time, approval work and cost alongside basic quality checks. If the question is “Will this copy retain more customers?”, make a customer outcome the primary measure before seeing the results.

One workable design is to assign eligible customers randomly to two versions of the same campaign or flow: copy prepared by the current human process and copy prepared with AI under a defined editing process. Keep the offer, product facts, audience eligibility and send schedule aligned. Keep each customer in the same assigned group across the tested sequence, so later emails do not blur which process they received. Choose a follow-up window that gives those customers a realistic chance to buy again, and define what counts as a repeat purchase before the first send.

This last distinction matters. The Wine Access no-newsletter group answered a different question from its human-versus-AI groups. Treating the newsletter’s effect as the AI writer’s effect would give the wrong answer.

Small tests may leave the order difference unresolved. In research using customer holdouts across email and catalog campaigns, Zantedeschi and co-authors found that estimates from individual campaigns could be imprecise when holdouts were small. Decide in advance what size of difference would matter and whether the available audience can distinguish it. If the result is too uncertain, say so; a higher click rate does not fill that gap.

What would count as a useful result?

If AI-assisted copy maintains the customer outcomes that matter while reducing production work, that may justify using it even without a retention lift. That is the kind of decision the Wine Access researchers examined for their retailer. If repeat purchases improve in a sound comparison, the team has evidence for that specific audience, campaign type and follow-up window; it still needs to check whether the effect holds elsewhere.

If only opens or clicks rise, call it an engagement result. If the purchase difference is too uncertain, do not call it a retention win or a loss. Keep the copy-quality checks in place either way: a draft can be quick to produce and still need corrections before it is fit to send.

AI writing changes how the message is made. Retention changes when customers return. Measure both, but do not use one as a substitute for the other.