Do AI-generated email subject lines perform better?

Sometimes, on the outcome actually measured. A field test found more item clicks from AI-generated titles than from a fixed template, but no significant increase in buyers. Another found higher click-through when a model trained on past tests improved human-written lines. Neither establishes that any AI-generated subject line beats a human one for your store. Test matched versions of your email and decide in advance whether clicks or orders matter.

An AI generator can add another subject-line candidate to a test. That tells you it can produce a candidate, not whether recipients respond better to it. To answer the performance question, you need to know what the AI line was compared with, which outcome changed, and what remained the same in the email.

This is a narrower question than whether AI-written email copy improves retention. A subject-line test can isolate the line on one send; it does not establish that the copy-production method raises repeat purchases.

What has an AI subject line actually beaten?

In a one-week randomized Mercari test reported in 2025, covering more than a million users who had accessed the app within a year, bought within six months, but not accessed it in seven days, AI titles beat a fixed template on item taps but did not establish a buyer-conversion gain. In a separate 36-campaign test presented in December 2024 by researchers including a Persado co-author, a model trained on past A/B results improved human-written lines on click-through; a general-purpose model did not. Neither establishes an order lift for your store.

Mercari did not compare AI with a skilled human subject-line writer. Its control was a fixed template. The recommended items in the email body came from the same algorithm in both arms, which makes the title change more interpretable, but the tested audience, language and one-week window limit what another store can carry over. The researchers also screened generated titles for sensitive wording and brand safety before sending them. The result is evidence for a reviewed system against that template, not for sending unchecked output.

The other test is closer to AI versus human writing, but its winning method was specialized. The researchers, two at Chicago Booth and one at Persado, trained a model on comparisons from 20,000 earlier campaigns over ten years and had it revise a human candidate. The 2024 presentation reports higher mean click-through across 36 new campaigns. A general-purpose model asked to improve the same human starting point did not show that gain. This is evidence that a particular trained workflow can improve clicks in that setting. It is not an estimate of what a general subject-line generator will do to your orders or revenue.

What does “better” mean for this campaign?

Choose the outcome before seeing the variants’ results. For a product visit, compare clicks per eligible or delivered recipient. For a purchase decision, compare buyers or orders per assigned recipient over the same stated window. Keep exclusions and denominators aligned, and inspect send and delivery counts. Treat opens as diagnostic: Apple says Protect Mail Activity can download remote email content on receipt regardless of engagement, so a tracked open need not mean a read.

Our guide to whether you can trust email open rates covers the other reporting checks before you compare campaigns.

The Mercari test shows why the distinction matters: its significant item-tap gain did not come with a significant overall buyer-conversion gain among targeted users. “No significant gain” does not prove equal buyer performance; it says that test did not establish a difference on that outcome. Likewise, the trained-model study’s click result does not establish a sales result. Record unsubscribes and complaints alongside the main outcome so an engaging line is not accepted without inspecting the response it drew.

How do you run an AI-versus-human subject-line test?

Start with one campaign you intended to send anyway. Write a human line and generate an AI alternative from the same product, offer and email body. Verify both lines against those facts. Randomly assign comparable eligible recipients, change the subject only, and declare a click or order outcome and its observation window before sending. Keep the variant assignments and delivery counts for the final comparison.

Keep the sender, preview text, body, offer, audience rules and send timing aligned across the two variants, or a better result cannot be credited specifically to the line. If personalization or different products are part of the experiment, label it as a broader treatment rather than an origin-only subject-line test.

In Klaviyo’s current campaign setup, the path is Campaigns → Create campaign → Single channel → Email. Enter the original subject. For an AI-generated B, choose an option under Generate a variation to A/B test before clicking Create A/B test; Klaviyo keeps the original in A and generates B from that option. For a manual B, click Create A/B test above the subject field. Its guide says to change only one factor in a content test. Check the actual generated variation before sending, including any preview text that might otherwise differ.

Set the test size with the decision in mind. In Klaviyo’s default winning-variation strategy, a test using less than the whole audience selects a winner after its test period and sends that version to the remainder. If you want both variants randomized across the whole campaign audience, set the content test to 100%; there is then no later winner send, and the reported winner can change as results arrive. It offers open rate and click rate as winning metrics, and placed-order rate where the account has a Placed Order metric and the test does not use its Personalized variations for each recipient strategy. Its guide says open-rate tests include Apple privacy (MPP) opens, so an early winner can reflect automatic opens. If you need an order outcome beyond the platform’s immediate winning rule, preserve variant assignment and evaluate both groups over your declared order window rather than treating the early winner label as the commercial answer.

Use a small decision record for the send:

Keep generation and review effort beside the recipient result. A line that wins clicks but needs extensive correction may still be a poor way to produce the next campaign. Neither field study provides a universal time-saving figure for your team, so record your own work rather than assuming the faster draft is the cheaper approved send.

Should you keep using AI for subject lines?

Use it as a candidate source when the line survives factual review and the matched test supports the outcome you chose. If the test only changes opens, say so. If clicks rise but the order result is unresolved, keep the click finding separate from a purchase claim. If the result is too imprecise to decide, retain the test record and run another comparable send instead of declaring a permanent winner.

The available field evidence supports a conditional answer: an AI method can improve email engagement in a defined comparison, and a general-purpose model does not automatically do so. Your useful result is the comparison you can reproduce for your audience, campaign and outcome—not the label attached to the writer.