AI agents can improve parts of the customer experience, but the available studies here do not establish a general improvement in customer retention. One agentic marketing trial found more days of app engagement than the existing messaging process. A separate service trial found shorter chats but lower ratings for the chats eligible for AI handling. Neither result tells us how many additional customers later bought again or renewed because of the agent.
AI agents and retention, in short. Judge an agent by the customer outcome it is meant to change. If the goal is repeat purchasing or renewal, compare the agent-enabled process with the existing process using a randomized customer holdout and a follow-up window chosen in advance. Track service failures and handoffs alongside that outcome. Faster chats, more clicks and more automated decisions answer narrower questions.
What kind of AI agent are we talking about?
An agent takes a sequence of decisions or carries out a task, rather than supplying one prediction. In marketing, it might choose a message, channel and send time from approved options; the next-best-action guide explains that decision. In service, it might interpret an issue, ask follow-up questions and carry out a standard resolution, with a person available to take over.
Those are different interventions. The agentic marketing study examined a system that assembled messages from components supplied by marketers and chose timing and channel based on customer context. The Alibaba service study examined an autonomous system that handled a limited set of standardized chats under human supervision. Neither is a test of every product sold as an “AI agent,” and the marketing study does not show that a language model wrote every message.
What does the evidence show?
The available trials show changes in app engagement and service speed. They do not establish that customers later bought again or renewed because of an agent. The clearest distinction is between an agent changing an intermediate measure and an agent retaining a customer.
| Study and scope | Observed result | Boundary |
|---|---|---|
| Delivery-app marketing: about 8.8 million inactive users, agentic versus usual messaging over 11 months | In the seven months without new manual content, the authors report 2.4% (±0.21) more engagement days per user; 0.2% (±0.27) relative lift in days with intent and conversion events. The ± figures are their uncertainty estimates. | Activity days, not a separate repeat-purchase or renewal rate. The authors work for the agent platform. |
| Alibaba service: randomized deployment across 647 workers and 680,676 chats in August 2024 | Across all chats, average duration fell 3.2%; changes in seven-day same-issue repeat contacts and ratings were not statistically significant. | Short-term service outcomes, not later purchasing or renewal. One author works for Alibaba. |
| Alibaba chats eligible for AI handling: less than 10% of chat volume | Average duration fell 16.8%; ratings fell 0.412 points on a five-point scale; the seven-day repeat-contact change was not statistically significant. | Only 17% of all chats received ratings. The eligible subset does not represent every support issue. |
In the marketing study, the control group continued to receive the app’s usual messages. For four months, marketers actively supplied content and optimized the agent; for the next seven, the agent operated without new manual content or optimization. The engagement result during that latter phase is relevant to reactivation. The much smaller reported estimate for intent and conversion days, relative to its stated uncertainty, does not establish a customer-retention lift. The vendor affiliation and single anonymous app also limit how far to carry the result.
The Alibaba study shows why service speed needs its own quality check. A rating describes the responding subset, and a seven-day contact about the same problem describes service friction. Neither records whether that customer remained a buyer.
The researchers also compared escalation types. In their matched, non-randomized subgroup comparison, chats escalated after the system detected customer frustration had a six-percentage-point higher seven-day same-issue repeat-contact rate and a 0.928-point lower rating than comparable fully human-handled chats. That pattern makes delayed recovery worth monitoring; it does not prove that escalation timing alone caused the difference.
How would you test whether an agent retains customers?
First, specify the action the agent will change. Is it choosing reactivation messages, handling routine service issues, or both? Each creates a different comparison. Then keep an eligible customer group on the current process while another group receives the agent-enabled process. Random assignment and a persistent holdout follow the design used in the marketing study; the comparison is agent versus your existing process, rather than agent versus no contact. The AI customer retention guide gives a template for stating that decision before a test.
Choose the customer outcome before launch: for example, repeat purchasing or renewal over a window that fits the buying cycle. Record the result for everyone assigned to each group, including people the agent never contacts or whose chat goes to a human. Otherwise, the analysis can select the easy cases the agent chose to handle.
| Question | Measure | What it can establish |
|---|---|---|
| Is the agent doing work? | Decisions delivered, chats handled and human handoffs | Adoption and operating load |
| Is the immediate experience improving? | Engagement, chat duration, same-issue repeat contacts, complaints and ratings | Response and service quality within the measured window |
| Are more customers staying? | Repeat purchasing or renewal in the assigned groups over the chosen window | Incremental retention for this agent policy and customer group |
The second row is a safeguard for the third. If a service agent reduces handling time while customers return with the same unresolved issue, that is a reason to inspect the workflow even before a retention result is available. If engagement rises but the purchase difference is too uncertain, report the engagement gain and leave the retention claim open.
For a service deployment, inspect eligible chats and the full queue separately. In the Alibaba study, ratings across all chats did not change significantly, while ratings for AI-eligible chats fell.