Can you trust an AI-generated email performance report?

Trust a checked claim, not the finished-looking report. Ask the AI for the account, campaign or flow ID, source report, date range and metric definition behind its conclusion. Reproduce the decision-making number in that source, inspect the attribution settings and check relevant order records. If a figure or explanation cannot be traced, leave it out of the decision.

An AI assistant can summarize connected email data or generate a report query. Neither form guarantees that the answer used the campaign, period or definition you meant. Klaviyo’s guidance for its ChatGPT app explicitly warns that generated analysis can be inaccurate or rest on wrong assumptions, and advises asking it to cite the segment names, flow names or campaign IDs behind important facts. Shopify documents how Sidekick turns plain language into a report query. In either case, the useful question is whether you can trace the answer back to an inspectable source.

This guide is for reviewing a generated analysis before you change a campaign or budget. For how to collect sales data and reconcile the store with the email platform, use the email sales-tracking guide. The audit below starts after those records exist.

What exactly did the AI analyze?

Save the original question and answer. Then require a source record for each claim that would change your decision. “Email revenue rose” is not yet a usable claim if you do not know which account, messages and report produced it. Record the account and message ID, source report and last-run time, metric and conversion event, date range and date basis, filters, and attribution settings.

These are fields to verify, not a prompt that magically makes the answer true. A missing campaign ID is a reason to open the source report yourself. In the custom Campaign Performance Report, the documented path is Analytics → Custom reports; you can choose metrics, filters and message grouping, then run or export the result. Its preview shows when it last ran, and an unscheduled report needs a rerun to show updated data. A report grouped by Campaign Message can export the message ID. Preserve the settings with the export so another reviewer can reproduce the same population.

Can you reproduce the rate from its parts?

Pick the figure that drives the recommendation and write down its numerator, denominator and rule. Open the source interface or export, then compute it independently. A rate that matches only after changing “people” to “events” has failed the definition check even if the arithmetic is correct.

For example, the documented campaign conversion rate for a selected metric is unique recipients who completed it within the email attribution window ÷ delivered emails. “Sent” includes bounces; delivered emails do not. The selected conversion metric can also be changed in the campaign view.

Consider this invented campaign, with all figures for the same message and observation period:

The recipient conversion rate is 100 ÷ 5,000 = 2.0%. Dividing 120 order events by 5,000 deliveries gives 2.4%, but that is not the stated recipient conversion rate. Some customers made more than one order in this illustration. If an AI report calls 2.4% the platform’s recipient conversion rate, correct the numerator before discussing why performance moved. The figures are an arithmetic example, not a result from a store or a normal rate to expect.

For opens, first settle what the report actually counts; the open-rate guide covers privacy loads and filtering. Do not infer that a numerical difference is an AI mistake until the two views use the same definition.

Do attribution and order records support the sales claim?

Reproducing a total does not settle what “email sales” means. Check the conversion event, eligible touchpoints and lookback windows in the account before accepting the AI’s explanation. In Klaviyo’s attribution documentation, new accounts default to last-touch credit with five-day email open and click windows; an account may have different settings. Campaign performance reports group by message send date, while single- and multi-metric reports group by event date. The same attributed purchase can therefore appear under a sent-in-October message while the order itself happened in November. Attribution-setting changes also recalculate historical figures and may take up to 36 hours to appear.

If the answer names specific purchases or a revenue jump, inspect a few relevant order IDs in the store record. Check creation time, financial status, value and refunds. The Shopify order export supplies those fields, but an order with several line items creates extra CSV rows: count distinct order IDs, not rows. Its sales-report documentation also distinguishes sales value from payments received and records refunds as negative values when processed. A spot check can expose a bad order claim; it cannot prove every order in a total is correct.

An order record verifies that the order exists and what happened to it. It does not prove that an email deserves credit. For that, return to the email platform’s attribution rule and the recorded interaction. If the store and ESP totals differ, the sales-tracking guide gives the fuller reconciliation. Keep a difference labelled “unresolved” until you identify its cause, rather than calling every mismatch a hallucination.

Is the conclusion stronger than the checked data?

Read each recommendation separately from its supporting figures. “This campaign was credited with $X under these settings” is an attribution statement. “This campaign created $X of additional sales” is a causal statement; the credited figure alone does not establish it. A report that gets the first number right can still overstate the second conclusion.

Use a short decision record for the claims that matter:

Save the AI output, report ID or export, settings, check date, corrected figures and the decision you made. That record lets someone revisit the conclusion if late conversions or changed attribution settings move the numbers. The report becomes useful when its important claims survive this check, with their limits attached.