Measuring AI support: CSAT, resolution rate and deflection
To measure AI support, track true resolution (the issue was solved and the customer did not come back), customer satisfaction after AI conversations, customer effort, time to resolution, handover rate and quality, and the accuracy of AI answers from regular transcript reviews. Treat deflection, the share of customers who never reached a person, with caution: it can hide customers who gave up. Compare AI and human performance side by side and review a sample of conversations every week.
Why is measuring AI support tricky?
AI agents handle large volumes of conversations quickly, and dashboards often highlight how many conversations never reached a human. But a customer who gives up in frustration also never reaches a human. Without the right metrics, a poorly performing agent can look like a success. Good measurement combines outcome metrics, customer feedback and regular reading of real conversations. For context on where AI support is heading, see AI customer service trends.
Which metrics matter most?
| Metric | What it shows | How to measure |
|---|---|---|
| Resolution rate | Issues solved without further contact | Resolved conversations with no repeat contact within a set period |
| CSAT after AI | How customers feel about AI help | Short survey after conversations |
| Customer effort | How easy it was to get help | Effort question or number of steps and repeats |
| Time to resolution | Speed from first message to solution | Help desk timestamps |
| Handover rate and quality | When AI passes to people and how well | Transfers, reasons and whether context was passed |
| Answer accuracy | Whether AI answers are correct | Labelled transcript samples |
What is wrong with deflection?
Deflection counts conversations that did not reach a person. It is useful as a workload measure, but it says nothing about whether the customer got what they needed. Pair it with resolution and repeat-contact rates: if deflection is high but many customers return within a few days or rate the experience poorly, the agent is pushing work around rather than solving problems.
How do you review transcripts?
- Sample weekly: Pick a random set of AI conversations, plus all with poor ratings.
- Label each one: Correct, partly correct or wrong; tone; handover quality.
- Find patterns: Missing articles, confusing policies, weak handover rules.
- Fix the cause: Update content, instructions or routing.
- Track improvement: Watch accuracy and satisfaction over the following weeks.
How should AI compare with human agents?
Report AI and human metrics side by side for similar request types. AI may resolve simple requests faster, while people resolve complex ones better. The goal is the right split, not making AI look good. Use the comparison to adjust which requests AI handles and when it hands over. Our guide to AI chatbot vs live chat vs voice covers channel choices.
What should a monthly dashboard show?
- Resolution rate for AI and human channels
- CSAT and effort scores by channel
- Top handover reasons
- Accuracy from transcript review
- Content gaps fixed this month
How do you measure cost and value?
Leaders often want to know whether AI support saves money. Measure cost per resolved conversation for AI and human channels, including AI usage fees, platform costs and the time spent maintaining content and reviewing transcripts. Compare that with the cost of human handling for the same request types. Add value measures such as faster resolution, round-the-clock availability, higher satisfaction and agent time freed for complex work. A small saving per conversation across thousands of conversations adds up, but only if quality holds. If satisfaction or resolution falls, the apparent saving can turn into lost customers and extra contacts later.
Who should own the metrics?
Give one person, often a support operations lead, ownership of AI support metrics and the weekly review. They work with content owners to fix knowledge gaps and with engineers or the platform vendor to adjust instructions and handover rules. Share the dashboard monthly with leadership so decisions about expanding or limiting AI are based on evidence.
What does this look like in practice?
A subscription company launches an AI agent and celebrates a high deflection rate. A month later, it notices a rise in repeat contacts and poor reviews. It adds a resolution metric based on repeat contacts within seven days, a one-question satisfaction survey after AI chats and a weekly review of 50 transcripts. The reviews reveal outdated billing articles and a handover rule that triggered too late. After fixes, true resolution and satisfaction rise, and deflection stabilises at a lower but healthier level.
What mistakes should you avoid?
- Reporting deflection without resolution
- Surveying only customers who reached a person
- Ignoring repeat contacts
- Never reading conversations
- Changing several things at once, so causes are unclear
How UIUX Studio helps
We design AI support journeys and the dashboards that measure them honestly. Talk to our Agentic AI & Automation team, and see how to build a support AI agent from your help docs.
Frequently asked questions
What is the most important AI support metric?
True resolution: the issue was solved and the customer did not need to return.
Is deflection a good metric?
Only alongside resolution and satisfaction; on its own it can mislead.
How do I measure CSAT for AI chats?
A short survey after AI conversations, analysed separately from human chats.
How often should I review transcripts?
Weekly, with a random sample plus poorly rated conversations.
Should AI and human metrics be compared?
Yes, for similar request types, to set the right split.