← Home/Case Studies

Building trust in AI auto-tagging

Company
Wootric (acquired by InMoment)
Product
CXInsights (CXI)
My role
Lead Product Designer
Focus
AI tagging, sentiment, trust

Context

What is CXI?

CXInsights (CXI) was Wootric’s app for analyzing open-ended survey feedback at scale.

Each comment was tagged by topic, like UX/UI or ONBOARDING. Every tag got its own sentiment score, and so did the comment as a whole. We grouped sentiment into Negative, Neutral, and Positive.


The problem

Customers didn’t trust the AI

“I think the AI could be better.”

Anonymous G2 customer review

Customers didn’t have confidence in AI to do auto-tagging and apply sentiment correctly.

For our Enterprise customers who had millions of responses, they wanted all their responses in one place with the AI run on top of it. So we needed a solution that scales.

A feedback card with a Smart Tag and its positive and negative tag sentiments
A Smart Tag with its sentiment, applied automatically.

Goals

Objectives

  1. 01

    Earn confidence

    Automatically tag feedback and apply sentiment scores in a way that gives customers confidence it’s been done correctly.

  2. 02

    Scale across customers

    Design the system so it scales across customers and industries.

  3. 03

    Make it actionable

    Add reporting that gives customers actionable insights and limit manual work.


Research

Listening to enterprise customers

After a dozen interviews with Enterprise customers, we narrowed down the list of issues.

“As with any AI tool, to get the detailed information we require, there is a lot of ‘training the tool’ to detect our needs, but working with their data scientist to get there.”

Wootric customer

“The autotagging, while nice, has no exclusions (so you can’t separate ‘bad’ from ‘not bad’), which requires some manual checking for accuracy.”

Wootric customer

“Text analytics that supports Danish, Swedish, Norwegian and Finnish.”

Wootric customer

Exploration

First test: generic LLM models

We started by testing generic models with our customers. They missed a lot of context. “Michelle crushed it” came back negative. A comment about uploading documents on mobile got tagged DOCS instead of PERFORMANCE. It wouldn’t work well enough for our customers.

Table of short feedback examples comparing generic algorithm tags with Wootric’s tags and sentiment
Generic algorithm examples vs the models we would build.

Approach

Custom models, built with customers

Customers wanted the AI to be at least as good as their own teams already were at tagging feedback. Our three personas were spread across a wide range of industries, so we partnered with customers to create training data for each one.

Logos of Wootric customers, including Buffer, Raygun, Docker, Grubhub and Hootsuite The three CXI personas: CX, PX and CS Program Owners

One tag set per industry

Each industry got its own set of Smart Tags, trained on that industry’s feedback. Every tag came with a plain-language definition, so customers knew exactly what it would catch.

E-Commerce

Feedback from users of any site with an e-commerce component.

14 tags
Billing
Any feedback about the payment process, like declined credit cards or double charges.
Customer Service
Comments about experiences with customer service for the platform.
Customer Suggestions
Suggestions for enhancements, or general notes on areas that can be improved.
Discounts
Comments about the benefit, or lack, of discounts and deals.
Fitting
Feedback about the “fit” of the product. Mainly applies to clothing retail.
Inventory
Selection or stock levels, like “Love the variety!” or “X was sold out.”
Order/Checkout
The ordering and checkout process, like “The checkout window was very clunky.”
Price
The relative cost of products, and value for the money spent.
Product Quality
General quality terms like “sturdy,” “breaks easily,” or “works great.”
Returns
The return process. Usually mentions “return” or “refund.”
Shipping/Packaging
Any part of shipping or packaging, from long waits to items arriving broken.
Subscription
Subscription renewal issues or cancellation.
Technical
The UI or technical experience, from good search to common bugs.
Value Prop
Comments that highlight or compliment a specific aspect of the product or platform.

Testing

CXI beta

We gave customers a beta version of CXI and let them use it while we gathered feedback.

The CXI beta feedback view with a list of tagged feedback cards
The CXI beta feedback view.

Feedback

What customers asked for

“If I update a tag or sentiment, that should update the model.”

Customer ask

Why we said no

A feedback comment with its tags, zoomed in on a single tag being changed

“Can I play with the model?”

Customer ask

Why we said no

Bar chart of F1 score percent delta by tag Chart of human vs LLM mean sentiment scores with 95% confidence intervals
Model evaluation: F1 score delta by tag, and human vs. LLM sentiment scores.

Solutions

Giving customers control without retraining

More auto-tagging control

Managing tags: scope, type, and description.
Hover cards for Smart, Text Match and Manual tags, each showing tag sentiment and how the tag was applied
Hover cards explain how each tag was applied and what sentiment it carries.

Multi-language

Portuguese feedback cards with English translations shown underneath each comment
Portuguese feedback with the English translation added inline.

Proof

AI vs. humans

Trust needed evidence, so we ran a test with a CXI customer.

5

team members tagging

500

responses each

1 model

as accurate as all five people combined


Results

Outcomes

“Text sentiment with machine learning tags is an awesome addition to the tool. We implemented the tool two years ago and are pleased to see the progress in the right direction.”

DocuSign

“Auto-tagging is very accurate, and the default feedback breakdown column is very helpful.”

RingCentral

“We love the fact that it automatically performs basic NPS analysis and automates the time-consuming task of organizing qualitative feedback.”

Resultados Digitais