What AI should not do with a customer record.

Three categories, kept apart

Nearly every failure in this area comes from collapsing three different things into one output. Keeping them visibly separate is most of the solution.

• Fact — this customer opened four cases this month. Checkable.
• Inference — this customer may be becoming frustrated. A reading, labelled, with its evidence.
• Recommendation — the account owner should call them. A suggestion for a person.

Never present inference as fact

"This customer is frustrated" stated flatly will change how your team writes to them, and it may well be wrong. Stated as an inference with the evidence attached, it is useful. The difference is a formatting decision with a real consequence.

Structure first, model second

A model reasoning over unstructured conversation text will produce confident, plausible and occasionally wrong statements about a specific customer's account. The answer is not a better prompt — the underlying facts have to be fields with sources, and the model's job is to read and explain them.

Provenance is not optional here

When an answer is about a customer, an unsourced claim will be repeated to that customer. Every answer should name the records behind it and when each was last updated. If a vendor cannot show you where an answer came from, it is not evidence.

Permissions must be inherited

The only safe design is that assistant queries run with the asking person's identity through the same checks as every screen — and in a customer-facing surface, with that customer's own scope. A separate AI permission model will eventually diverge, and the divergence is a breach.

Anybody who can email you can inject content

This is specific to support and it is easy to miss. Inbound messages, attachments and imported articles all become content the AI reads. That content must be data, never instruction, as a property of the design rather than a filter attempting to spot bad input.

Draw the line at money, access and complaints

Refunds, credits, cancellations, contract changes, account access and complaints should never be handled automatically, no matter how routine they look or how good the model gets. Those are the decisions where being wrong is expensive and irreversible.

Questions worth asking any vendor

• Does your AI have its own permission model, or does it inherit the user's?
• Can I see the records behind any answer?
• Is my customers' conversation data used to train models available outside my account?
• What happens if a customer's email contains text designed to manipulate the assistant?
• Which categories can never be answered automatically, and can I change that?
• Does the product ever characterise or score an individual customer?