Observability

RelyAI is built so you never have to take AI on trust. Every request, conversation and tool call is logged, and a suite of admin surfaces lets you see exactly what the assistant did, how well it performed, and what it cost. Each surface is its own entry in the RelyAI sidebar group.

Logging never stops

Each observability surface can be hidden by turning its feature flag off on RelyAI → Settings → Feature Flags — but hiding a surface only hides the view. The underlying logging keeps running, so nothing is lost while a dashboard is switched off.

Usage Dashboard

Open RelyAI → Usage Dashboard for the health-and-cost picture at a glance:

• Runs and success rate across your AI activity
• P95 latency and average duration so you can spot slow tasks
• Estimated cost for the period
• A breakdown by task to see where the spend and volume go
• Daily runs over the last 14 days

RelyAI → Usage Dashboard

Conversations

RelyAI → Conversations lists every multi-turn session. Drill into one to replay the messages, tool calls and feedback exactly as they happened — useful for understanding a customer complaint or confirming what the agent actually did.

Audit Viewer

RelyAI → Audit Viewer is your tool-call forensics record: every agent action, with filters, a status time-series, top failures, P95 latency and a dedicated destructive-ops filter so you can review just the actions that changed something. Export any filtered view to CSV for your own records or compliance review.

Feedback

RelyAI → Feedback collects up/down ratings from across every AI surface, alongside sentiment categorisation, so you can tell at a glance where the assistant is helping and where it's frustrating people.

Self-Improve

RelyAI → Self-Improve turns all that logging into a to-do list. It clusters tool failures by severity, lists low-rated conversations and any URL hallucinations, and highlights knowledge-base hot zones — the topics people ask about most where answers fall short — so you know exactly what to fix next.

Permissions matrix

On RelyAI → Settings → Permissions you'll find a read-only matrix of who can use which AI tool. It's introspection only — it reports what AcelleMail already enforces and never changes it — and a customer can open a "View my permissions" view of their own access.

Related pages

• Agent tools — the actions the audit viewer records
• Feature flags and permissions — hide a surface without stopping logging