Product
Observability: From Plausible to Checkable
Harry Yu
When CX teams evaluate an AI agent, they look at CSAT and resolution rate. Those metrics say that something went wrong. They don't say what, why, or what to change.
So teams investigate the way they always have: pull the negative-CSAT tickets, read them in bulk, look for a pattern. But an agent's decision doesn't live in the transcript. It was assembled from customer data in a backend system, an article in the knowledge base, an Action the agent took through your API, and a step in a Playbook; and when those inputs are disconnected from the reply, the pattern you find tells you something failed without telling you which of the four to fix.
Fix the wrong one and the problem stays. Editing a playbook step won't correct a bad customer variable or an outdated policy.
Anyone who has used ChatGPT or Claude knows the shape of this. The answer sounds convincing; before you act on it, you want to know where it came from. An AI agent deciding a customer's refund, reward, or eligibility deserves the same standard. The final response isn't enough. You need to know what it relied on.
One reply is a stack of decisions
Observability moves the unit of trust from the dashboard to the individual turn. Instead of asking whether the agent is performing well on average, you ask what this specific reply was built from and whether the pieces hold up.
That question has an answer because resolving a ticket has a path, whoever is doing it. Faced with a customer inquiry, a good support rep first decides whether this is something to handle or something to hand off. Then they work out what the customer actually wants. Then they either look the answer up or work out what went wrong and follow the procedure for it. These aren't habits or personality. They're the decisions the job requires, and a reply that skips one of them is a guess.
Level3AI's agent makes each of these decisions as a distinct stage:
Detect decides whether to proceed at all: a request for a human, an emergency, an off-topic message, or a sign the issue is already closed. Any of these stops the process here and triggers a pre-defined reply.
Triage decides what the customer wants, and routes: a question goes one way, a problem or a request for action goes another.
Knowledge answers a question from your documentation, retrieving the articles it will compose the reply from.
Investigation identifies which specific Journey Issue the customer has, asking a clarifying question if it can't yet tell.
Playbook executes the procedure for that issue, step by step, choosing a branch at each fork.
The reply you read is the last inch of that path. Select any agent message in a ticket and the Observability panel shows the full path behind it: each decision, what it was based on, and the exact customer input the agent was reasoning over.
Seeing these decisions requires that they actually exist as separate stages. In most AI agents, they don't.
The structured architecture that enables Observability
Most AI agents are built one of two ways, and neither makes its decisions inspectable.
The first is a single prompt and a single call. Conversation, instructions, retrieved articles, and customer data go in as one block; a reply comes out. Nothing happened in between that you can point to. When the reply is wrong, there is no way to tell which line produced which sentence because instructions, context, and reasoning were folded into a single pass. There is also no way to predict what an edit will change, because everything in the prompt can interact with everything else.
The second is a loop. The model is given a goal and a set of tools, decides what to do next, takes an action, evaluates the result, and repeats the process until it determines that the goal has been met. This is more observable than the monolith: each tool call is a real event, so there is a log. What it lacks is a reference. The path was chosen at runtime, so you can see the steps taken but there is nothing to check them against. You know the goal; you never specified the route. Checkability needs a specification, and a loop has only a destination.
Level3AI's workflow agent makes each of the stages above (Detect, Triage, Knowledge, Investigation, Playbook) its own call to the LLM model, with defined inputs and a returned output, both recorded at the boundary. Triage doesn't narrate a classification; it emits one, and that value routes the message. Because the path is specified in advance, what happened can be laid against what was supposed to happen. That comparison is what turns a log into something checkable.
The same choice recurs inside the Playbook, the stage that carries most of the procedure and most of the risk. A stateless Playbook is prose that the model re-reads every turn, working out afresh where it is and what comes next. It's fast to author. It also stops working at scale: with thirty Playbooks, dozens of Variables, and a refund in the balance, a model re-deriving its position each turn will read the same instructions differently on different runs, and leave no record of which step it thought it was on.
In Level3AI's structured Playbook, a state machine implemented in code, not the model, holds the current position and decides which step and branch to execute next. The model supplies language and judgment inside each step. "Which step ran" is a fact, not a wish.
Each architecture makes a different trade. A single call is the fastest and cheapest to run, and the cheapest to author: write one prompt, ship it. What it gives up is any ability to inspect or predictably edit the result. A loop buys flexibility: the agent can handle situations nobody anticipated at the price of a call count decided at runtime, so latency and cost vary from turn to turn and the path can't be checked against anything. A workflow agent spends more effort upfront: the stages and the Playbook steps have to be designed rather than described. In return, the number of calls per turn is fixed, so latency is predictable; each call is small and specific, so instructions are followed more reliably; and every decision is recorded at a boundary, so when something goes wrong, the time to find out why is measured in minutes, not in hours of re-reading transcripts, re-reading the prompt, and guessing.
That is the trade Level3AI has made. It optimizes for the moment after something goes wrong, because at enterprise scale, that is the expensive moment.
So when something goes wrong: can you tell exactly which step failed, and why?
From Observability to QA and Coaching
QA and Coaching both produce findings, and a finding without an address is an opinion about a transcript. Observability gives every finding an address: the Knowledge Base article retrieved, the Playbook branch and step executed, the Variable that changed and its before/after values, or which Action the agent took and the request/response behind it.
That address is what lets QA answer the question that matters: was the source wrong, or was the AI wrong? An outdated article, a Variable that an API never populated, an Action that failed after the agent had already said it succeeded, a Playbook branch that mishandles a missing value, and a model that misjudged the customer's intent all look the same in a transcript. In the Observability panel of Level3AI, these issues are four different findings. Look at what each one actually takes to fix in the table below. All of them have nothing to do with the model.
From there the flywheel turns. Insights aggregates findings by source, so two hundred escalations become a ranked list with owners: three articles, one API, one branch. Coaching applies the fix where the error lives, in the article or the Playbook step instead of the agent's phrasing. Eval replays the failing turn against the change to confirm it holds. Deploy, and new conversations feed new records back into QA.
Observability is the substrate of that flywheel. It doesn't score conversations or apply feedback; it makes each step checkable.
Conclusion
An AI agent's reply will always sound plausible. That's what language models do. The question for a CX team is whether it can also be checked and whether the reply resolves to a Variable you can inspect, an article you can open, an Action you can verify, or a Playbook step you can read.
When it does, the work changes. "Something went wrong" becomes "here is what went wrong, and why." Root cause stops being a reading exercise and becomes a lookup. And every fix has an address, so it can be validated before it reaches a customer.
That is what Observability is for. Not to explain the agent, but to make it checkable.
