Agent Observability

The core capability of Agent observability should extend from observation to debugging, requiring complete execution trajectories, intermediate states and causal relationships, not just metrics such as error rates.

When choosing an Agent observability solution, if it can only provide metrics and alerting, then it is closer to Datadog than to Databricks; a complete solution should be able to record multi-step execution trajectories, export structured data for evaluation, and support version comparison and experiment management.

Databricks expanded from a single function (data querying) into a platform covering the full data lifecycle, and Agent observability may follow a similar evolutionary path.

An Agent observability platform needs to support experiment management, being able to run multiple Agent variants in parallel, collect the same evaluation metrics (such as task success rate, number of tool calls, user satisfaction), and perform statistical significance tests, similar to MLflow's experiment tracking but adapted to Agent characteristics.

Agent observability focuses on the input context, tool selection, parameter construction, and final decision of each LLM inference, generating traceable traces that link the complete timeline from user intent to the next decision.

The author's core practical recommendation is: print len(messages) alongside the retry count.

Traditional model monitoring can cover latency, drift, and accuracy, but when an agent makes a series of decisions and calls multiple tools in sequence, these metrics are almost unable to explain what is happening.

For most teams, secondary development based on open-source frameworks is the most pragmatic compromise for building agent observability.

In agent scenarios, a complete agent task should be treated as a single Trace, with each LLM call, tool call, and memory retrieval being a Span.