Observing AI Agents with OpenSearch: Observability Practices for Hermes-agent

When AI Agents Enter Production, How Do We Observe Them?

In this talk by Stefano Pampaloni — RIOS Network Ambassador and Head of CECOM — he raises a question that's becoming increasingly real for engineers: when autonomous AI agents are actually running in production, how do we perceive, understand, and observe them?

This question isn't limited to any single agent framework (such as the Hermes-agent referenced throughout the talk). It's a universal challenge facing all autonomous agent systems. Traditional IT observability is fairly mature, but it falls short when dealing with non-deterministic systems that rely on LLM reasoning.

Why Traditional Observability Falls Short

The talk emphasizes that OpenTelemetry has become the de facto standard in observability. Hosted by the CNCF (Cloud Native Computing Foundation), it's vendor-neutral and covers traces, metrics, and logs. Most critically, it provides unified semantic conventions across all of these signals.

Progress on the OpenSearch side is equally noteworthy: from Data Prepper's support for OpenTelemetry to OpenSearch 3.6 becoming a native OpenTelemetry platform, the observability stack has matured considerably. Metrics storage is currently dominated by Prometheus, but the roadmap includes Prometheus API integration to position OpenSearch as a long-term metrics store.

The real problem is that agent systems operate on fundamentally different logic than traditional systems. Traditional monitoring asks whether a service is alive and whether calls succeed — if something breaks, we know where. But agents autonomously plan, call various APIs, interact with LLMs, and produce non-deterministic intermediate results. One key insight from the talk:

Even if every API call succeeds, an agent might produce nothing of value.

In other words, you might see dozens of "successful" calls and normal token usage, yet the agent has actually been stuck in a meaningless loop or drifting toward the wrong goal. Looking at the success or failure of individual calls is meaningless — you have to observe the whole run.

---

OpenTelemetry (OTel) is an open-source observability framework founded by the CNCF in 2019, created by merging the OpenTracing and OpenCensus projects. Its core value is providing a vendor-neutral, unified specification: developers instrument their code once with the OTel SDK and can simultaneously export traces, metrics, and logs to any compatible backend (such as Jaeger, Prometheus, or OpenSearch), avoiding vendor lock-in. "Semantic Conventions" are a key design element of OTel — they define standard naming and meaning for attributes in specific domains (such as HTTP requests, database operations, and GenAI calls), enabling telemetry data from different systems to be uniformly interpreted and correlated. This shared semantic layer is precisely what allows a supervisor agent to understand the true meaning of data reported by various components across different subsystems, rather than just seeing a jumble of key-value pairs.

---

The Hermes OpenTelemetry Plugin: Closing the Semantic Gap

Based on this insight, the team built an OpenTelemetry plugin for Hermes. The starting point was the standard GenAI semantic conventions, but they found the standard insufficient to describe the full lifecycle of an agent — so they extended it while staying compliant, adding fields specific to the Hermes context.

The goal is to correlate different signals (traces, logs, metrics) to truly reconstruct "how the agent arrived at its current state," rather than seeing isolated log entries. The team also gave serious thought to privacy — many similar plugins collect everything indiscriminately, which is hard to justify from a privacy standpoint. Hermes's plugin therefore collects only metadata by default, with configurable parameters to control what additional information is captured.

A separate design element is a SPAN validation mechanism that independently determines whether an agent's actions are authorized or out of bounds. This works in tandem with the collection plugin, forming two layers of governance capability.

Connecting Agents to OpenSearch: Closing the Data and Analytics Loop

The central highlight of the talk is a demonstration of bidirectional integration between Hermes and OpenSearch. On one side, telemetry data generated by the agent is written into the OpenSearch observability stack. On the other, the Hermes agent itself can act as an analyst — querying and interpreting data stored in OpenSearch.

This introduces the concept of "Agentic Health" — a platform for assessing the health of AI agents. It can define key tasks, run tests, use AI to analyze agent behavior, and even test different scenarios on the same system to compare results. At its core, it answers: "What did we expect the agent to do, what did it actually do, and do those match?"

The integration is technically implemented via OpenSearch's MCP server (Model Context Protocol). Once the Hermes Supervisor connects to OpenSearch's MCP server, it can invoke existing tools in OpenSearch within a minute — including AI agents running on OpenSearch. The tools demonstrated included "OpenSearch Observability Investigation," "Hermes Session Health," and "Hermes Golden Path."

The value of this design is that no matter which interface you use, everything points to a single source of truth — OpenSearch. It preserves the flexibility of autonomous agents while giving humans a unified entry point for governance and control.

---

MCP (Model Context Protocol) is a protocol proposed and open-sourced by Anthropic in late 2024, designed to provide LLMs and AI agents with a standardized "tool-calling interface." The concept is similar to a USB port: an MCP Server exposes a set of tools and resources, and any MCP-compatible client (agent) can connect and invoke those capabilities without writing custom adapter code for each external system. By implementing an MCP Server, OpenSearch allows agents like Hermes to discover and call all registered analysis tools — including other AI agents running on OpenSearch — using a single unified protocol. This standardization dramatically reduces the integration cost between heterogeneous systems, and is the technical foundation for the "connect and use tools within a minute" capability highlighted in the talk.

---

Using Agents to Govern Agents — While Keeping Humans in the Loop

One architecturally thought-provoking idea is using a dedicated supervisor agent to manage a fleet of other agents. European-style autonomous agents (like Hermes) tend to self-iterate — creating new things, modifying existing code, learning and rewriting scripts — so they grow quickly and require a higher-level coordinator.

This supervisor integrates the full stack described above: OpenTelemetry, the observability stack, and agent lifecycle data, performing dynamic analysis on top of all of it. A key design principle is that the supervisor agent doesn't need to re-collect data itself — it works directly from what's already in OpenSearch, and pulls humans back into the decision loop (human-in-the-loop) when necessary.

In the live demo, Stefano ran three agents simultaneously on his desktop: a lifecycle agent running in Docker, one running locally on a Mac, and the supervisor itself. He interacted with a personal assistant via Telegram and demonstrated in the Hermes Desktop interface the complete chain of the agent reading information from OpenSearch and generating an issue analysis report — including who initiated it, who is watching it, and who has memorized the relevant events.

---

Human-in-the-Loop (HITL) in AI systems refers to introducing human review or confirmation at critical decision points, rather than letting the system run fully autonomously. In autonomous agent scenarios, HITL is especially important: agents may modify code, call external APIs, or trigger irreversible actions — without a human intervention mechanism, the blast radius of a bad decision is difficult to contain. In this architecture, the supervisor agent doesn't replace human oversight; it acts as an "automatic filtering layer." It continuously analyzes whether lower-level agents are deviating from expected behavior, and when it detects an anomaly or an action requiring authorization, it packages the context and surfaces it to a human operator for a decision. This preserves the efficiency of autonomous operation while avoiding the uncontrollable risks of "fully automated, unsupervised" systems — making it the mainstream safety architecture for deploying autonomous agents in production today.

---

Engineering Trade-offs: LLM Selection and Runtime Environment

During the Q&A, Stefano shared several pragmatic engineering observations:

• Model selection: For the demo, he used the OpenAI API (for its performance) and has also used Anthropic's models; local LLMs run inside Docker. For narrow, specialized tasks like these, he believes local LLMs are well worth using — they protect privacy and are fully capable for specific scenarios.

• Privacy controls: The system collects only metadata by default, but configuration parameters let you publish additional information, allowing you to calibrate the trade-off between analytical depth and privacy protection.

• Runtime environment challenges: They currently run agents in Docker as developer assistants in DevOps/GitOps scenarios, but have encountered session management challenges — how to reuse sessions and handle large numbers of concurrent sessions. The team is evaluating Kubernetes and sandbox platforms like Daytona for running agents that are "inactive but need to be woken up at any time."

Three Key Takeaways

Stefano closed the talk with three sentences that frame the core of agent observability:

1. OpenTelemetry matters because it is the common language that provides unified semantics;
2. OpenSearch is the data landing and storage layer;
3. The agent is the control plane, responsible for analysis and governance on top of all that data.

Starting from the original question — "how is the agent doing?" — the answer lands on "use another dedicated agent to observe and govern it." This is a direction still in active evolution, but it clearly identifies the next unavoidable question in operationalizing agent systems: observability is no longer a nice-to-have. In production, it's a hard requirement.