AI Agent Security Monitoring: Using Post-Trained Open-Source Models to Guard Production Environments
Armada Security uses post-trained open-source models to build a dedicated behavioral monitoring layer for production AI agents.
As AI agents gain real-world permissions — database writes, API calls, and more — their autonomous decision-making introduces security risks that traditional rule engines can't handle. Armada Security researcher Yonatan Oren's approach is to post-train open-source models, injecting domain-specific security knowledge and anomaly detection capabilities to create a dedicated supervisory agent. Using open-source models keeps sensitive operation logs within controlled boundaries and supports deep customization; the solution runs on Fireworks Forge, lowering the engineering barrier for post-training and deployment. This reflects the rapid emergence of "agent security" as an independent discipline — the core imperative being that monitoring and constraints must scale alongside the permissions you grant.
When You Hand Over Permissions to an AI Agent
More and more teams are deploying AI agents in production environments, granting them the ability to execute real-world operations. But a critical question emerges: once you've given an agent those permissions, do you really know what it will do next?
This is the core issue that Yonatan Oren, founding researcher at Armada Security, has been focused on. In a recent technical talk, he revealed that his team is exploring a new approach — using post-training on open-source models to build a supervisory system specifically designed to monitor the behavior of security agents. The topic is set to be explored further at an in-person event in San Francisco.
Why Agents Need a Dedicated Monitoring Layer
Traditional software behaves deterministically: the logic is hardcoded, and the output is predictable. LLM-based agents are fundamentally different — they make autonomous decisions when given access to permissions and tools, and their behavioral paths are impossible to enumerate in advance. This unpredictability is especially dangerous in production environments. An agent with database write access, API call capabilities, or system command execution privileges might take unexpected actions in certain edge cases.
Rule engines and static policies alone can't cover every scenario. Rules can't anticipate how an agent reasons through complex contexts, and they struggle to determine whether a given action is legitimate business logic or a potential privilege escalation or anomaly. This creates a demand for an "agent that monitors agent behavior" — using a model to understand another model's actions.
The Approach: Post-Training Open-Source Models for Supervision
Yonatan Oren's team post-trains open-source models specifically for monitoring security agents. This approach has several layers worth unpacking.
Why Open-Source Models
Choosing open-source models over closed-source API calls comes down to practical considerations. Security monitoring often involves handling sensitive internal data and operation logs. Self-hosting open-source models keeps that data within a controlled boundary, eliminating the risk of external exposure. Open-source models also allow deep customization and fine-tuning, enabling teams to build highly specialized supervisory capabilities tailored to their specific security scenarios — without being constrained by the capability limits of an external API.
Post-Training as the Source of Monitoring Capability
General-purpose LLMs aren't naturally well-suited to judging whether a security agent's actions are compliant. Through post-training, teams can inject domain-specific security knowledge, anomalous behavior patterns, and compliance evaluation criteria into the model — effectively turning it into a professional "watchdog." This type of supervisory model needs to understand an agent's intent, assess the risk level of operations, and trigger alerts or block actions when necessary.
What is post-training? Post-training refers to the process of continuing to train a foundation model after its initial pre-training phase, targeting specific tasks or domains. Common forms include supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and direct preference optimization (DPO). Compared to training from scratch, post-training is far less computationally expensive, since the base model already has language understanding and reasoning capabilities — post-training only adjusts the model's behavioral tendencies in specific scenarios. For security monitoring purposes, teams typically need to prepare training samples labeled as "normal operations" and "anomalous operations," teaching the model to identify risk signals in real agent behavior logs. This process is essentially about transferring a domain expert's judgment into the model's weights, enabling the supervisory model to evaluate large volumes of operations for compliance at near-real-time speed — without requiring manual review every time.
Deploying on Fireworks Forge
According to the shared content, this monitoring solution runs on the Fireworks Forge platform. Infrastructure platforms focused on model fine-tuning and deployment lower the engineering barrier for post-training and inference serving, enabling security teams to put custom supervisory models into production much faster.
What is Fireworks AI? Fireworks AI is an infrastructure platform focused on high-performance LLM inference and fine-tuning. Its Forge product provides enterprises with capabilities including private model fine-tuning and managed inference endpoints. For security teams, the core value of using such a platform is twofold: it allows the post-training workflow to run in a private or semi-managed environment, reducing sensitive data exposure; and it abstracts away complex engineering details like distributed training, model version management, and low-latency inference serving — freeing algorithm teams to focus on data and training strategy rather than infrastructure. This is especially valuable for security startups with limited headcount, dramatically shortening the time from research prototype to production deployment.
Agent Security: A Fast-Emerging New Domain
As agents move from demos into production, "agent security" is rapidly evolving into an independent technical discipline. It differs from traditional application security or network security — the core challenge is how to constrain and observe a system capable of autonomous decision-making.
Several development directions are foreseeable: behavioral observability (recording and explaining every step of an agent's decisions), least-privilege access (dynamically granting rather than permanently opening permissions), and real-time supervision and intervention (stepping in before a dangerous operation executes). What Armada Security is doing — "using a model to monitor a model" — falls into that last category, and is among the most technically challenging.
For teams already deploying agents in production, this is an important reminder: once you grant permissions, monitoring and constraint mechanisms must keep pace. Otherwise, the convenience of autonomy can quickly become an uncontrollable risk.
A note on prompt injection: Prompt injection is currently one of the most closely watched attack vectors in agent security. Attackers can embed malicious instructions into external content that agents process — such as web pages, documents, or emails — tricking the agent into executing unauthorized operations, such as leaking system prompts, invoking dangerous tools, or tampering with data. The attack logic is similar to traditional SQL injection, but defense is significantly harder, because agents inherently need to understand and execute natural language instructions, making it difficult to distinguish legitimate commands from malicious injections at a syntactic level. The "model-monitors-model" supervisory approach can, to some extent, serve as a second line of defense against prompt injection — detecting anomalies by checking whether the agent's behavioral intent remains consistent with its original task objective.
Summary
The practice Yonatan Oren shared points to an unavoidable engineering challenge in the age of agents: how to keep AI systems controllable after they've been granted permissions. Post-training open-source models to supervise security agents is a path that balances data security with customization flexibility. As more teams push agents into production, the value of this kind of monitoring infrastructure will only grow.
Related articles
Why Does the Tesla Roadster Need to Fly? A Deep Dive into Musk's Wild Demo
Tesla's Roadster launch showed wall-climbing, upside-down driving, and flying stunts—but The Vergecast notes the demo car may lack a battery, the Rocket version is invite-only and not street-legal, and the website terms don't guarantee delivery. After nine years of delays, why does this car still need to exist?
AI Is Not a Replacement: Human-AI Collaboration Through the Lens of Factory Root Cause Analysis
Starting from factory root cause analysis, this article explains how AI accelerates evidence chain construction rather than replacing human experts, exploring the pragmatic implementation logic and business value of 'AI supporting people rather than replacing them.'
AI Agent Decision Auditing: Building an Evidence Chain in Regulated Workflows
Starting from a Reddit discussion, this article analyzes three core issues in AI agent decision auditing within regulated workflows: what fields audit records should contain, how to handle feedback latency for erroneous decisions, and how to continuously calibrate LLM judge consistency with human judgment.