Soteria: Formal Verification + Runtime Enforcement for Both Safety and Utility in LLM Agents

Soteria uses pre-execution formal verification and runtime enforcement to raise LLM agent success rates to 1.7x baseline with zero violations.
The article introduces the Soteria framework, which resolves the tension between safety constraints and task utility in multi-step tool calling by LLM agents. Existing methods either interrupt tasks through post-hoc blocking or rely on prompts without formal guarantees. Soteria uses a three-stage "plan-verify-execute" process: it generates a hierarchical plan, rigorously compares it against formal specifications, and allows only verified plans to execute; at runtime, if an action is blocked, the verified plan serves as a navigation map to help the agent find alternative paths. Experiments show an average task success rate 1.7 times the best baseline with zero policy violations. The article also notes challenges such as specification writing costs, planning quality depending on the underlying model, and runtime latency, but emphasizes that safety and utility are not a zero-sum game.
LLM agents performing multi-step tool calls must simultaneously satisfy two goals: completing the user's task (utility) and complying with domain policies and correctness constraints (safety). These two goals are inherently in tension—blocking unsafe actions avoids violations, but it can also leave the agent with no viable path forward. Existing methods often sacrifice one to preserve the other. The recently proposed Soteria framework attempts to reconcile this conflict through formally verified hierarchical planning and runtime specification enforcement.
The Safety-Utility Dilemma
Multi-step tool calling is a core capability for deploying LLM agents, but each step can cross a safety red line. For example, an agent handling refund requests that directly executes user instructions while ignoring amount-limit policies could cause financial loss; conversely, if it simply blocks all suspicious actions, it may fail to complete legitimate tasks, severely degrading user experience.
Existing approaches fall roughly into two categories. One focuses on post-hoc filtering, i.e., blocking violating actions, but once the agent is interrupted it often lacks a recovery strategy, and task success rates drop significantly. The other focuses on ex-ante constraints, using prompts or fine-tuning to make the model "inclined" toward safe behavior, but without formal guarantees, violations can still occur. Soteria's starting point is precisely to break this either-or trade-off.

Soteria's Core Mechanism: Verify First, Then Execute
Soteria's design can be summarized as a three-stage process: "plan-verify-execute." Before the agent calls any tool, it first generates a structured hierarchical plan. This plan is rigorously compared against formal specifications, and only plans that pass verification are allowed to proceed to execution.
The key is that verification is not a post-hoc check of the final result, but an a priori proof about the plan trajectory. Formal methods ensure that every step in the plan complies with domain policies, thereby eliminating the possibility of violations at the source. Compared with approaches that rely on the model's self-discipline, this "prove-then-act" approach provides stronger safety guarantees.
Formal verification refers to techniques that use mathematical methods to prove that system behavior satisfies a given specification. In agent settings, specifications are typically described using temporal logic (such as Linear Temporal Logic, LTL) or automata, writing policies like "refunds must not exceed the limit" and "authentication must precede operations" as decidable formal constraints. A verifier then performs model checking on the plan trajectories generated by the agent to determine whether they satisfy the specification on all possible execution paths. Unlike testing or post-hoc monitoring, formal verification provides guarantees covering all paths rather than sampling-based checks. The cost is that specifications must be written precisely and unambiguously, and verification complexity grows with the state space, so in practice abstraction and hierarchy are often used to keep it within acceptable bounds.
Runtime Enforcement: There Is Still a Way Forward When Blocked
The conflict between safety and utility is sharpest at runtime: when an agent attempts an action prohibited by the specification, traditional methods simply reject it, leaving the agent without direction. Soteria's ingenuity is that a verified plan plays a dual role at runtime—it is both a guarantee of compliant trajectories and a navigation map when blocked.
Specifically, when an action is intercepted, the agent can backtrack to the verified plan to find alternative paths or adjust subgoals, rather than blindly retrying or abandoning the task. This mechanism turns "blocking" from a dead end into a branching point, enabling the agent to keep making progress within safety boundaries.
Runtime enforcement is a technique that dynamically monitors and intervenes in behavior during system execution. It typically uses a monitor to observe the agent's action sequence in real time; once an action that is about to violate the specification is detected, it is blocked or corrected. The difference from static verification is that runtime enforcement deals with actually occurring execution traces and can handle dynamic environmental changes and planning deviations. Classic enforcement mechanisms include safety automata and edit automata: the former only makes allow/deny decisions, while the latter can also output a replacement action that conforms to the specification after blocking, thereby helping the system continue. Soteria's use of verified plans as navigation maps essentially gives the enforcement mechanism the ability to "provide an alternative path after blocking" rather than simply rejecting.
Experimental Performance: Large Success Rate Gains with Zero Violations
According to the original paper, Soteria achieves an average task success rate 1.7 times that of the best baseline across multiple benchmarks covering diverse tool-use tasks, with zero policy specification violations in all tests. This result directly rebuts the pessimistic assumption that "safety necessarily sacrifices utility" and shows that, through sound design, the two can improve together.
As a caveat, the 1.7x success rate improvement was obtained under specific benchmarks and baseline comparisons; real-world deployment performance will also be affected by specification quality, task complexity, and other factors. But the zero-violation result itself is landmark, proving the feasibility of combining formal verification with runtime enforcement.
Baseline in machine learning and system evaluation refers to the reference method used for comparison. Here, "best baseline" means the authors selected the best-performing among multiple existing approaches as the comparison object, rather than comparing against an average method, making the 1.7x improvement more compelling. It is important to distinguish two metrics: task success rate and policy compliance rate. The former measures whether the agent completes the user's goal, while the latter measures whether its trajectory violates domain specifications. Most methods exhibit a trade-off between these two metrics; Soteria's reported "high success rate with zero violations" is precisely the quantitative embodiment of its core claim. When interpreting such results, readers should also pay attention to the benchmark's task distribution, specification complexity, and whether adversarial scenarios are included.
Implications for Agent Deployment
Soteria's value lies not only in a framework but also in the engineering approach it offers: upgrading safety constraints from "soft prompts" to "hard specifications," and balancing efficiency and compliance through the decoupling of planning and execution. For high-risk scenarios such as finance, healthcare, and enterprise process automation, this verifiable safety guarantee is particularly critical.
Of course, the method also faces challenges. Writing formal specifications requires domain expert involvement and is costly; the quality of hierarchical plan generation depends on the underlying LLM's capabilities; and runtime enforcement may introduce additional latency. But none of this prevents it from pointing to a promising direction—safety and utility need not be a zero-sum game.
As LLM agents move from demos to production, how to make them both smart and reliable in open environments will remain a core issue. Soteria's attempt reminds us that formal methods are not ivory-tower theoretical tools; they can serve as a practical foundation for the safe deployment of agents.
One-line Summary
Soteria uses pre-execution formal verification and runtime enforcement to raise LLM agent success rates to 1.7x baseline with zero violations.
Paragraph Summary
The article introduces the Soteria framework, which aims to resolve the dilemma between safety constraints and task utility in multi-step tool calling by LLM agents. Existing methods either interrupt tasks through post-hoc blocking or rely on prompts without formal guarantees. Soteria adopts a three-stage "plan-verify-execute" approach: it first generates a hierarchical plan, rigorously compares it against formal specifications, and allows only verified plans to execute; at runtime, if an action is blocked, the verified plan serves as a navigation map to help the agent find alternative paths rather than hitting a dead end. Experiments show an average task success rate 1.7 times the best baseline across multiple tool-use benchmarks with zero policy violations. The article also notes challenges such as specification writing costs, planning quality depending on the underlying model, and runtime latency, but emphasizes that safety and utility are not a zero-sum game, offering a verifiable safety approach for deploying agents in high-risk scenarios.
Related articles

Gemini 4 Neon Review: Benchmark Scores Rival Carbon, API Pricing and Quota Breakdown
Google Gemini 4 Neon is now live, benchmark scores closely tracking Carbon, API pricing 3/15 at twice Hydrogen's rate. This article compiles Bilibili creator testing: Neon can replace 90% of Carbon scenarios, AntiGravity quota consumption is only one-third of Carbon, plus subscription channel restrictions and 2API ban risks.

Gemini 4 Helium Coming Soon? Signals of Google Loosening Subscription Restrictions
Google team member hints releases will continue; community speculates Gemini 4 Helium (Flash Lite) may be next. Subscription ban controversy gets a response as product lead says "Soon," suggesting restrictions may be lifted. This article covers Gemini developments, OpenAI Coding Agent and risk management, Doubao real-time audio-video API, and compute business dynamics.

AI-Assisted React Development in Practice: Optimizing a Video App and Handling Errors
A 14-year veteran developer demonstrates AI-assisted React video app optimization: compressing video from 80MB to 5MB (~94%) via ImageKit, plus reliable error handling and retry mechanisms. Includes planning, code understanding, and verification methods for working with AI.