LLM Selection Strategy for Multi-Agent SOC Applications: Rule-Based Routing vs. LLM-Driven Decisions

When building Multi-Agent Security Operations Center (SOC) applications, one core architectural question is often underestimated: how do you select the right large language model (LLM) for different stages of your workflow? A Reddit developer recently shared their practical dilemma — one that's highly relevant to any engineer building complex Agent systems on LangGraph.

A Real Architectural Dilemma

This developer built a multi-agent Agentic SOC application on LangGraph, with several collaborating agents:

• Router: Handles request dispatching and scheduling
• Basic Triage: Performs initial alert classification
• Advanced Triage: Handles complex, in-depth analysis
• Memory Candidate Generation: Accumulates reusable knowledge for the system

Currently, all reasoning tasks in the system are handled by Claude. The developer wants to introduce multiple LLM models to optimize cost and performance, but got stuck at a critical decision point: where should the LLM selection logic live? Should the router agent follow pre-defined, explicit selection rules, or should it dynamically call an LLM to decide which model to use?

LangGraph is a graph-based Agent orchestration framework developed by the LangChain team. It lets developers define multiple LLM calls, tool uses, and conditional branches as nodes and edges in a directed graph. Unlike linear Chains, LangGraph supports cycles, state persistence, and multi-agent collaboration — making it particularly suitable for complex scenarios that require multi-step reasoning and dynamic decision-making. In a SOC application, it can model alert ingestion, triage, analysis, and response as collaborating nodes, each independently bound to a different LLM or tool. This is exactly the technical foundation for the "multi-model routing" problem discussed in this article.

The Core Difference Between the Two Approaches

This may look like an implementation detail, but it's really a classic trade-off between determinism and flexibility.

Rule-Driven Model Routing

The first approach is to hardcode selection criteria in the router. For example: Basic Triage tasks are simple and frequently called, so they can be routed to cheaper, faster lightweight models (such as Haiku-tier or small open-source models); Advanced Triage involves complex reasoning, so it goes to a more capable model (such as Claude Sonnet/Opus or flagship GPT models).

The advantages of this approach are clear:

• High predictability: Which model handles which task type is deterministic, making debugging and auditing straightforward
• Controllable costs: Token consumption per path can be estimated precisely
• Lower latency: Eliminates an extra LLM inference call for routing
• Meets security scenario requirements: SOC is a security domain where explainability and traceability are critical

Claude Haiku, Sonnet, and Opus are Anthropic's capability-tier naming for the Claude model family — corresponding to lightweight/fast/low-cost, balanced performance/mid-cost, and flagship reasoning/high-cost respectively. This mirrors OpenAI's GPT-4o mini / GPT-4o / o1 gradient structure. In multi-model architectures, this "capability tier" concept is central to cost optimization: using Haiku-tier models for simple classification tasks can reduce per-call costs to as little as 1/20th of a flagship model. In a high-frequency SOC alert environment, the cumulative cost difference is substantial.

LLM-Driven Dynamic Selection

The second approach is to have the router call an LLM itself, letting the model dynamically decide which model to use next based on the request content. Its appeal lies in flexibility — it can handle edge cases that rules struggle to cover and theoretically enables finer-grained allocation based on semantic complexity.

But the costs are equally significant: every routing decision requires an additional LLM call, adding both latency and cost. More troublingly, it introduces uncertainty — a model used to "decide which model to use" can itself make wrong decisions, and such errors are notoriously hard to diagnose in security contexts.

Recommendations for SOC Scenarios

Given the characteristics of security operations, most experienced architects lean toward a hybrid strategy: rule-based routing as the primary mechanism, with LLM judgment as a fallback.

Start with Rules for Deterministic Paths

A large portion of SOC tasks are known and stable in type. Metadata like alert severity, event type, and data source are usually sufficient to drive a deterministic routing ruleset. Hardcoding this logic saves money, improves stability, and satisfies compliance audit requirements for tracing "why this decision was made by this model."

Introduce LLM Judgment Only in Ambiguous Cases

When rules can't clearly classify a request — for example, when alert content is ambiguous or spans multiple triage levels — fall back to an LLM for a second-pass judgment. This preserves flexibility while keeping uncertainty and added cost to a minimum.

Make Selection Criteria Explicit

Regardless of which approach you use, it's worth explicitly defining the mapping from "task complexity → model capability tier." You can annotate each Agent with its required reasoning intensity, and have the router match it to the appropriate model tier. This "capability tiering + explicit mapping" design is far more robust than letting a black-box LLM make implicit decisions.

Deeper Engineering Lessons

This case reflects a general principle in Agentic system design: if deterministic logic can solve it, don't hand it to an LLM. LLMs shine at open-ended, semantic tasks — but for structured decisions like scheduling and routing, rule engines are typically more reliable and cheaper.

As multi-LLM architectures become the norm, model routing itself is evolving into a standalone engineering discipline. The community has already produced several dedicated model routing frameworks, and the core idea is largely the same: use lightweight classification logic to direct requests to models that match their cost and capability requirements, reserving expensive flagship models for truly complex tasks. For SOC scenarios that must balance cost, performance, and explainability, starting with deterministic rules and introducing intelligent judgment incrementally is the more prudent evolutionary path.

Among the model routing frameworks emerging in the community, notable examples include RouteLLM (open-sourced by the LMSys team) and commercial services like NotDiamond. RouteLLM's core idea is to train a lightweight classifier that predicts "whether a strong model is needed" based on input prompt characteristics — dramatically reducing the proportion routed to strong models with minimal accuracy loss. These tools decouple the routing decision from business logic, forming an independent inference layer. This aligns with the "explicit mapping" philosophy recommended in this article, but offers a more data-driven, automated alternative — a useful complement or upgrade path to pure rule-based routing.