AirOps in Practice: Building Enterprise-Grade AI Agents with the Claude Agent SDK

From Workflows to Agents: AirOps's Evolution

One of the central challenges in productizing AI has always been this: how do you make AI agents genuinely usable by non-technical users? In a technical talk, Dylan from the AirOps product team gave a deep dive into how they transitioned from a traditional workflow builder to an agent architecture powered by the Claude Agent SDK.

AirOps is a growth marketing platform focused on AI Search — essentially the evolution of traditional SEO in the age of generative AI, academically known as Answer Engine Optimization (AEO). Where traditional SEO optimizes a webpage's ranking in Google results, AEO targets how often generative engines like ChatGPT, Perplexity, Gemini, and Claude cite and recommend a brand when answering user questions. Instead of presenting a list of links, these engines generate natural language answers — so brand visibility shifts from "which page do you appear on" to "does the model actively cite you?" When a user asks Claude "I want to buy these sunglasses," how does a brand ensure it shows up in that answer? AirOps helps brands understand their AI search performance (measured through new metrics like "citation rate" and "share of voice"), identify gaps, take action, and measure results.

In its early days, AirOps relied heavily on a node-based workflow builder — a drag-and-drop design similar to n8n — that chains API calls, data transformations, and conditional logic into a visual pipeline, where each "node" represents an operation and data flows between nodes along directed edges. This approach had an obvious complexity ceiling: you had to teach content marketers what Liquid template language was, what JSON data structures looked like. Making matters worse, these workflows had short lifecycles — whenever Claude released a new model version, the underlying input/output format could drift semantically, forcing customers to update every step. Change step 1, and you might not realize step 20 references a related variable — a classic case of cascading dependencies.

Three Architectural Attempts: Why They Chose the Claude Agent SDK

AirOps's goal was clear: lower the barrier for content marketers while maintaining quality standards. Brands don't want to produce "AI slop" — they need governance mechanisms to uphold enterprise standards. The team went through three architectural iterations to get there.

Attempt 1: Document-Compiled Workflows

The first attempt involved taking a Google Doc-style document and having an LLM compile it into a workflow in the background. The idea was interesting but brittle — too many error-prone nodes, with high ongoing costs to debug and fix.

Attempt 2: Traditional Agent Orchestration Frameworks

Next, they turned to traditional frameworks (like LangChain or AutoGen) that define nodes and let an LLM determine the decision path. These frameworks hard-code decision logic in Python or YAML, with fixed node routing that requires code changes and redeployment to modify. Output quality was acceptable, but flexibility was poor — every time they wanted to adjust how sub-agents were orchestrated, they had to change code and update routing.

Attempt 3: Claude Agent SDK

The real turning point came with the release of Opus 4.5. This version brought significant improvements in tool-calling precision and instruction-following — enough for the team to trust the model's capabilities without worrying it would break established standards. They went all-in on the Claude Agent SDK.

The key difference between the Claude Agent SDK and traditional orchestration frameworks lies in its design philosophy: it lets you describe agent behavior in natural language (Markdown files), externalizing orchestration logic into human-readable documents rather than hard-coding it in program logic. This mirrors the evolution from imperative configuration to declarative description — similar to how infrastructure-as-code transformed DevOps. The most exciting realization: you can orchestrate agents using nothing but Markdown files, shaping behavior by manipulating the environment and harness (the runtime framework) to provide skills and context, without hard-coding implementations. This dramatically accelerates iteration and lets non-engineers participate directly in tuning agent behavior.

AirOps Next: Quill, Playbooks, and Real-World Results

Built on this new architecture, AirOps launched AirOps Next, centered on two core components:

• Quill: An "agent captain" designed for content marketers, with access to AI search data, brand context, and a Brand Kit. It follows content creation instructions and helps users move directly from dashboard insights to action.
• Playbooks: A new building experience. For developers, it's essentially a "Skill" that supports collaboration, governance, and version control — making skills truly accessible to everyone.

The results have been striking. In a case study with Parallel, a client achieved a 130% increase in citation rate and a 42% increase in share of voice, going live within one week. Previously, with the workflow builder, enterprise clients typically needed a month of back-and-forth iteration. One client described the agent as feeling "more like a mid-level strategist on the team."

Friction Point 1: Infinite Use Cases Force Intentional Focus

Dylan made a sharp observation: the power of agents is itself a trap. The first time you use Claude Code or an agent, it's easy to spiral into "it can do everything" thinking. The real challenge is consciously narrowing your focus to the problems you actually need to solve.

For content marketing workflows, the team ultimately locked in on two critical areas.

A Document-Style IDE

This is the Playbooks view. Marketers are intimately familiar with documents — they've used Google Docs for years, and a document interface feels natural in a way that node-based workflows never do. This reflects a core product philosophy: the abstraction level of a tool must align with the mental model of its target users. At the same time, the team preserved the transparency that users valued most in the workflow builder — even in document form, users can clearly see what tools were used at each step, what context was fed in, and retain full control over the entire set of instructions.

Mandatory Human Review

Dylan considers this one of AirOps's most distinctive design choices, grounded in the AI safety concept of Human-in-the-Loop (HITL). Most coding agents run to completion and then go through PR review — a post-task audit. AirOps instead weaves governance, configurability, and accountability into the running process itself, preserving human intervention capability at key decision points.

In the actual product, users can assign a reviewer at the end of each section. When the agent reaches that step, it triggers a tool that blocks the workflow — similar to a "Semaphore" in distributed systems. Only the designated "gatekeeper" can unlock and proceed; others can leave comments but cannot approve. Review tasks are presented through two views — an Inbox and a Grid — where the latter supports large-scale parallel skill runs, with each row corresponding to a specific task, keeping accountability chains clear even at high throughput.

Friction Point 2: Consistency — Guaranteeing Output Quality Through Harness Engineering

The biggest concern in moving from workflows to agents is output consistency. AirOps's Sales VP offered a brilliant analogy: an agent is like a car. The model (Claude Opus or Sonnet) is the engine, but everything built around the engine — the harness engineering — determines how the car actually performs. The team focused intensely on two areas: tool design and context orchestration.

Purpose-Built Tools Over Raw Tool Stacks

Early on, the team gave the agent a pile of raw tools (traffic data, citation data, crawlers, etc.) and let it analyze page issues on its own. The agent frequently went on "exploration adventures," with token consumption ballooning.

The solution was to build purpose-built tools: consolidating operations that Claude repeatedly needed to perform into a single interface. For example, given a URL, one tool immediately returns complete information about the page, a structured content gap comparison, target keywords, and target prompts. This is similar to the "code mode" mindset — using a single piece of code to fetch exactly the context needed in one call, rather than bouncing between multiple tool calls.

Sub-Agents: Isolating Context to Preserve Focus

Dylan emphasized that even though Opus 4.7 supports million-token context windows, "having a large window doesn't mean you should fill it." Context rot remains a serious problem. This is well-supported by research: experiments show that models distribute attention unevenly across context positions, with large swaths of middle content often being overlooked (the "Lost in the Middle" effect). Furthermore, noise introduced early — stale search results, rejected drafts, intermediate computation steps — occupies context space and skews the model's judgment on the current task.

The team therefore introduced multiple specialized sub-agents, essentially decomposing complex tasks into focused, short-context tasks where each sub-agent sees only the information relevant to its own responsibility:

• Compliance-checking sub-agent: Independently evaluates whether content meets brand guidelines, returning a score and a list of issues — without polluting the main context.
• Writing sub-agent: Has a focused context window, free from the noise of research data or old compliance check results.
• Brand Kit sub-agent: Fetches all relevant brand context at the start of a run, stores it as an internal artifact, and the main loop references that artifact throughout — avoiding repeated fetches that create inconsistency.
• Custom sub-agents: Created by solutions architects on demand for flexible extensibility.

Quantified Results

This optimization delivered measurable gains: purpose-built tools reduced token consumption in specific stages by 8%; replacing 20 tool calls with a single entry point dramatically improved response speed; and on the quality side, 10 enterprise customers achieved self-serve content publishing within two weeks — something that previously required hands-on onboarding for each client.

What's Next: Self-Improvement and Content Benchmarking

Dylan shared two new friction points the team is actively working through:

Self-improvement and feedback loops: How do you build "dreaming sequences"? How do you structurally summarize run traces and collect the most relevant memories? He made a particularly insightful observation: forgetting is itself a feature. The ability to actively discard certain memories is just as important as the ability to retain them — an insight that resonates with neuroscience research on forgetting as an active cognitive process.

Benchmarking content creation agents: This is one of the hardest open problems in both NLP research and product engineering today. Code generation has objective benchmarks like HumanEval and SWE-bench grounded in "did it run correctly?" But content creation is full of taste judgments and subjective opinions — there's no clear right or wrong answer. Academic approaches include reference-based metrics (ROUGE, BERTScore) and LLM-as-Judge methods, but the former requires high-quality reference samples and the latter risks bias from evaluator-evaluated homogeneity. AirOps's more practical challenge is: after every harness adjustment (adding a sub-agent or modifying a tool), how do you build repeatable regression tests to ensure system-level changes don't cause silent quality degradation — rather than just going by "gut feel"?

Conclusion: Always Chasing the Moving Friction Point

Dylan's core insight is one that every AI product builder should sit with: when Claude makes execution easy, it's tempting to think there are no more problems to solve — but every problem solved just moves the friction point somewhere new. From node-based workflows to declarative agent orchestration, from raw tool stacks to purpose-built tool design, from infinite use cases to intentional focus, from post-task audit to in-process governance — every evolution at AirOps is a perfect illustration of this truth. It is precisely this relentless pursuit of friction points that enables truly production-grade AI agents to be built — agents that can leave the technical sphere and genuinely serve non-technical users like content marketers. That may be the most real, and most valuable, lesson in bringing AI applications to production.

Key Takeaways