Microsoft agent-hooks CTK Tested: 8 Frameworks in the Demo, 3 Rows in the Compliance Table

Microsoft's agent-hooks launch post claims that the same refund scenario runs the same 20-line decision flow identically across eight frameworks. But the compliance test kit (CTK) and conformance table shipped in the same repo list only three rows. I installed the SDK, ran the CTK myself, and read the conformance files line by line—the gap is wider than the launch post implies.

Running the CTK

The CTK has no CLI entry point. It's a Python module, callable in four lines:

from agent_hooks.ctk import load_vectors, run_vectors
from agent_hooks.ctk.reference import ReferenceHarness

vectors = load_vectors()
results = run_vectors(ReferenceHarness, vectors)
print(len(results), sum(r.status == "pass" for r in results))

**Capability gating** is an easily overlooked but conclusion-determining design in the conformance test kit: a host implementation first declares its supported capability set (e.g., `incremental_output`) in a declaration file; the CTK uses that declaration to filter test vectors, running only vectors within the host's declared scope, marking out-of-scope ones as `skip` rather than `fail`. This mechanism is friendly to implementers—it doesn't punish hosts for not implementing a capability—but the cost is that the "pass rate" metric gets diluted: different denominators yield different results, and a host that doesn't support incremental mediation can never fail on those vectors. For buyers, the right question isn't "what's the pass rate" but "which contract behaviors do the skipped vectors cover, and does my scenario need them?" Reading only the pass-rate number makes it easy to mistake "untested" for "passed."
# 73 69

On agent-hooks-sdk==0.1.0b2 with Python 3.13, the result is 73 vectors, 69 passed, 4 skipped, 0 failed, in 0.37 seconds. No model, no network, no API key needed—the most reproducible part of the entire stack.

The four skipped items are the point, and they're not failures:

AH-CTK-110  skip  incremental mediation: ...
AH-CTK-111  skip  incremental mediation: ...
AH-CTK-112  skip  incremental mediation: ...
AH-CTK-113  skip  incremental mediation: ...

The reason field is missing capabilities: ["incremental_output"]. The CTK is capability-gated—hosts declare what they can do first, and vectors outside that declaration are skipped rather than failed. Remove those four, and the entire streaming/incremental portion of the contract goes untested.

The design is defensible, and the repo says so explicitly. conformance/CLAIMS.md defines "conformance" as passing all vectors in the CTK "applicable to the host's declared scope," and requires hosts with incremental mediation capability to declare incremental_output. But reading as a buyer rather than a spec author, "passes 100% of applicable vectors" and "passes 100% of vectors" sound identical in a press release, and they're completely different things.

The conformance table has only three rows

The full declaration table at the time of this run:

Host	Adapter	Report
reference-agent (in-repo)	0.1.0	my run: 69/73, 4 capability-gated skips
microsoft/agent-framework (Python)	core 1.13.0	47/51 passed, 4 capability-gated skips
agent-control-spec (Rust)	rust@0.1.0-alpha.5	46/47 on the 47-vector corpus, 1 skipped

Three rows. Of the eight frameworks named in the launch post, LangGraph, OpenAI Agents SDK, Semantic Kernel, LlamaIndex, CrewAI, and Claude Agent SDK have no file record whatsoever.

I looked for adapters first, then declarations. The Python package confirms it: grepping the entire agent_hooks package for langgraph, langchain, openai, crewai, llama_index, semantic_kernel, claude yields zero hits. The only host binding shipped on the Python side is create_agent_hooks_middleware in agent_framework. Want to use agent-hooks inside a LangGraph or OpenAI Agents SDK loop today? Write the adapter yourself—the four-method harness interface is just an interface, nobody's written one for you.

The harness interface here refers to the adapter layer between the contract and a concrete host: it maps agent-hooks' interception points (like pre_tool_call) onto the target framework's actual control flow—where to insert a check before a given function call, which state object carries the decision, how exceptions or return values become block signals. The article mentions a "four-method harness interface," meaning the interface itself is tiny, but the real work isn't implementing those four methods—it's figuring out whether the target framework has a stable, insertable seam. That's why adapters are slow to appear: the interface can be copied, but the seam may need upstream framework changes first. Without an adapter, the contract stays in the demo and never lands in a real loop.

ToolNode and the before_tool_call hook are key to understanding the LangGraph issue. LangGraph routes the model's tool calls into a ToolNode node for execution; the model's decision (which tool to call, which arguments to pass) is generated before entering the node, and the node itself carries out the actual execution. A middleware-style before_tool_call hook needs control in that pre-execution gap to inspect, modify, or block the call. LangGraph currently has no stable, contract-defined first-class hook covering that gap, so contracts like agent-hooks that rely on "pre-execution interception" have no landing point. Issue #8102 requests exactly this seam—until it's accepted, any claimed compatibility can only be simulated with wrapper code outside the framework, with no stability guarantees.

Not a scandal, but that's the whole truth

There's nothing dishonest here, and I want to be clear about why.

The launch post says "we ran the same scenario across eight frameworks," and that's true. Microsoft wrote a demo, wired it into eight hosts, and the interception contract is host-agnostic enough that the guard code transfers. What it doesn't say: the SDK actually shipped binds only one. The demo is Microsoft's code running on eight hosts; the conformance declarations are artifacts each host must file a PR for. The repo says it plainly: if you maintain a framework and want your name on a conformance report, the harness interface is only four methods, and Microsoft will help with the integration.

Fair. But six of eight haven't taken that invitation yet.

The repo is equally candid about limits, and I'd rather quote than paraphrase: "A conformance declaration is not a security certification… it does not test adversarial bypass, and it does not guarantee that production code paths match the harness." MAF's declaration also lags a version—it cites core 1.13.0 at commit 4b1afd9052, while 1.21.0 was current when I installed; the declaration is pinned to a 51-vector corpus, but the plugin now loads 73. Declarations age.

The clue is in LangGraph's own issue tracker

To feel the real coverage of "eight frameworks," don't read the launch post—read langgraph#8102: opened 2026-06-16, still Open, unassigned, labeled external.

LangGraph routes the model's tool calls into ToolNode (_run_one / _arun_one) for execution. In practice there's no simple first-class before_tool_call hook that can intercept, inspect, block, or modify a call before execution with a stable contract.

This is precisely the seam the entire agent-hooks model depends on, requested. It's been open for four months with no response. A framework where even the pre-execution tool seam is still at the RFC stage hasn't adopted a contract whose fourth interception point is called pre_tool_call—no matter what the demo showed.

Practical recommendations

Three things, in order:

1. Treat conformance/CLAIMS.md as the list of supported hosts, not the launch post. It's one table, pins commits, and is the repo's own arbiter of "supported."
2. Run the CTK yourself before adopting. Four lines of code, sub-second, and it tells you which parts of the contract your host actually covers. If 4 vectors are skipped, you know which class of questions has no answer yet.
3. If your framework isn't in the table, the honest reading is "the spec is portable, the adapter isn't written." That's a real decision, not a formality—the adapter is where the contract meets your loop's actual control flow.

The contract itself is well done: capability gating, per-participant conformance reporting—more honest than most governance-style releases. The mistake is reading "eight frameworks" as a compatibility claim. It's a demo.

One-line Summary

agent-hooks' "eight-framework demo" is just a demo—the shipped conformance table has only three rows.

Paragraph Summary

The author ran Microsoft agent-hooks' conformance test kit (CTK) locally, getting 73 vectors, 69 passed, 4 capability-gated skips, 0 failures, reproducible without a model or network. The key discrepancy: the launch post claims eight frameworks run the same decision flow, but the repo's conformance declaration table lists only three hosts; six of the named frameworks have no file record, and the Python package contains no corresponding adapters. The four skipped vectors happen to cover "intercept deny after streaming output," the exact question you'd want answered before adoption; MAF's declaration lags a version, and LangGraph's needed pre-execution tool seam remains an unanswered RFC. The author concludes the contract itself is honestly designed and verifiable, but "eight frameworks" should be read as a demo, not a compatibility promise.