Managing Six AWS Inference Backends with a Single MCP Server: A Practical Walkthrough

From Terminal Scripts to an MCP Server: One File That Handles Six Inference Backends

How painful is it to deploy inference services on AWS? Six backends, four or five regions, five billable resource types, two AWS Neuron chip instances, and nodes that are all inaccessible from the public internet — this is the real-world situation many teams face when comparing multi-backend inference setups. Developer xbill9 has open-sourced the aws-inference-strands project on GitHub, packaging this entire workflow into a single MCP server file so that one chat session can handle the full lifecycle: launch, health check, log viewing, inference requests, and teardown.

The server is built on the MCPServer class from MCP Python SDK 2.1.1, negotiates protocol version 2025-11-25, and starts via stdio. It achieves a restrained yet thorough execution of four design principles.

Six Backends, One Chat Entry Point

The inference backends actually running behind this demo include: Amazon Bedrock (managed service), a SageMaker endpoint, vLLM on an EC2 g6 instance, an inference service on a g5g instance, and Gemma 4 models on Inferentia2 (inf2) and Trainium (trn1). The six backends are distributed across up to four regions, with two of them running on AWS Neuron chips — meaning billing, availability zone capacity, and health status all differ.

The original approach involved five shell scripts: stage/up.sh to launch resources, stage/forward.sh to set up port forwarding, stage/check.py for health checks, stage/ask.py to send requests, and stage/down.sh to destroy resources. Each script output information aimed at a person at the terminal. What the MCP server does is expose these operations to an Agent, turning questions like "what's running, what's being billed," "has inf2 started serving yet," and "show me the vLLM logs on g6" into code-verifiable answers.

MCP (Model Context Protocol) is a protocol proposed and open-sourced by Anthropic in late 2024, aiming to establish a unified calling convention between large language models and external tools/data sources. It uses JSON-RPC 2.0 as its message format and supports two transport methods: stdio and Streamable HTTP. The core concepts of MCP consist of three parts: Tools (functions the model can actively call, such as launch_backends in this article), Resources (context data the model can read), and Prompts (preset prompt templates). The server declares the names, input schemas, and annotations of available tools to the client via tools/list, and the client (such as Claude Desktop, Strands Agents, Cursor) then issues tools/call based on model decisions. The protocol version is negotiated through an initialization handshake, with client and server each announcing supported version numbers and taking the newer version that both support. The value of MCP lies in turning "each Agent framework writes its own tool adaptation layer" into "implement once, reuse across multiple endpoints" — which is precisely the prerequisite for this article's ability to serve multiple Agent scenarios with a single Python file.

AWS Neuron is the SDK and runtime stack AWS provides for its self-developed inference accelerator chips Inferentia and Trainium, including a compiler (neuronx-cc), runtime libraries, and integration layers for PyTorch/TensorFlow. Unlike GPUs, Neuron chips require models to first be compiled into NEFF (Neuron Executable File Format) targeting the specific chip architecture before execution — a step that often takes minutes to tens of minutes, with compilation artifacts cached. In this article, inf2 taking 20.4 minutes to pass health checks after launch, with "READY in 473.8s" appearing in the logs, is exactly caused by model compilation and NeuronCore loading. Understanding this helps explain why inf2 exhibits the phenomenon of "port 8080 already open but requests timing out": the service process binds the port first, but the underlying NeuronCore is still loading the compiled model, so requests are queued rather than rejected.

Four Design Principles Hold Up One File

The server's core design isn't about piling on features, but about reuse and encapsulation.

Reuse existing code paths. The ask_backend tool calls the backends.chat_once function — stage/ask.py, stage/check.py, the benchmark, and the second demo all use the same function. count_ids calls demo1_counting.count, which is the same function used for in-process tool calls. This means the numbers you get from the MCP server and the numbers you get from scripts come from the same line of code — no two sets of logic contradicting each other.

Launch and teardown call stage scripts directly. launch_backends starts stage/up.sh, teardown starts stage/down.sh — the MCP path and the manual path described in the docs create and destroy exactly the same resources.

No inbound ports opened. The security group for each instance has no inbound rules. Operations requiring instance access execute shell commands on the machine via SSM Run Command.

Tags as the single inventory. stage/up.sh tags each instance with ManagedBy=strands-demo-run, and the server's lookups in all four regions filter based on this tag. Even if a backend drifts to another region due to capacity errors, it can still be found.

Since boto3 is synchronous, each call runs in a thread pool via asyncio.to_thread, allowing one tool to issue requests to four regions simultaneously:

async def _instances() -> list[dict]:
    per_region = await asyncio.gather(*[_region_instances(r) for r in REGIONS])
    return [i for found in per_region for i in found]

SSM Run Command is an AWS Systems Manager capability that allows dispatching shell commands to EC2 instances and retrieving stdout without opening any inbound ports or requiring SSH keys. It works by having the pre-installed SSM Agent on the instance proactively establish a long connection to the SSM service (outbound long polling); commands are pushed from the service to the Agent, then executed locally on the instance. This pattern is naturally suited to zero-inbound security group scenarios and is the technical foundation that makes this article's "no inbound ports opened" principle tenable. The trade-offs: when the Agent isn't registered, the call immediately returns InvalidInstanceId (which is one of the signals used in this article to determine whether an instance is ready), and command output has size and timeout limits, so long output must be truncated or written to disk.

MCP Tool Annotations is the protocol's mechanism for declaring behavioral characteristics of tools. It currently defines four boolean fields: readOnlyHint, destructiveHint, idempotentHint, and openWorldHint. Clients can use these to differentiate prompts in the UI — for example, popping up secondary confirmation for tools with destructiveHint=true, or skipping approval for tools with readOnlyHint=true. It must be emphasized that annotations are voluntary declarations; the protocol does not enforce validation, and servers can absolutely mislabel them. Therefore, the reliability of annotations depends on the implementer's discipline. This article uses them as "safety signals" rather than "safety boundaries" — a pragmatic and correct positioning.

Read/Write/Destructive Tiers Across Eight Tools

tools/list returns eight tools, each with MCP tool annotations, so clients can distinguish inventory queries, cost-incurring operations, and deletion operations before calling. Annotations fall into three categories: READ_ONLY (read-only and idempotent), WRITE (non-destructive writes), DESTRUCTIVE (destructive operations).

Tool	Annotation	Scope
list_backends	Read-only	EC2, SSM, SageMaker queries across four regions
backend_health	Read-only	One chat request, EC2 via SSM
backend_logs	Read-only	Service logs, via SSM
ask_backend	Read-only	One chat request, via port forwarding
count_ids	Read-only	Demo 1's in-memory IDs
launch_log	Read-only	stage/up.log or stage/down.log
launch_backends	Write	stage/up.sh, begins billing
teardown	Destructive	stage/down.sh, terminates all resources

list_backends completes five AWS queries in 0.78 seconds in a single call (one EC2 query per region across four regions, plus one SageMaker query). Each running instance adds one additional SSM query. The ask_backend_configured field it returns has a noteworthy detail: even when all five backends are offline, it still lists all six backend names — because it reads the URLs configured in the demo.env file, and that file persists after port forwarding ends. The tool description also makes this explicit.

How Launch, Health Checks, and Logs Actually Behave

launch_backends accepts names like g6, inf2, trn1, g5g, and sm, validates them, and then launches stage/up.sh in the background. The script selects an AMI, tries availability zones one by one, falls back to the next region on failure, and writes operations to stage/up.log.

Measured time from launch to serving: g6 passed health checks 12.2 minutes after the launch call, and inf2 took 20.4 minutes. After inf2's first successful pass (subsequent requests completed in 0.27 seconds), g6's first answer took 1.21 seconds, with the next three taking between 0.02 and 0.03 seconds. The Triton kernel JIT compilation warnings in the g6 logs explain the latency spike on the first answer.

Health checks execute curl on the instance via SSM; requests never leave the instance, so neither port forwarding nor inbound rules are needed. Failure replies precisely reflect problems at different stages: InvalidInstanceId means the SSM Agent hasn't registered yet; a millisecond-level return of 000 means the port is closed; a return of 000 after 90 seconds means the port is open but requests are timing out — a phenomenon that only occurs on inf2, because its server opens port 8080 while NeuronCores are still loading the compiled model.

backend_logs selects the correct log command based on backend type: docker logs for g6's vLLM container and inf2/trn1's Neuron containers, journalctl -u vllm.service for g5g. It falls back to cloud-init logs when containers don't yet exist. inf2's logs show "READY in 473.8s" — the model took nearly 8 minutes to actually become ready.

Triton kernel JIT compilation latency: Inference frameworks like vLLM rely on high-performance kernels written in OpenAI Triton. These kernels are typically distributed as source code and are just-in-time (JIT) compiled by Triton when first executed on the target GPU architecture, then cached locally. The compilation process involves PTX generation, ptxas assembly, and cubin loading; a single kernel may take tens of milliseconds to several seconds, and a model's forward pass may involve dozens of kernels — so a second-level latency spike on first inference is normal. In this article, g6's first answer taking 1.21 seconds and the next three dropping to 0.02–0.03 seconds is a direct manifestation of this mechanism. If you want to eliminate the cold-start spike, you can do a "warm-up inference" during deployment to compile all kernels into TRITON_CACHE_DIR, so subsequent requests hit the cache directly.

Demo 1: Cross-MCP vs. In-Process Comparison

Demo 1's question is to count how many of eleven IDs are greater than or equal to 10; the correct answer is 8. count_ids accepts a predicate operator and value, returns the exact count, minimum, and maximum, and echoes the executed predicate. For example, >= 10 returns 8 and > 10 returns 7.

demo1_counting.py --tools mcp exposes this tool to a Strands Agent via the MCP protocol. Strands' MCPClient starts the server via stdio, and the tool filter exposes only count_ids. Because the tool runs in the server process, the demo can't record calls in real time; instead, it reads tool call records from the Agent's conversation messages and scores them using the same function.

Results from 20 runs each on Bedrock: both the in-process tool and the MCP tool achieved 20/20 correct, 20/20 used the correct filter, zero errors, and a median of 1.07 seconds. The cost of starting the server occurs only once per demo run and isn't counted in the timed Agent calls.

Engineering Details Worth Noting

Every tool catches exceptions and returns a single line of text starting with ❌, containing the AWS error code and message, a ValueError for invalid backend names, or a RuntimeError naming the four regions searched. But there's a caveat: the protocol-level isError flag is false in all failure replies. If a client judges success or failure based on this flag, it would treat failed health checks as successes. The original article suggests setting it when clients actually rely on this flag.

Numeric clamping is also well done: tail is clamped between 1 and 2,000 lines, max_tokens between 1 and 1,024, and backend is validated against a fixed list before any AWS call. A model request for a million lines of logs only gets 2,000 lines.

Long-running tasks execute detached in the background, with start_new_session=True ensuring scripts survive after the MCP session closes, and output written to log files for launch_log to read.

One Unresolved Gap

The server doesn't open any port forwarding, so ask_backend and the demos still require running stage/forward.sh from the terminal. Additionally, the demo.env file's lifecycle outlives the port forwarding's, causing ask_backend to list backends that are actually offline until port forwarding runs again. This is a side effect of the design choice to reuse existing files.

Overall, this project demonstrates a pragmatic pattern for exposing complex multi-backend infrastructure operations to AI Agents: rather than reimplementing a set of logic, wrap existing, validated code paths, convey safety signals through tool annotations, and maintain resource inventory through tags rather than state files. These design choices have direct reference value for teams that need Agents to operate cloud resources in production environments.

One-line Summary

An open-source project wraps the full lifecycle operations of six AWS inference backends into a single MCP server file, validating a pragmatic pattern for Agents operating cloud resources.

Paragraph Summary

This article introduces the GitHub open-source project aws-inference-strands, which uses a single MCP server file to uniformly manage six AWS inference backends — Amazon Bedrock, a SageMaker endpoint, vLLM on EC2 g6, a g5g inference service, and Gemma 4 models on Inferentia2 and Trainium. The server is built on the MCP Python SDK, starts via stdio, and exposes eight tools with three-tier read-only/write/destructive annotations, replacing five previously scattered shell scripts. Four core design principles: reuse existing code paths, call stage scripts directly, open no inbound ports (everything goes through SSM Run Command), and maintain resource inventory through tags rather than state files. Measured results show g6 passing health checks 12.2 minutes after launch, inf2 taking 20.4 minutes; list_backends completing four-region parallel queries in 0.78 seconds in a single call; and Demo 1 achieving 20/20 correct with a median of 1.07 seconds across 20 runs each on both MCP and in-process paths, proving the encapsulation introduces no additional overhead. The article also points out two unresolved gaps: the isError flag not being set and the demo.env file lifecycle issue.