One Agent, Six AWS Inference Backends: Accuracy, Bandwidth, and Cost Benchmarked

The same model, the same weights, running on different AWS inference backends produced a gap of 36 to 132 tokens/s—and hourly pricing barely explains it. This companion article to the "Inference on AWS" talk uses a single Strands Agent to send the same question to six backends: Amazon Bedrock's managed API, a SageMaker endpoint, vLLM on EC2 g6 and g5g, and Gemma 4 hand-ported to Inferentia2 and Trainium.

The entire experimental analysis rests on just three variables: how many bits the weights are stored in, how fast the chip can read those weights out, and how fully the inference engine can utilize the chip.

The Six Backends at a Glance

The experiment used the Gemma 4 E2B model to ensure the same weights were compared across backends:

Backend	Inference Carrier	Model
Amazon Bedrock	Managed API	us.amazon.nova-micro-v1:0
SageMaker ml.g6.xlarge	AWS vLLM 0.31.0 container	Quantized Gemma 4 E2B
EC2 g6.xlarge (NVIDIA L4)	vLLM 0.30	Same quantized package
EC2 g5g.2xlarge (Graviton2 + T4G)	vLLM 0.27.2rc0, patched for sm_75	Gemma 4 E2B, fp16
EC2 inf2.xlarge (Inferentia2)	Hand-ported, compiled to Neuron	Gemma 4 E2B
EC2 trn1.2xlarge (Trainium)	Same image	Same build

Step 1: Precision Determines What You Give Up

The core trade-off of quantization formats is straightforward: fewer bits per weight means less data to move per token, and smaller memory and disk footprint. But a small format only actually gets faster when the hardware natively supports it; otherwise the weights get "widened" back to 16 bits before computation, and you pay for accuracy loss with slower speed instead.

Looking at 3,880 test records, only two results really deserve attention:

First, the int4 build from Google's QAT (quantization-aware training) checkpoint was less than 0.5 points below bf16 in accuracy (0.678 vs 0.683); the int8 build round-tripped directly from bf16 actually lost 1.3 points. This shows that the key to quantization effectiveness isn't bit width, but whether the weights "knew about" low precision during the training stage.

Second, E2B's embedding table accounts for the vast majority of the weights. Compressing the embedding table to int4 as well dropped the model from 6.32 GiB straight down to 2.85 GiB—and that reduction comes mainly from the embedding table, not the Transformer body.

Native support, in turn, determines whether small formats translate into speed:

Chip	bf16	fp8	int8	int4
NVIDIA L4 (g6/SageMaker)	Supported	Supported	Supported	Widened to 16-bit
NVIDIA T4G (g5g)	Emulated	Unsupported	Supported	Widened to 16-bit
NeuronCore-v2 (inf2/trn1)	Supported	Hardware-supported, compiler pending	Supported	Widened to bf16, 27× slower per call

The difference between quantization-aware training (QAT) and post-training quantization (PTQ): The key distinction mentioned in this article is whether the weights "knew about" low precision during the training stage. QAT inserts fake-quantization nodes into the training forward pass, letting the model perceive during training the rounding error that weights and activations will be compressed to int4 or int8, and gradients adjust accordingly, so the model learns representations robust to quantization. PTQ instead takes a model already trained in bf16 and directly applies round-to-nearest or a more complex calibration to the weights; the model was never optimized for low precision, so rounding error strikes the output distribution directly. This explains why QAT int4 only lost 0.5 points while int8 rounded directly from bf16 lost 1.3 points—a smaller bit width isn't necessarily worse; whether the training stage was involved is the watershed.

Why the embedding table dominates E2B's size: Gemma 4 E2B's "E" stands for effective parameters, i.e., roughly 2B parameters actually activated per token, but the word embedding table needs to store a full vector for the entire vocabulary. Vocabularies typically have tens of thousands to hundreds of thousands of entries, each with dimensions in the thousands, so the embedding table can easily occupy several GiB, far exceeding the Transformer body. Compressing the embedding table to int4 brought the model from 6.32 GiB down to 2.85 GiB, precisely showing that the bottleneck is the vocabulary rather than the attention or FFN layers. This also means quantization strategies must prioritize the embedding, otherwise overall compression gains are limited.

Step 2: Bandwidth Sets the Ceiling, the Engine Determines How Close You Get

Generating each token requires reading all the weights once, so memory bandwidth directly determines the per-request tokens/s ceiling. The efficiency of the inference engine determines how close to that ceiling you can actually run.

Backend	Accelerator	Memory	Bandwidth	Single-request tok/s
g6.xlarge	NVIDIA L4	24 GB	300 GB/s	131.9
SageMaker ml.g6.xlarge	NVIDIA L4	24 GB	300 GB/s	96.5
g5g.2xlarge	NVIDIA T4G	16 GB	320 GB/s (measured 277)	34.9
inf2.xlarge	Inferentia2	2×16 GiB	~817 GB/s	36.5
trn1.2xlarge	Trainium	2×16 GiB	~613 GB/s	36.0
Bedrock	Nova Micro	Undisclosed	Undisclosed	130.4

The most counterintuitive thing in this table is the Neuron series: Inferentia2 and Trainium have 2.0 to 2.7 times the memory bandwidth of the L4, yet their decode speed is only 0.28 times the L4's. The bandwidth advantage is completely eaten by the software stack.

Neuron's memory is 2×16 GiB, with each NeuronCore exclusively owning 16 GiB. The model must either be split across two cores or fit within 16 GiB—a hard constraint for a model like E2B where the embedding table dominates.

Why memory bandwidth determines the decode ceiling: Autoregressive decoding requires reading all model weights from memory once per generated token and multiplying by the current hidden state. Therefore the per-request tokens/s ceiling equals roughly memory bandwidth divided by the model weight bytes, times an efficiency factor. Taking the L4's 300 GB/s and a 6.32 GiB bf16 model, the theoretical ceiling is about 47 tokens/s; if weights are compressed to 2.85 GiB int4, the theoretical ceiling rises to about 105 tokens/s, and the actual measured 131.9 tokens/s indicates that the L4's L2 cache and vLLM's batching/speculative decoding further increased effective bandwidth utilization. Conversely, although Neuron's nominal bandwidth is 817 GB/s, int4 weights are copied into dense bf16 before every matrix multiply, effectively inflating weight size back to 16 bits, completely canceling the bandwidth advantage through this "widening" step.

What "widened to 16-bit" means: When the hardware or compiler doesn't support native int4 tensor cores, loaded int4 weights must first be unpacked and expanded to bf16 or fp16 before entering the matrix multiply units. This unpack-expand process not only consumes extra compute but also makes the actual data volume participating in computation revert to the 16-bit level, and bandwidth and capacity advantages evaporate. T4G's int4 widened to 16-bit and Neuron's int4 widened to bf16 are the same mechanism; while the L4's int4 is also widened, its Ada Lovelace architecture has extra data-path optimizations for int4, so it still maintains fairly high throughput, yet measured results are still worse than native fp8/int8 paths.

Step 3: The Real Cost of the Managed Combination

Bedrock bills per token, and Nova Micro's price is $0.035 per million input tokens and $0.14 per million output tokens. A few lines of code get you connected:

from strands.models import BedrockModel
model = BedrockModel(model_id="us.amazon.nova-micro-v1:0", region_name="us-east-1")

SageMaker requires three API calls—create-model, create-endpoint-config, create-endpoint—with vLLM parameters passed via SM_VLLM_* environment variables. In testing, the endpoint took about 9 minutes from deployment to InService.

Here's the key data: SageMaker runs on exactly the same L4 GPU as g6 with exactly the same weights loaded, yet adds about 0.24 seconds of managed-path overhead per request—87 tokens took 0.660 seconds on g6 versus 0.902 seconds on SageMaker. That latency cost ultimately shows up in per-token cost: SageMaker's $3.24 is roughly double g6's $1.69.

Step 4: vLLM's Best Showing on L4

g6.xlarge runs vLLM 0.30, and enabling tool calling takes just two flags:

--enable-auto-tool-choice --tool-call-parser gemma4

The L4's specs are Ada Lovelace architecture, 24 GB VRAM, 300 GB/s bandwidth, 121 TFLOPS BF16 tensor, 242.5 TFLOPS FP8 tensor, 72W single-slot PCIe. It was ready 16 minutes after launch, with single-request decode at 131.9 tokens/s—the fastest of the five instances.

Step 5: Arm + CUDA, Past and Present

The g5g pairs a Graviton2 Arm host with an NVIDIA T4G, which AWS calls "the first GPU-accelerated Arm instance in a major cloud." But running Gemma 4 on it requires building vLLM from source to support sm_75, plus a shared memory patch, because the arm64 image doesn't include Turing kernels.

Interestingly, the same Arm+GPU combination returned to the desktop this fall. NVIDIA announced RTX Spark on May 31, 2026:

	EC2 g5g	NVIDIA RTX Spark
CPU	Graviton2, Arm	20-core Grace, Arm
GPU	T4G, Turing	Blackwell RTX, 6144 CUDA cores
CPU to GPU	PCIe	NVLink-C2C
Memory	16 GB GPU + 16 GB host	Up to 128 GB unified memory

The aarch64 and CUDA build work forced on g5g back then is exactly the operating foundation for machines like this.

sm_75 and the compilation constraint of Turing kernels: NVIDIA GPU compute capability is expressed as sm_XX; the T4G is based on the Turing architecture, corresponding to sm_75. vLLM's official prebuilt arm64 images typically only include kernels for newer architectures (such as sm_80, sm_86, sm_89), because Arm+GPU instances on the cloud were long limited to the g5g generation. As a result, vLLM must be built from source on g5g with -gencode arch=compute_75,code=sm_75 explicitly specified, otherwise the runtime can't find the corresponding kernel and errors out immediately. The shared memory patch is needed because Turing's shared memory capacity and scheduling characteristics differ from Ampere's, and some vLLM kernels assume larger smem by default, causing out-of-bounds access or a sharp performance drop on the T4G. These historical burdens precisely explain why the g5g, despite not having weak hardware, can only reach 34.9 tokens/s—the software stack was never natively optimized for it.

Step 6: Why Neuron Is Hard to Serve

Inferentia2 and Trainium run the same hand-ported image; testing on inf2.xlarge showed:

• No vLLM distribution can run Gemma 4 on Neuron—the engine is hand-ported and doesn't support tool definitions
• Graphs are precompiled with fixed shapes (512 tokens total, prompt 128); new shapes require about 3.9 seconds to compile
• Memory is 2×16 GiB—a single gather exceeding about 4-5 GB returns zeros without erroring
• int4 weights are copied into dense bf16 before every matrix multiplication, making each call 27× slower than pure bf16

This is why bandwidth is 2-2.7× yet speed is only 0.28×: the chip has enough bandwidth, but the software stack for this model hasn't caught up.

Step 7: One Agent Orchestrating Everything

The experiment's Strands Agent asks a simple question—how many of 11 ids are greater than or equal to 10—and provides a count_ids tool to compute it. Each backend that supports tool calling runs three times:

bedrock    run 1/3  ok   quoted=8  filters=['id >= 10']  1.19s
g6         run 2/3  ok   quoted=8  filters=['id >= 10']  0.4s
g5g        run 1/3  ok   quoted=8  filters=['id >= 10']  2.35s
sagemaker  run 1/3  ok   quoted=8  filters=['id >= 10']  0.63s

All twelve runs answered 8 correctly and all issued the correct filter condition. Why grade the filter condition too? Because a wrong filter condition also returns a precise number with a clean provenance—it looks right.

A second Agent on Bedrock treats the six backends as tools and returns a scoreboard sorted by code:

| vLLM on EC2 g6 (L4) | Paris | 0.151 | 13.2 |
| SageMaker endpoint (vLLM, L4) | Paris | 0.426 | 4.7 |
| Amazon Bedrock | Paris | 0.861 | 2.3 |

Why tool-call grading should look at the filter condition and not just the final answer: In an Agent scenario, the model can choose to answer directly or call a tool. If the model generates a wrong filter condition (e.g., id > 10 instead of id >= 10) but the input data distribution happens to make the result still 8, then an external observer sees a completely correct number and a seemingly reasonable provenance. This "right answer, wrong process" is extremely hard to catch through result validation. This experiment includes the filter condition in grading to verify whether the model truly understands tool semantics, rather than just happening to answer correctly. All twelve outputs were id >= 10, showing that models on these backends are reliable in both tool-call format and semantics.

Step 8: What Each Backend Actually Costs

cost.py pulls on-demand prices from the AWS Pricing API and calculates:

Backend	$/hour	24×7 $/month	$/million output tokens (single request)	$/million output tokens (16 concurrent)
EC2 g6.xlarge	0.8048	588	1.69	0.150
SageMaker ml.g6.xlarge	1.1267	822	3.24	0.291
EC2 g5g.2xlarge	0.556	406	4.43	Not tested
EC2 inf2.xlarge	0.7582	553	5.77	Not tested
EC2 trn1.2xlarge	1.3438	981	10.37	Not tested
Bedrock Nova Micro	—	$0 when idle	0.14	0.14

One key judgment call: if g6 runs all month, it costs more than Bedrock Nova Micro—unless it can sustain an average of 710 output tokens/s 7×24 (calculated at 5 input tokens per 1 output token).

So Which Should You Choose?

• Bedrock: the first choice when its models suffice; nothing to manage, lowest cost per token
• g6 + vLLM: the choice when you need your own weights; quantize from QAT checkpoints, and both speed and unit price are the best among instances
• SageMaker: runs the same container but is billed as a managed endpoint, about twice g6's cost per token
• g5g: the Arm + CUDA path
• Inferentia2 / Trainium: only cost-effective once the software stack supports your model

Key Takeaways

• 🟢 QAT weights converted to int4 differ from bf16 by less than 0.5 points in accuracy; the int4 embedding table compresses E2B from 6.32 GiB to 2.85 GiB
• 🟢 vLLM on g6 decodes fastest, at 131.9 tok/s single-request and $1.69 per million output tokens, dropping to $0.150 at 16 concurrent
• 🟢 All tool-calling-capable backends chose the correct filter condition 12/12 times
• ⚠️ SageMaker adds about 0.24 seconds per request on the same GPU, at roughly twice g6's cost per token
• ⚠️ Bandwidth is the ceiling, the engine determines how close you get: Neuron has 2.0-2.7× the L4's bandwidth but only 0.28× the speed
• ❌ No vLLM distribution can serve Gemma 4 on Neuron; these backends can only run hand-ported versions with fixed shapes and no tool-calling support

Experiment scope note: one rehearsal in us-east-2 (g5g and Bedrock in us-east-1), three timed requests per backend. Memory data comes from a single MI300X, accuracy data from a single TPU v5e. Bedrock ran Nova Micro, which is not the same model as the Gemma 4 builds in the other rows. On-demand prices come from the AWS Pricing API.

The methodology's incremental validation strategy is worth borrowing: get every path working first with the small E2B model—parameters, port forwarding, and tool parsing all fail the same way at any size, while E2B loads on Neuron in about 2 minutes versus nearly 4 minutes for 26B.