AWS AI Agent Tests Passed but Still Missed the Mark: A Real Experiment Retrospective

Title

AWS AI Agent Tests Passed but Still Missed the Mark: A Real Experiment Retrospective

Meta Description

An AWS AI Agent experiment reveals: generated code passes all tests with correct numbers, but the tests don't cover all requirements. Only by independently computing results first, then reading generated tests, can you judge whether Agent output is truly reliable.

Article Content

Does code generated by an AI Agent passing all tests mean the task is complete? A developer ran a small experiment in a personal AWS lab and discovered: passing tests doesn't mean requirements are fully covered.

Experiment Background: Starting from a Deployment History

Instead of connecting to any production environment, the developer prepared a self-made CSV file. This file recorded 12 deployment checks, each 5 minutes apart, containing fields such as timestamp, health status, expected version, running version, healthy instance count, expected instance count, and active alarm count.

The data is fictional, with no real production incidents as background.

The task itself is not complex: calculate two numbers — the total count of unhealthy checks, and the duration of each completed unhealthy period. The second number requires a clear rule: an unhealthy period begins at the first unhealthy check and ends at the first healthy check after it; if the file ends while still unhealthy, that period is considered incomplete.

Before handing the task to the Agent, the developer deliberately calculated it by hand first. Taking the first unhealthy check at 10:10 as an example, the next healthy check appeared at 10:25, so that period was 15 minutes. This step is crucial: establishing an independent expectation first is the only way to judge whether the Agent's output is truly reliable.

How the Agent Writes and Runs Code

The model used was Amazon Nova 2 Lite, invoked through a managed Agent environment. The developer provided the model with calculation rules, three small test cases, and the target CSV, but gave no implementation code.

The model generated a Python program on its own and sent it to Amazon Bedrock AgentCore Code Interpreter to run in an isolated execution environment. The entire flow can be summarized as:

Provided CSV → Model generates Python → Code Interpreter executes → Results returned

The developer did two things: first, compared the returned numbers with the pre-calculated results; second, read the test code generated by the model. The retained execution trace showed only one Code Interpreter call, containing the generated program, with the tool output containing test pass markers and calculation results. This means a genuine verification is possible: see what actually ran, rather than trusting the model's summary of the run.

Amazon Nova 2 Lite is one of the foundation model series Amazon announced at the 2024 re:Invent conference, positioned as a low-latency, low-cost small-to-medium model suitable for code generation, data analysis, and other response-speed-sensitive scenarios. Bedrock AgentCore is an extension capability of Amazon Bedrock that provides a managed runtime environment for AI Agents, including sandbox tools like Code Interpreter. Code Interpreter executes model-generated code in a container isolated from the main model, preventing scripts from affecting the host system while allowing the Agent to obtain real execution results through tool calls. This "model generation + sandbox execution" architecture is a common form of Agent completing computational tasks today, and is the foundation for retaining execution traces and verifying actual execution content in this experiment.

Numbers All Correct, But Tests Have Gaps

Before the Agent ran, the developer had already independently calculated the expected results, and the output matched completely:

Check Item	Result
Total samples	12
Unhealthy samples	5
Unhealthy sample percentage	41.7%
Completed unhealthy periods	2
Period durations	15 minutes and 10 minutes

41.7% means 5 out of 12 samples were unhealthy, not that the application was down 41.7% of the time. The program also printed MODEL_GENERATED_TESTS=PASS.

The problem lies in test coverage. The developer's prompt required writing assertions for every declared test expectation — that is, checks that would stop the program if expectations weren't met. But the generated tests only covered period counts and durations, without explicitly checking the start and end timestamps of each expectation.

The timestamps in the target output are correct, so the answer itself is not wrong. The gap lies in what the tests actually proved.

"Tests passing" means the assertions that were written passed; it doesn't mean every requirement was tested. Comparing results and reading assertions answer two different questions.

Test coverage typically measures how many lines or branches the test code executed, but the "gap" here is not line coverage — it's requirement coverage — whether every assertion declared in the prompt has a corresponding test. The two are easily confused: even if line coverage reaches 100%, if the tests themselves don't check start and end timestamps, that requirement remains unverified. In AI-generated code scenarios, models tend to write assertions around the most core, most visible outputs (like period counts and durations), while easily overlooking boundary conditions or auxiliary fields. Therefore, reading generated test code is essentially performing requirement traceability, mapping each original requirement to a specific assertion.

What This Run Can and Cannot Show

This experiment demonstrates: the Agent can generate and execute analysis code, and the output matches the developer's independently calculated results; it also demonstrates why reading generated test code is essential.

But it cannot show that every future run will succeed. Malformed inputs, access restrictions, session behavior, and repeated run checks all remain to be evaluated in follow-up work.

For such analysis tasks, the developer recommends keeping three things together: the input, the code that actually ran, and its output. Then, before trusting the "pass" marker, compare the generated tests against the original requirements item by item.

One-line Summary

AI Agent tests passing all green doesn't equal requirement coverage — independently verify and check assertions item by item.

Paragraph Summary

A developer ran an experiment in a personal AWS lab using a fictional deployment CSV: having Amazon Nova 2 Lite generate and execute Python code through Bedrock AgentCore Code Interpreter to calculate the total count of unhealthy checks and the duration of completed unhealthy periods. The developer first calculated expected results by hand, then compared them with the Agent output — the numbers matched exactly, and the program printed a test pass marker. But after reading the generated test code, it became clear that the tests only covered period counts and durations, without assertions for each expected start and end timestamp. Tests passing only means the written assertions passed; it doesn't mean every requirement was tested. The developer therefore recommends keeping the input, the actual code that ran, and the output, and comparing tests against original requirements item by item before trusting the "pass" marker.