AI-Assisted Test Failure Analysis: Treating LLMs as a Diagnostic Layer, Not a Self-Healing Tool

Automated tests run fast, but what really eats up engineers' time is the investigation process after a test fails. Playwright can efficiently execute thousands of end-to-end tests, but detection is only the first step — the real workload lies in diagnosis: why it failed, collecting logs, inspecting network requests, and reproducing edge cases locally.

A study published on the GeekyAnts tech blog, From Test Failure to Root Cause: Building an AI-Assisted Playwright Failure Analysis Pipeline, proposes an approach worth paying attention to: letting an LLM serve as an analysis layer for test failure artifacts, rather than having AI automatically modify test scripts.

Below is a critical breakdown of this architecture, including what makes sense and where engineering teams need to be cautious during implementation.

Architecture Breakdown: Deterministic Execution vs AI Reasoning

The core premise of this research holds up: keep test execution deterministic, and hand the work of diagnostic synthesis to AI.

The pipeline roughly follows this chain:

Playwright Execution (Pass/Fail) 
       │
       ▼
Artifact Generation (Traces, Screenshots, Network Logs)
       │
       ▼
Data Sanitization & Redaction Layer
       │
       ▼
Structured Failure Object Generation
       │
       ▼
AI Analysis & Root-Cause Classification
       │
       ▼
Human Engineer Review & Validation

The most critical design decision here is: rather than throwing raw logs directly at the LLM, the Playwright trace viewer artifacts are first parsed into structured payloads containing console errors, failed API status codes (e.g., POST /api/payment -> 500), durations, and failure context.

The AI then classifies the problem into one of five categories:

• Application bug
• Test flakiness or locator failure
• Network infrastructure failure
• Test data issue
• Environment instability

This classification granularity isn't particularly fine, but it covers the most common failure patterns in CI, enough for engineers to know roughly which direction to investigate before opening the trace viewer.

Why "Self-Healing Tests" Is a Dangerous Trap

Many commercial AI testing solutions tout "self-healing" capabilities: automatically rewriting assertions or locators when tests fail. This research correctly points out the operational risk — if the application has actually regressed and the AI model misjudges it as an expired locator, automatically modifying the test is equivalent to masking a production bug.

Restricting AI to a purely diagnostic, advisory role is what preserves the credibility of the test suite. Once tests can modify themselves, they can hardly serve as a source of quality signals.

Playwright Trace Viewer is a built-in debugging tool in Playwright that records a test execution as a replayable timeline, including DOM snapshots before and after each action, console output, network requests and responses, and millisecond-precise timing information. After a test fails, engineers typically need to manually open the trace file and replay frame by frame to locate the problem. One of the key improvements in this research is: instead of having the LLM face raw trace binary files or unstructured logs directly, a program first parses the trace to produce structured failure objects, which are then handed to AI for reasoning. This reduces token consumption while making the model's input boundaries more deterministic, reducing erroneous attributions caused by missing context.

Critical Assessment: Where the Architecture Needs Hardening

This pipeline can indeed reduce Mean Time to Diagnosis (MTTD), but from an engineering management perspective, there are several implementation-level challenges that need to be resolved before production deployment.

Data Sanitization and Security

Test logs, DOM snapshots, and network traces often contain sensitive payloads, OAuth bearer tokens, PII, or internal infrastructure URLs. Passing raw context to external LLM providers means significant compliance risk. Implementing a strict regex- and AST-driven sanitization layer before model ingestion is a must, not an option.

This point is most easily underestimated in actual implementation. Many teams get the pipeline working with test environment data during the PoC phase, but run into compliance issues when switching to a staging environment containing real user data.

PII (Personally Identifiable Information) refers to data that can be used to identify a specific natural person, such as names, email addresses, phone numbers, ID numbers, IP addresses, etc. Test environments often inject simulated user data to approximate real scenarios, and staging environments may even contain incompletely sanitized samples imported from production. When this data enters external LLM providers along with traces, screenshots, or network logs, it may trigger compliance obligations under GDPR, CCPA, or China's Personal Information Protection Law. The "AST-driven sanitization" mentioned in the article refers to using abstract syntax trees to parse HTML/DOM or JSON structures and precisely filter sensitive fields by node type, which is more effective than pure regex at avoiding both leaks and false positives.

Hallucination and the "UNKNOWN" State

LLMs are naturally inclined to give answers, even when faced with incomplete data. If what Playwright captured was an assertion timeout caused by a silent backend deadlock, the AI might hallucinate a conclusion of "locator failure."

The system prompt must explicitly enforce: when evidence is insufficient, output an UNKNOWN classification and route to a human owner, rather than providing a misleading wrong direction. This is reflected in the architecture diagram as the final step, "Human Engineer Review & Validation," but the prerequisite is that the AI must honestly admit what it doesn't know.

MTTD (Mean Time to Detect / Diagnose) is a metric commonly used in SRE and DevOps, referring to the average time from when a problem occurs or a test failure is recorded to when an engineer clearly identifies the root cause. In this scenario, MTTD measures the total time a team spends digging through CI logs, replaying traces, and comparing code commits. The premise of introducing an AI analysis layer is that it can significantly compress this metric, rather than merely replacing output with seemingly professional classification labels. The article emphasizes measuring baseline MTTD first precisely because without a baseline, you cannot distinguish between "AI actually accelerated diagnosis" and "engineers just repeated the same investigation in a different way."

Practical Recommendations: Measure Baseline MTTD Before Introducing AI

If developers in an engineering organization spend a large amount of time digging through CI logs, then introducing an analysis pipeline based on Playwright artifacts will deliver immediate returns.

But before implementation, there is one thing worth doing first: measure your current baseline MTTD. Without baseline data, you cannot determine whether the AI analysis layer actually compresses the debugging cycle.

In concrete implementation, three elements constitute the minimum viable version of this solution:

1. Structured evidence collection: Parse traces, screenshots, and network logs into a unified format, rather than having AI interpret unstructured text.
2. Strict payload sanitization: Complete sensitive information filtering before entering the LLM.
3. Human-in-the-loop validation: AI provides classifications and suggestions, engineers make the final judgment, and test scripts themselves are not automatically modified.

The essence of this approach is placing AI in a "diagnostic assistant" position rather than an "auto-repair" position. For teams already using modern testing frameworks like Playwright, this is a relatively controllable-risk, clearly beneficial AI implementation scenario.

One-line Summary

GeekyAnts proposes letting LLMs only diagnose Playwright failures rather than auto-modifying tests, with controllable risk.

Paragraph Summary

GeekyAnts' research proposes an AI-assisted Playwright failure analysis pipeline whose core design restricts AI to a diagnostic role rather than an auto-repair role: Playwright maintains deterministic execution, and the produced traces, screenshots, and network logs are first sanitized and structured before being handed to an LLM for classification into one of five categories — application bug, test flakiness, network failure, test data, or environment instability — with final validation by an engineer. The article argues that "self-healing tests" are a dangerous trap, because if AI misjudges a real regression as an expired locator and automatically rewrites the test, it masks a production bug. Before implementation, two issues must be resolved: sensitive data sanitization and enforcing the model to output UNKNOWN when evidence is insufficient. The article also recommends measuring baseline MTTD first to validate benefits.