Slack's Agentic Testing in Practice: How AI Agents Tackle the E2E Test Flakiness Problem

The Age-Old Problem with End-to-End Testing

In modern Continuous Delivery environments, end-to-end (E2E) testing has always been a love-hate affair for engineering teams. It validates the full system chain — from the UI down to backend services — ensuring software works correctly in real-world scenarios. But E2E tests are also the most brittle and expensive to maintain.

Slack's engineering team recently proposed a method called Agentic Testing, which targets exactly this pain point. As they've shared, E2E test failures often don't stem from actual functional regressions — they're caused by UI element changes or API interface updates. In other words, the test breaks, but the product actually still works. These "false positive" failures consume enormous amounts of engineering time.

The Cascading Cost of Flaky Tests

In large distributed systems, this problem is amplified further. A single frontend button ID change or an API field rename can cause hundreds or thousands of test cases to fail. Engineers are forced to spend significant time distinguishing between "real failures" and "stale test scripts." This maintenance overhead grows exponentially with system complexity, ultimately slowing down the entire delivery pipeline.

What Is Agentic Testing?

The core idea behind Agentic Testing is to introduce AI Agents into the E2E testing stack, giving tests a degree of "adaptive" capability — improving resilience in the face of dynamic change.

Traditional automated tests are hardcoded: scripts explicitly specify which element to click, which text to match, and which return value to assert. The moment any of these anchors change, the script breaks. With AI agents in the mix, tests no longer rely rigidly on fixed selectors. Instead, they can complete test objectives based on an understanding of page semantics and business intent — much like a human tester would.

A Paradigm Shift: From "Executing Scripts" to "Understanding Intent"

This represents a paradigm shift from executing instructions to understanding intent. For example, when a "Submit" button changes from blue to green and moves from the bottom-right corner to the top-right, a traditional script fails due to a locator mismatch. An AI agent with semantic understanding, however, can recognize "this is still a submit action" and continue the test flow rather than simply reporting a failure.

The real value here is that it frees engineers from tedious script maintenance, allowing tests to stay focused on the fundamental question — "is the feature actually working?" — rather than getting bogged down in "can the script still run?"

Where AI Agents Fit in the Testing Stack

Agentic Testing isn't about completely replacing existing testing infrastructure. It's about exploring where AI agents should sit within the E2E testing stack and what role they should play.

Based on Slack's engineering approach, agents are best suited for handling steps that require dynamic judgment and fault tolerance, while stable, high-certainty core assertions can still use traditional methods. This layered design leverages AI's flexibility while maintaining reliability for critical validations.

Balancing Resilience and Determinism

Introducing AI agents brings its own challenges. A core value of testing is determinism — the same input should produce the same result. Because AI agents exercise autonomous judgment, they can introduce non-determinism or even incorrect decisions. Finding the right balance between "resilience" and "determinism" is therefore critical to whether Agentic Testing can truly succeed in production.

One practical design principle: let the agent handle how to achieve the goal (e.g., how to locate and click the submit button), while engineers explicitly define what the goal is and what success looks like (e.g., the expected success state after submission). This preserves clarity of intent while giving the execution layer enough flexibility to adapt.

What Agentic Testing Means for Engineering Practice

Slack's exploration reflects a broader trend: AI agents are gradually moving beyond chat assistants and code generation, penetrating into the infrastructure layer of software engineering. As the gatekeeper of software quality, making testing more intelligent carries profound implications.

For teams struggling with brittle E2E tests, Agentic Testing offers a compelling direction: rather than continuously pouring human effort into patching fragile scripts, build a testing system that is inherently capable of adapting to change.

Proceed with Careful Evaluation

That said, this technology is still in its early stages. The operational cost, execution speed, judgment accuracy, and explainability of AI agents are all factors that must be carefully considered before real-world deployment. Test failure causes need to be clearly traceable, and AI-based decisions can sometimes introduce a "black box" problem.

Overall, Agentic Testing represents a pragmatic direction for technical evolution — not trying to upend everything with AI, but precisely embedding AI capabilities into the most painful bottlenecks to improve the overall resilience of large distributed systems in continuous delivery. As more engineering teams experiment and share feedback, this model is likely to mature and become an important component of the next generation E2E testing stack.