Century-Old Algorithm Beats SOTA: Have Time Series Anomaly Detection Benchmarks Become Obsolete?
Century-old SPC algorithm beats SOTA on TSAD benchmarks, exposing potential benchmark inadequacy.
A researcher demonstrated that Statistical Process Control (SPC), a 1920s-era algorithm, can beat state-of-the-art deep learning models on the widely-used TSB-AD-M time series anomaly detection benchmark. This raises serious questions about whether current TSAD benchmarks are too simplistic to meaningfully evaluate progress, and calls for more challenging, real-world datasets to drive genuine advances in the field.
A Pointed Challenge to the Time Series Anomaly Detection Field
Time Series Anomaly Detection (TSAD) has become a hot research topic at top-tier conferences like NeurIPS, SIGKDD, and VLDB. A steady stream of deep learning models continues to climb the leaderboards on various benchmarks. However, a researcher recently dropped a bombshell on Reddit: they claimed to have beaten current state-of-the-art (SOTA) methods on mainstream benchmarks using a simple algorithm with a century of history — Statistical Process Control (SPC).
This claim strikes at the heart of the field. If an industrial statistics tool that matured in the 1920s can match or even surpass deep neural networks with millions of parameters, then has the so-called "progress" in TSAD over the past decade been nothing more than an illusion built on fragile benchmarks?
The Core Controversy: Are TSAD Benchmarks Too Easy?
TSB-AD-M Benchmark Under Direct Fire
The author's criticism is squarely aimed at the TSB-AD-M benchmark proposed by Paparrizos et al. — the standard dataset used by the vast majority of TSAD papers to evaluate performance. The author states that after personally testing these benchmark datasets, they found that "in most cases" a simple SPC method could beat SOTA approaches.
In one ECG (electrocardiogram) example they provided, SPC even achieved perfect detection results. They specifically called out dozens of sequences labeled "TAO" in the dataset, arguing these were "even more trivial" for SPC — meaning the anomaly patterns were so obvious that even the most basic statistical thresholds could easily identify them.
What Is Statistical Process Control (SPC)?
For those unfamiliar, SPC is an industrial quality control method developed by Walter Shewhart at Bell Labs in the 1920s. Its core idea is remarkably simple: set upper and lower control limits based on the mean and standard deviation (e.g., the classic 3σ rule), and flag any data point that falls outside this range as anomalous.
A method that can practically be computed by hand should have no business competing with modern deep learning models on the same stage. The very fact that it can win exposes the fundamental problem with the benchmark data: the anomalies are too conspicuous and lack any real challenge.
The Author's Central Argument: The Benchmarks Have Failed, Not the Algorithms
You might not have noticed, but the author maintained considerable rigor alongside their sharp criticism. They explicitly clarified their position:
"To be clear, I am not making any judgments about the algorithms proposed in these papers. But the TSB-AD benchmark is clearly too simple to support meaningful conclusions."
This is an important distinction. They are not attacking the quality of the models researchers have designed, but rather arguing that the yardstick used to evaluate those models has broken down. When a ruler can't distinguish between good and bad, any "performance improvement" rankings derived from it become meaningless.
The author then proposed a thought-provoking criterion: if a century-old algorithm can beat SOTA, then we "probably shouldn't be too amazed by these results." They called on the entire community to engage in serious introspection and stated bluntly: "Most of the progress over the past decade appears to be illusory."
A Constructive Proposal: Building More Challenging Anomaly Detection Benchmarks
Criticism alone is easy; proposing solutions is where the real contribution lies. The author didn't stop at tearing things down — they indicated they had already completed "90% of the work" toward introducing more challenging TSAD problems.
They listed several real-world, more complex application scenarios as candidate directions for new benchmarks:
- Physiological monitoring data from sled dogs
- Behavioral tracking of tuna
- Operational status of fuel cells
- Sensor signals in smart manufacturing
What these scenarios share is that anomaly patterns tend to be subtle, hidden, and deeply entangled with normal data — impossible to separate using simple statistical thresholds. The author also mentioned more advanced time series analysis tools like Pan Matrix Profile, suggesting that truly valuable research should compete on these harder problems.
Broader Implications of This Debate for AI Research
Benchmark Saturation Is a Pervasive Problem in Machine Learning
This challenge is far from an isolated case. Computer vision, natural language processing, and other fields have all encountered the dilemma of "benchmark saturation" — when performance on a dataset is pushed close to its ceiling, marginal improvements often come from tricks that overfit the benchmark itself rather than genuine capability breakthroughs. This debate in TSAD is essentially the same problem replaying in the time series analysis domain.
The Value of Simple Baselines Has Been Chronically Underestimated
This case once again confirms a principle that is repeatedly emphasized yet often ignored: any new method should be compared against sufficiently strong simple baselines. "Old-fashioned" approaches like SPC, moving averages, and naive statistical rules should serve as the first hurdle when evaluating complex models. If a deep model consuming enormous computational resources can't significantly outperform the 3σ rule, its practical value deserves a serious question mark.
A Single-Source Perspective Warrants Cautious Interpretation
It should be noted that the views in this article primarily come from this researcher's one-sided account on Reddit and their supporting materials (including YouTube videos and several slide decks). The author themselves acknowledged they "haven't resolved all the triviality issues." Readers should therefore treat this as an important challenge that provokes thought, not a definitive verdict on the entire field. Whether the TSAD community will seriously respond to this criticism, and how benchmark design will evolve, remains to be seen.
Conclusion
Behind the provocative headline "century-old algorithm beats SOTA" lies a serious question that the entire machine learning community should reflect on: Is the yardstick we use to measure progress itself reliable? When benchmarks are too easy, shuffling positions on a leaderboard may be nothing more than a numbers game, while genuine technical progress gets obscured.
This debate reminds us that something more important than chasing SOTA exists: constantly questioning and updating the benchmarks we rely on for evaluation. Only by building on datasets that are truly challenging and closely aligned with real-world needs can time series anomaly detection research escape the trap of "illusory progress" and move toward creating real value.
Related articles
Tailcat: Tailscale's Official Decentralized Minimalist Networking Solution
Tailcat is Tailscale's official decentralized networking project that strips control plane dependencies, offering self-hosting users a more autonomous, privacy-focused WireGuard mesh experience.
Configuring OpenTelemetry Logs in Rails: From Integration to Production
Learn how to configure OpenTelemetry logs in Rails, covering OTel SDK setup, trace context injection, structured log export, and performance optimization for seamless log-trace correlation.
4DOF Robotic Arm DIY Tutorial: A Progressive Guide from Potentiometer Control to Inverse Kinematics
Complete guide to building a 4DOF robotic arm: from potentiometer control to Python serial communication, inverse kinematics, PyBullet simulation, and vision-based grasping for Arduino robotics beginners.