RIP Harvest: The End of Distributed Tracing and a Rethink of Observability

A Symbol of an Era: Why Harvest Is Being Farewelled

Renowned distributed systems engineer Aphyr (Kyle Kingsbury) published "RIP, Harvest" on his personal blog, sparking widespread discussion on Hacker News with 216 points and 65 comments. The "RIP" in the title is not merely an emotional expression but a farewell to a monitoring/observability system (or related project) called Harvest. Aphyr has long been known for the Jepsen testing framework, and his judgments on distributed system consistency and observability carry considerable weight in the engineering community. This eulogy-style blog post is therefore not just a product obituary but a retrospective on the design philosophy of monitoring tools.

Since the original page does not provide the full text, this article analyzes—based on the context of the Hacker News discussion and Aphyr's consistent technical stance—why tools like Harvest struggle to sustain themselves in today's environment, and what lessons they offer for observability practices.

Harvest's Positioning: Why It Was Once Worth Remembering

From Aphyr's past technical writing, it is clear that he has never focused on "the more metrics, the better," but rather on whether a tool can honestly reflect system state. Systems like Harvest typically handle metric collection, aggregation, and alerting—the foundational infrastructure of distributed system observation. Their design challenge lies in both handling high-cardinality data and remaining available when failures occur.

Many users in the Hacker News comments recalled Harvest's performance in specific scenarios, arguing that it was more direct and controllable than mainstream alternatives in certain dimensions. Behind this nostalgia is engineers' demand for "tool controllability"—when the monitoring system itself becomes a black box, troubleshooting becomes a second-order problem. Harvest's departure, to some extent, means this demand has been marginalized under commercial pressure.

High cardinality refers to the extremely large number of label/tag combinations carried by a metric—for example, using user IDs, request IDs, or URL paths as label dimensions can rapidly produce millions or even billions of distinct time series. Traditional monitoring systems like Graphite and early Prometheus used inverted indexes or simple key-value stores to manage these series; as cardinality rises, memory usage and query overhead grow superlinearly, eventually overwhelming the entire collection backend. This is precisely the core tension that systems like Harvest face in their design: the richer the label dimensions, the more insight available for troubleshooting, but the harder storage and query costs are to control. Around the high-cardinality problem, the industry later evolved sampling, pre-aggregation, and time-series database architectures that separate columnar storage from inverted indexes (such as Prometheus's TSDB, Thanos, Mimir, etc.), all of which essentially trade off between "dimensional richness" and "query economy." Only by understanding this background can one see the real gap between the architectural assumptions of the Harvest era and today's massive-label scenarios.

Three Dilemmas of Observability as Seen Through Harvest

The Conflict Between Maintenance Costs and Business Models

Open-source or semi-open-source monitoring systems have long faced the dilemma of "many users, few payers." If Harvest could not establish a stable commercial loop, maintainers' investment would be unsustainable. Some in the HN discussion pointed out that such tools are often supported by a small number of core developers, and once the primary maintainers shift their attention, the project quickly stalls. This is not a technical failure but a sustainability failure.

Open source sustainability is a topic that has been repeatedly discussed in recent years: a project is used in production by many companies, yet its maintainers are often just one or two people with no stable source of income. Common business models include open core (core open source, enterprise features paid), hosted cloud services (SaaS), and dual licensing, but monitoring tools are especially difficult because their value often manifests in the operational experience after large-scale deployment, and that experience is hard to carve into billable modules. Once core maintainers leave due to burnout or funding gaps, the project falls into a state of de facto abandonware. The "not a technical failure, but a sustainability failure" mentioned in the HN discussion corresponds precisely to this structural problem. For users, this means that when evaluating an open-source tool, beyond GitHub stars and feature lists, one must also examine its governance structure, funding model, and the number of key maintainers.

The Trade-off Between High Cardinality and Query Costs

The metric dimensions generated by modern distributed systems have expanded dramatically, and the high-cardinality problem leaves traditional collection approaches struggling. The architectural assumptions of the Harvest era are in tension with today's scenarios that routinely require handling massive label combinations. Some commenters noted that although alternatives are more feature-complete, query latency and storage costs have actually increased—showing that new tools are not necessarily improvements in every dimension.

Alert Fatigue and Declining Signal-to-Noise Ratio

In his distributed system testing, Aphyr has repeatedly emphasized the importance of "genuine failure signals." If Harvest centered on alerting, it inevitably had to face alert fatigue: the more rules are added, the more genuinely important signals get drowned out. The discussion in the comments points to this as well—after the tool's demise, teams need to rethink which metrics are truly worth monitoring.

Alert fatigue refers to the phenomenon where, when the number of alerts exceeds a team's capacity to handle them, engineers gradually become desensitized to alerts and even habitually ignore or mute notifications, causing genuinely severe failures to be drowned out. Its causes typically include: alert rules lacking severity tiers, arbitrary threshold settings, and treating "all anomalies" as "requiring immediate human intervention." Industry countermeasures include SLO-based alerting (triggering only when the error budget is being rapidly consumed), alert grouping and inhibition, and clearly distinguishing "actionable" signals from "record-only" signals. Aphyr's emphasis on "genuine failure signals" essentially requires alerts to be tied to the actual impact on users, rather than to arbitrary fluctuations in internal metrics. Re-examining alert rules after Harvest's departure is precisely about treating signal-to-noise ratio as the primary metric of observability efforts, rather than pursuing alert coverage.

Lessons for Engineering Teams

Harvest's farewell should not be read simply as "another project died." It reminds engineering teams to factor maintainability and exit costs into their evaluations: even a powerful tool carries the risk of sudden shutdown if community activity is low and core maintainers are few. Many comments on HN suggest that critical observability pipelines should retain replaceable abstraction layers to avoid lock-in to a single solution.

Moreover, Aphyr's writing style has consistently opposed "using tools to mask a lack of understanding of the system." After Harvest's departure, teams should return to fundamentals: clarify observability goals, streamline metrics, and regularly review alert rules. Tools come and go, but a clear understanding of system behavior is the long-term asset.

Conclusion

The discussion sparked by "RIP, Harvest" is essentially a collective examination of the health of the observability tool ecosystem. The 216 points and 65 comments show that this issue strikes a common anxiety among engineers. Harvest's demise is not an endpoint but a reminder: while chasing new tools, do not forget the commitment to controllability and sustainability.

One-line Summary

Aphyr's "RIP, Harvest" sparks a collective reflection on the sustainability of monitoring tools and the design philosophy of observability.

Paragraph Summary

Aphyr published "RIP, Harvest" to bid farewell to a monitoring/observability system, garnering 216 points and 65 comments on Hacker News. Based on the discussion context, the article argues that Harvest's demise was not a technical failure but a sustainability failure: open-source monitoring tools have long faced the business model dilemma of "many users, few payers," and once core maintainers leave, the project quickly stalls. Meanwhile, the query and storage costs brought by high-cardinality metrics, along with the declining signal-to-noise ratio caused by alert fatigue, constitute three dilemmas of observability practice. The author recommends that engineering teams factor maintainability and exit costs into their evaluations, retain replaceable observability abstraction layers, and return to fundamentals: clarify observability goals, streamline metrics, and regularly review alert rules.