SaaS API Metric Alerting: 4 Node.js Logistics Agent Signals

TL;DR: If you only need to evaluate a single threshold and are willing to own alert delivery yourself, go with custom-metric-based failure alerting. For logistics AI agents, count completed loops, failed loops, and heartbeats, and attribute cost and latency back to the same run. Choose a managed alerting product when paging and escalation must work without writing code.

Why generic API error rates aren't enough

A logistics agent's workflow might be: read a waybill exception, call a model, query a carrier tool, draft a resolution. A single aggregate API error rate can't answer the key questions — are expensive loops more likely to fail? Does a cheap first step mask a costly retry chain?

The genuinely useful unit of measurement is a single agent run. Around that unit, you should emit three classes of metrics: completed_loops, failed_loops, and a heartbeat for each expected work cycle. Bind a stable run ID to each loop in the application and keep the cost and latency it accumulates. This cleanly separates three questions: did the loop finish? How much did it cost? Was the worker actually running?

Keep labels bounded. Carrier, workflow version, and outcome work as dimensions; waybill IDs and free-form error messages don't belong in metric labels, because their cardinality grows with traffic. Put that kind of detail in logs, and follow OWASP's guidance on excluding sensitive data.

The metric label cardinality problem: In time-series systems like Prometheus, every additional unique combination of label values creates a new time series. High-cardinality labels like waybill IDs make the number of series grow linearly with order volume, directly driving up storage cost and query latency — and in severe cases, taking down the monitoring backend. So labels should only use bounded-value dimensions such as carrier, workflow version, and outcome; high-churn information like waybill IDs and free-form error messages should be written to logs and correlated to metrics via trace_id, not stuffed into labels. This is one of the reasons OWASP recommends isolating and filtering sensitive data.

Poll an API contract, not a monitoring vendor

The application shouldn't know how a given vendor expresses a query. Put the translation logic in an adapter and let the alerting policy consume a small result object. I keep this in the product repo, because it encodes product policy: failure rate exceeds the chosen threshold, sample size is large enough to avoid noise, heartbeat is missing.

My first mistake was treating failed_loops as the entire alert condition. Counters only change when code executes. If the scheduler never starts the logistics worker at all, job_failures stays a flat line that looks like health. Zero failures can also mean zero work.

Picture a nightly 2 a.m. reconciliation job: under normal conditions it emits a heartbeat, starts 40 agent loops, completes 38, and reports 2 failures. When the scheduler breaks, none of those data points appear. A query on the failure counter may return the same no-growth result in both a healthy zero-failure scenario and a broken zero-run scenario. So the alerting policy must compare the latest heartbeat against the expected interval, rather than inferring health from a flat counter.

Here's a minimal Node.js decision function showing how the alert policy should land:

type AgentWindow = {
  completedLoops: number;
  failedLoops: number;
  heartbeatAgeMs: number;
  costUsd: number;
};

function decideAlert(window: AgentWindow): AlertDecision {
  const attempts = window.completedLoops + window.failedLoops;

  if (window.heartbeatAgeMs > 10 * 60 * 1_000) {
    return { fire: true, reason: "missing-heartbeat" };
  }

  if (attempts >= 20 && window.failedLoops / attempts > 0.05) {
    return { fire: true, reason: "high-failure-rate" };
  }

  return { fire: false, reason: "healthy" };
}

As an initial review, you can start with a 20-run minimum sample, a 5% failure threshold, and a 10-minute heartbeat age as visible configuration. These are example policy values, not universal defaults — tune them to observed load behavior.

The polling itself can live outside this policy function, because the query's filter parameters may not be declared in API discovery. Check the current schema and response shape, validate, then do the vendor-specific mapping in the adapter. A high-frequency quote agent and a nightly reconciliation agent need different windows and minimum sample sizes.

A cron wrapper calls the reader, evaluates the policy, and hands a de-duplicated incident key to Slack or email delivery. De-duplication matters: the poller may see the same bad window multiple times. Alert on the failure transition, then remind on a deliberate cadence rather than firing on every poll.

Catch the quietest failures with an external heartbeat

A counter can't report that the process responsible for incrementing it never started. That's why the fourth signal must be an externally observed check-in. Metrics need a heartbeat per expected interval, and Healthchecks or a similar dead-man's switch can watch the schedule from outside the process. That beats letting the monitored worker vouch for the fact that it didn't start. Leave ample margin for normal scheduling jitter.

Observe "missing" explicitly, rather than waiting for an error to appear.

Metrics also aren't a substitute for tracing. Logs can carry trace_id and span_id for correlation, but if debugging the full path of every model and carrier call is the task itself, use an OpenTelemetry-compatible tracing backend. Source-map symbolication, crash dump handling, and session replay likewise belong to specialized tools, not this counter-plus-poller design.

Dead-man's switch: This is a "reverse" alerting idea — instead of waiting for a system to actively report an error, the monitored party is required to periodically send an "I'm alive" signal. The monitoring side records the timestamp of each signal, and once no new signal arrives within the expected interval, it declares an anomaly and fires an alert. The key point is "observing absence": it's determined by a component independent of the monitored process, so it can catch silent failures that produce no error logs at all — like a scheduler that never started, or a worker that crashed with nobody to restart it. Services like Healthchecks are typical implementations; leaving margin for normal scheduling jitter avoids false positives.

When to choose something else

Prometheus + Alertmanager wins when you want explicit alerting rules, grouping, inhibition, silences, and notification routing while keeping an open-source metrics model. The cost is operational ownership, unless you use a managed Prometheus service. For a founder, that distinction matters more than the license: time spent tuning and upgrading that stack is time not spent shipping.

Datadog is the stronger choice when you want one managed product covering metric exploration, dashboards, monitors, and notification workflows. It reduces glue code. But before committing, review custom metric volume and organization-wide product adoption, and use current documentation rather than stale price comparisons.

CloudWatch is natural when the agent and its queues are already on AWS: alerts and dashboards sit close to the infrastructure, and there's one less control plane. But its model is less appealing when attribution must span multiple clouds or vendors and the team wants a vendor-neutral application boundary.

Infrai fits a narrower scenario: threshold dashboards and alerts are enough, a poller is acceptable, and keeping the backend vendor swappable under a single REST capability contract has value. The cost is that you own the polling and delivery path. It's not the right fit for native paging, synthetic monitoring, session replay, source-map symbolication, or distributed span trees — those are different jobs.

Choice	Cost attribution	Alert delivery	Operational fit
Infrai metrics	Query custom counters via one REST contract	Build your own poller and Slack/email delivery	Small systems already using its shared backend contract
Prometheus + Alertmanager	Labels and recording rules	Native routing via Alertmanager	Teams willing to run or buy a Prometheus stack
Datadog	Custom metrics, monitors, dashboards	Managed monitors and notifications	Teams that value an integrated managed workflow
Amazon CloudWatch	AWS metrics, alarms, dashboards	Managed alarms and actions	Workloads already centered on AWS

Prometheus + Alertmanager's alerting model: Prometheus scrapes metrics on a fixed cycle and evaluates alert rules with PromQL; once a rule matches, it enters a "pending" state and only transitions to "firing" after it has held for a certain duration (the for clause), avoiding false positives from momentary jitter. Alertmanager then takes over the subsequent routing, grouping, inhibition, silences, and escalation, distributing alerts to different channels. This model is powerful but requires self-hosting and self-operation, covering storage, high availability, and version upgrades; if you don't want that operational burden, you can switch to a managed Prometheus service. By contrast, Datadog and CloudWatch package these capabilities into managed products, at the cost of stronger vendor lock-in.

A release rule that holds up

Write down the service-level decision first: what's the minimum number of agent runs, what's the failure rate threshold, what's the maximum heartbeat age, and who must receive the alert. Then price engineering time, not just ingestion cost.

Use custom polling when the rule fits on one screen and notification delivery is non-critical. Migrate to Prometheus + Alertmanager, Datadog, CloudWatch, or another managed alerting system when the routing policy itself becomes a product. For a solo SaaS, the best observability stack is the smallest one that detects revenue-impacting failures without quietly creating a second platform you have to operate.

One-line Summary

Design four classes of metrics and heartbeat alerting for logistics agents at the single-run level, and pick your stack by operational cost.

Paragraph Summary

The article argues that observability for logistics AI agents should use "a single run" as the unit of measurement, emitting three signals — completed_loops, failed_loops, and heartbeats — around it, and preserving cost and latency attribution. A generic API error rate can't distinguish whether expensive loops fail more often, so it's no substitute. The key insight is that counters only reflect whether code executed: when the scheduler doesn't start, zero failures look identical to health, so an external dead-man's switch must explicitly observe missing heartbeats. In implementation, vendor query translation belongs in an adapter, the alerting policy consumes only a small result object, and it's built from three parts — minimum sample size, failure rate threshold, and heartbeat age — with de-duplication and paced reminders. When grouping, silences, escalation, and notification routing are needed, move to Prometheus+Alertmanager, Datadog, or CloudWatch.