Saket

Saket's team added a step counter to the agent state, triggering an alert when a single graph run exceeds approximately 40 steps, a threshold derived from roughly 2× the P99 of legitimate step counts over one week (the most complex legitimate workflow being approximately 16 steps).

Tool servers have low CPU due to handling I/O-intensive tasks, so using CPU as the autoscaling signal incorrectly suggests no scale-out is needed; the correct signals are request queue depth or active connection count, and after Saket's team switched to queue-depth-based autoscaling, scaling lag dropped from approximately 3 minutes to approximately 30 seconds.

Guardrail thresholds should be set to empirical values: run the task distribution in a test environment, observe the P99 step count for legitimate task completion, and set the threshold at approximately 1.5 times that value.

When diagnosing production incidents, one should stop the bleeding first (immediately kill or pause the Agent) before diagnosing the root cause.

After refactoring a routing node from deterministic keyword checks to LLM-based intent classification, each query incurred an additional overhead of approximately 300 tokens, which at their traffic scale amounted to roughly $40 more per day.

Token input/output should be tracked per agent rather than only per request, in order to identify which node is causing problems.

Troubleshooting an agent infinite-loop incident should follow this sequence: check the run step count for that session, the tool call sequence, and the cumulative token expenditure.

In the simulated incident, the research agent ran 214 steps, the document search tool was called once every 3–4 steps, approximately $47 had already been spent, and the root cause was that the planner node over-decomposed the user query and invented sub-questions not asked in the original query.