AI-Node-Sentinel
An open-source scaffold for GPU cluster fault triage. Its workflow includes telemetry ingestion, XID event extraction, table lookup and completion, triage rule application and structured report output, with reserved integration points for LLM inference.
Timeline (last 90 days)
The open-source project AI-Node-Sentinel is at version v0.1, and its triage decision tree is a deliberate placeholder implementation that returns NEEDS_HUMAN_REVIEW and a confidence of 0.0 for every event.
AI-Node-Sentinel's LLM integration point is in agent/llm.py with the protocol reason(system_prompt, evidence) -> str, and the orchestrator only appends the LLM's returned text with an [LLM] prefix to the conclusion's rationale, never parsing JSON or overriding deterministic conclusions.
In AI-Node-Sentinel, the LangChain provider is a placeholder implementation whose constructor directly raises NotImplementedError, with no model-calling logic completed yet; all LLM experiments in the article use the author's own fake provider rather than LangChain or a real model.
AI-Node-Sentinel's Xid lookup table covers only seven codes (13, 31, 48, 74, 79, 119, 120), and all codes outside the table return None and are escalated to a human, with refusal to guess being the most important safety property of the pipeline.
The AI-Node-Sentinel orchestrator uses int(event.payload['xid']) to extract the encoding, with no validation and no per-event isolation; a malformed event mixed in will throw KeyError, ValueError or TypeError, causing the conclusions for the entire telemetry window to be discarded.
In AI-Node-Sentinel, after plugging in a fake provider that perfectly follows the prompt and returns DRAIN_NODE with 0.95 confidence, the final action is still NEEDS_HUMAN_REVIEW and confidence is still 0.0; only the rationale has [LLM] text appended, showing that the LLM has no voting power.
The evidence string assembled by the AI-Node-Sentinel orchestrator contains only fields such as node, gpu, xid, name, severity and ts; the category and notes fields in the lookup table entries, as well as the temperature and ECC readings for the same GPU, never enter the evidence, because the evidence is constructed inside the per-event triage function and has no window context.
In AI-Node-Sentinel, if the provider always throws RuntimeError, the exception passes straight through triage_event and causes the entire run to fail, the same failure mode as malformed data.
The AI-Node-Sentinel orchestrator loads the prompt using the relative path open('prompts/triage_system.md'), so running from a directory other than the repository root throws FileNotFoundError.
The AI-Node-Sentinel orchestrator performs stateless per-event triage: three XID 79s on the same GPU within three hours produce three identical conclusions that know nothing of each other (same action, same confidence, same rationale), an architectural lack of state management.
4 more timeline events
All Facts (14)
The open-source project AI-Node-Sentinel is at version v0.1, and its triage decision tree is a deliberate placeholder implementation that returns NEEDS_HUMAN_REVIEW and a confidence of 0.0 for every event.
50%UnverifiedAI-Node-Sentinel's LLM integration point is in agent/llm.py with the protocol reason(system_prompt, evidence) -> str, and the orchestrator only appends the LLM's returned text with an [LLM] prefix to the conclusion's rationale, never parsing JSON or overriding deterministic conclusions.
50%UnverifiedIn AI-Node-Sentinel, the LangChain provider is a placeholder implementation whose constructor directly raises NotImplementedError, with no model-calling logic completed yet; all LLM experiments in the article use the author's own fake provider rather than LangChain or a real model.
50%UnverifiedAI-Node-Sentinel's Xid lookup table covers only seven codes (13, 31, 48, 74, 79, 119, 120), and all codes outside the table return None and are escalated to a human, with refusal to guess being the most important safety property of the pipeline.
50%UnverifiedThe AI-Node-Sentinel orchestrator uses int(event.payload['xid']) to extract the encoding, with no validation and no per-event isolation; a malformed event mixed in will throw KeyError, ValueError or TypeError, causing the conclusions for the entire telemetry window to be discarded.
50%UnverifiedIn AI-Node-Sentinel, after plugging in a fake provider that perfectly follows the prompt and returns DRAIN_NODE with 0.95 confidence, the final action is still NEEDS_HUMAN_REVIEW and confidence is still 0.0; only the rationale has [LLM] text appended, showing that the LLM has no voting power.
50%UnverifiedThe evidence string assembled by the AI-Node-Sentinel orchestrator contains only fields such as node, gpu, xid, name, severity and ts; the category and notes fields in the lookup table entries, as well as the temperature and ECC readings for the same GPU, never enter the evidence, because the evidence is constructed inside the per-event triage function and has no window context.
50%UnverifiedIn AI-Node-Sentinel, if the provider always throws RuntimeError, the exception passes straight through triage_event and causes the entire run to fail, the same failure mode as malformed data.
50%UnverifiedThe AI-Node-Sentinel orchestrator loads the prompt using the relative path open('prompts/triage_system.md'), so running from a directory other than the repository root throws FileNotFoundError.
50%UnverifiedThe AI-Node-Sentinel orchestrator performs stateless per-event triage: three XID 79s on the same GPU within three hours produce three identical conclusions that know nothing of each other (same action, same confidence, same rationale), an architectural lack of state management.
50%UnverifiedIn AI-Node-Sentinel, non-XID events are discarded at the extraction stage and produce no conclusion and are not attached as context to XID conclusions; in a 64-node baseline window, 1024 of 1285 events were filtered out this way.
50%UnverifiedIn AI-Node-Sentinel's simulated window of 64 nodes × 8 GPUs with seed 42, there are 1285 events in total and 261 XID events, and the deterministic lookup pipeline takes about 0.7 milliseconds.
50%UnverifiedThe 11% to 13% unknown-encoding share in the text is a constructive artifact of the simulator's injection pool design (one of the eight encodings is a deliberately constructed unknown 999) and cannot serve as a reference for the unknown rate of a real cluster.
50%UnverifiedThe conclusions of the six fault-injection experiments in the text are universal: not one of the six breakpoints is a model-capability problem; all are wiring problems (missing per-event isolation, unfulfilled output contract, insufficient evidence depth, exception propagation, path dependency, and statelessness plus corroboration blind spots).
50%