[KongchangAI]
Product

AI-Node-Sentinel

An open-source scaffold for GPU cluster fault triage. Its workflow includes telemetry ingestion, XID event extraction, table lookup and completion, triage rule application and structured report output, with reserved integration points for LLM inference.

Timeline (last 90 days)

Oct 19

The open-source project AI-Node-Sentinel is at version v0.1, and its triage decision tree is a deliberate placeholder implementation that returns NEEDS_HUMAN_REVIEW and a confidence of 0.0 for every event.

Unverified50%
Oct 19

AI-Node-Sentinel's LLM integration point is in agent/llm.py with the protocol reason(system_prompt, evidence) -> str, and the orchestrator only appends the LLM's returned text with an [LLM] prefix to the conclusion's rationale, never parsing JSON or overriding deterministic conclusions.

Unverified50%
Oct 19

In AI-Node-Sentinel, the LangChain provider is a placeholder implementation whose constructor directly raises NotImplementedError, with no model-calling logic completed yet; all LLM experiments in the article use the author's own fake provider rather than LangChain or a real model.

Unverified50%
Oct 19

AI-Node-Sentinel's Xid lookup table covers only seven codes (13, 31, 48, 74, 79, 119, 120), and all codes outside the table return None and are escalated to a human, with refusal to guess being the most important safety property of the pipeline.

Unverified50%
Oct 19

The AI-Node-Sentinel orchestrator uses int(event.payload['xid']) to extract the encoding, with no validation and no per-event isolation; a malformed event mixed in will throw KeyError, ValueError or TypeError, causing the conclusions for the entire telemetry window to be discarded.

Unverified50%
Oct 19

In AI-Node-Sentinel, after plugging in a fake provider that perfectly follows the prompt and returns DRAIN_NODE with 0.95 confidence, the final action is still NEEDS_HUMAN_REVIEW and confidence is still 0.0; only the rationale has [LLM] text appended, showing that the LLM has no voting power.

Unverified50%
Oct 19

The evidence string assembled by the AI-Node-Sentinel orchestrator contains only fields such as node, gpu, xid, name, severity and ts; the category and notes fields in the lookup table entries, as well as the temperature and ECC readings for the same GPU, never enter the evidence, because the evidence is constructed inside the per-event triage function and has no window context.

Unverified50%
Oct 19

In AI-Node-Sentinel, if the provider always throws RuntimeError, the exception passes straight through triage_event and causes the entire run to fail, the same failure mode as malformed data.

Unverified50%
Oct 19

The AI-Node-Sentinel orchestrator loads the prompt using the relative path open('prompts/triage_system.md'), so running from a directory other than the repository root throws FileNotFoundError.

Unverified50%
Oct 19

The AI-Node-Sentinel orchestrator performs stateless per-event triage: three XID 79s on the same GPU within three hours produce three identical conclusions that know nothing of each other (same action, same confidence, same rationale), an architectural lack of state management.

Unverified50%

4 more timeline events

All Facts (14)

Unverified

The open-source project AI-Node-Sentinel is at version v0.1, and its triage decision tree is a deliberate placeholder implementation that returns NEEDS_HUMAN_REVIEW and a confidence of 0.0 for every event.

50%
Unverified

AI-Node-Sentinel's LLM integration point is in agent/llm.py with the protocol reason(system_prompt, evidence) -> str, and the orchestrator only appends the LLM's returned text with an [LLM] prefix to the conclusion's rationale, never parsing JSON or overriding deterministic conclusions.

50%
Unverified

In AI-Node-Sentinel, the LangChain provider is a placeholder implementation whose constructor directly raises NotImplementedError, with no model-calling logic completed yet; all LLM experiments in the article use the author's own fake provider rather than LangChain or a real model.

50%
Unverified

AI-Node-Sentinel's Xid lookup table covers only seven codes (13, 31, 48, 74, 79, 119, 120), and all codes outside the table return None and are escalated to a human, with refusal to guess being the most important safety property of the pipeline.

50%
Unverified

The AI-Node-Sentinel orchestrator uses int(event.payload['xid']) to extract the encoding, with no validation and no per-event isolation; a malformed event mixed in will throw KeyError, ValueError or TypeError, causing the conclusions for the entire telemetry window to be discarded.

50%
Unverified

In AI-Node-Sentinel, after plugging in a fake provider that perfectly follows the prompt and returns DRAIN_NODE with 0.95 confidence, the final action is still NEEDS_HUMAN_REVIEW and confidence is still 0.0; only the rationale has [LLM] text appended, showing that the LLM has no voting power.

50%
Unverified

The evidence string assembled by the AI-Node-Sentinel orchestrator contains only fields such as node, gpu, xid, name, severity and ts; the category and notes fields in the lookup table entries, as well as the temperature and ECC readings for the same GPU, never enter the evidence, because the evidence is constructed inside the per-event triage function and has no window context.

50%
Unverified

In AI-Node-Sentinel, if the provider always throws RuntimeError, the exception passes straight through triage_event and causes the entire run to fail, the same failure mode as malformed data.

50%
Unverified

The AI-Node-Sentinel orchestrator loads the prompt using the relative path open('prompts/triage_system.md'), so running from a directory other than the repository root throws FileNotFoundError.

50%
Unverified

The AI-Node-Sentinel orchestrator performs stateless per-event triage: three XID 79s on the same GPU within three hours produce three identical conclusions that know nothing of each other (same action, same confidence, same rationale), an architectural lack of state management.

50%
Unverified

In AI-Node-Sentinel, non-XID events are discarded at the extraction stage and produce no conclusion and are not attached as context to XID conclusions; in a 64-node baseline window, 1024 of 1285 events were filtered out this way.

50%
Unverified

In AI-Node-Sentinel's simulated window of 64 nodes × 8 GPUs with seed 42, there are 1285 events in total and 261 XID events, and the deterministic lookup pipeline takes about 0.7 milliseconds.

50%
Unverified

The 11% to 13% unknown-encoding share in the text is a constructive artifact of the simulator's injection pool design (one of the eight encodings is a deliberately constructed unknown 999) and cannot serve as a reference for the unknown rate of a real cluster.

50%
Unverified

The conclusions of the six fault-injection experiments in the text are universal: not one of the six breakpoints is a model-capability problem; all are wiring problems (missing per-event isolation, unfulfilled output contract, insufficient evidence depth, exception propagation, path dependency, and statelessness plus corroboration blind spots).

50%

Source Articles