Safety Classifier
A judgment layer that assesses risk levels before tool calls in AI coding tools, performing static and contextual analysis of parameters, paths, and command contents based on rules, heuristics, or lightweight models, and outputting a decision to allow, block, or require confirmation.
Timeline (last 90 days)
Claude Code moved the safety classifier from local to server-side.
A server-side classifier enables instant gradual rollout, A/B testing, and rollback for all users, and also facilitates centralized collection of interception samples to iterate the model.
Moving the classifier to the server side means call intents and parameters leave the local environment, so teams handling sensitive code need to include this in compliance assessments.
The security of current LLM Agents mainly relies on two lines of defense, alignment training and safety classifiers, but neither of these can rule out the possibility that a carefully constructed indirect prompt injection bypasses all checks.
In addition to RLHF, mainstream AI image generation models also employ safety classifiers that detect output content in real time during inference.
All Facts (5)
A server-side classifier enables instant gradual rollout, A/B testing, and rollback for all users, and also facilitates centralized collection of interception samples to iterate the model.
50%UnverifiedMoving the classifier to the server side means call intents and parameters leave the local environment, so teams handling sensitive code need to include this in compliance assessments.
50%UnverifiedThe security of current LLM Agents mainly relies on two lines of defense, alignment training and safety classifiers, but neither of these can rule out the possibility that a carefully constructed indirect prompt injection bypasses all checks.
50%UnverifiedIn addition to RLHF, mainstream AI image generation models also employ safety classifiers that detect output content in real time during inference.
50%UnverifiedClaude Code moved the safety classifier from local to server-side.
50%