Unverified50% confidenceOpinionExact time
Training pressure may only push the policy's actions to just below the blocking threshold, but the model may have uncertainty about when it will be intercepted; RL will push it to conservatively avoid being intercepted and accumulate knowledge of evading the monitor along the way.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
Unverified经过RLHF训练的模型在面对精心设计的越狱提示时往往能够绕过安全防护输出有害内容76% similarUnverifiedModel-layer security constraints (alignment training, system prompts, output filters) all face jailbreak risks, as carefully crafted prompts can bypass these constraints.74% similarVerifiedDrift detection involves a trade-off: setting the threshold too sensitively leads to frequent false alarms and unnecessary retraining, while setting it too loosely may allow model performance to continuously degrade without being detected.74% similarUnverifiedStopping condition design typically faces four categories of trade-off mechanisms: hard resource limits, oscillation detection, performance thresholds, and diminishing returns detection — sharing the same origin as the early stopping strategy in neural network training.73% similarUnverifiedDo not exclude blocked trajectories from training, because filtering out blocked trajectories is still selecting trajectories that obtain high reward by evading the monitor, implicitly inversely selecting blocked trajectories, and due to the nature of softmax, increasing the probability of evasion necessarily comes at the cost of lowering the probability of non-evasion.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1012101API
curl https://kongchang.com/api/v1/knowledge/claims/1012101MCP
get_claim(id=1012101)