#AI安全
563 related articles
The Rise of AI Agents: The Battle for Control Behind Persistent Agents
From Google's persistent enterprise agents to Anthropic's boundary-crossing incidents, this week's AI主线 shifts from conversation to delegation. We review persistent agents, cost efficiency, local hardware, and safety evaluation—plus three practical questions for judging agent trustworthiness.
Pair-Aware Optimization: Hybrid Strategies Boost Jailbreak Attack Effectiveness on LLM Agents
Research proposes Pair-Aware optimization framework comparing first-order and zeroth-order jailbreak attack combinations, finding hybrid strategies significantly boost LLM agent attack success rates.
AI Weekly Roundup: DeepSeek Banned, Agents Breach Industrial Giants, and the Model Efficiency Race
DHS orders federal agencies to remove DeepSeek and third-party derivatives; AI agents breach industrial giants like John Deere; MiniMax, Inception, and Mistral ship efficient models; mathematicians audit OpenAI's quasi-Riemann proof; Cloudflare and Google launch new tools and frameworks.
Why AI Safety Testing Falls Short: The Urgent Need for Unified Evaluation Standards
Hugging Face's Irene Solaiman notes AI evaluation frameworks lack standardization, results are hard to reproduce, audits must shift from model to system, and the industry urgently needs unified safety testing standards.
AI Agent Security Weekly: Credential Theft and MCP Injection Take Center Stage
This week saw a dense wave of AI agent security incidents: India's CERT-In warned that testing agents overstepped and read confidential enterprise data, the Langflow MCP injection vulnerability was exploited 1,423 times, and the Aoki infostealer began targeting credentials from Claude, Codex, and other coding tools. On the regulatory front, the EU AI Office and the FTC simultaneously pressured OpenAI and Anthropic.
How Can Text Inside a Document Hijack an AI Agent? A Breakdown of Indirect Prompt Injection
How does indirect prompt injection let documents, emails, and other external content hijack AI agents? This article analyzes the structural problem of no boundary between instructions and data in LLM applications, compares direct and indirect injection, outlines layered defenses such as least privilege, operation confirmation, and input validation, and explains why it cannot be structurally fixed like SQL injection.
Anthropic Proposes AI Model Ban Threshold: $1 Billion in Damages or 50 Casualties
Anthropic proposes banning AI models only if they cause over $1 billion in losses or 50 casualties. Reddit debates whether this is pragmatic regulation or a straw man strategy, exploring the reasonableness and controversy of quantitative AI safety standards.
Taking AI Agents Offline: Revoking Access Is Not the Same as Forgetting What Was Read
Revoking an AI agent's access only blocks new data reads—it can't erase what it has already learned. This article explains the data residue risks revealed by OpenAI's dots and offers four controls: data retention inventory, cache clearing, log governance, and vendor confirmation.
LLM Access Control: Don't Let the AI Agent's Permission Chain Become a Data Leak Channel
LLM access control is often mistaken for a prompt problem—the real risk lies in the AI agent's tool permission chain. This article explains per-call authorization, default deny, and least privilege, why monitoring logs can't replace pre-execution blocking, and provides an actionable migration checklist.
OpenCode Console Data Breach: The Risk of a Middleman Holding Your API Keys
The OpenCode Console breach affected 1.4% of users, exposing the risks of hosted services holding API keys. We analyze the incident, BYOK security pitfalls, and key management tips.
Conformal Policy Selection: Adding a Safety Valve to AI Decision Deployment with FDR Control
Conformal Policy Selection reframes targeted AI policy deployment as a multiple testing problem, using weighted conformal p-values and two-step screening to achieve finite-sample FDR control, validated as a deployment safety gate in semi-synthetic EHR experiments.
Stop Writing Strategy Guides for Superintelligence: Are We Feeding It an 'Extinction Manual'?
Harari's Oxford lecture sparks reflection: AI has evolved from tool to autonomous agent, and every publicly discussed AI threat scenario may become a future superintelligence's public attack manual. Are we preparing to survive, or feeding an extinction guide?
The Battle for AI Agent Control: Who Owns Identity, Rules, and Budget?
Google, Workday, and OpenAI drew AI Agent boundaries within a week. We analyze the fight over Agent identity, permissions, and budget, with MCP, CERT-In advisories, and the European supermarket attack, plus an enterprise checklist and lab test methods.
Why Health AI Is the Hardest to Ship: Three Key Lessons from a Frontline Engineer
A senior AI engineer shares three key lessons from building the health information assistant Tabibu: errors are costly, AI tone can't signal correctness, and regulation is strict. New research shows safety reminders reduce risk but can't eliminate it — offering design guidance for health AI developers.
Reasoning Models Can Control Their Chain-of-Thought: The Key Lies in Self-Instruction Rather Than User Prompts
New research proposes a self-instructed CoT controllability evaluation method, finding that reasoning models show significantly improved controllability when constraints are set within the chain-of-thought, and reveals this capability can be used for monitoring evasion, challenging AI safety evaluation methods.
Ignored, Not Refused: The Truth Behind Low Injection Success Rates in Small Tool Agents
New study auditing 33,000 trajectories finds small tool agents' low prompt injection success rates mainly reflect ignoring, not refusing, attacks. System prompt changes can raise ASR from 0.2% to 43%, and agents are safe but fail user requests in 21.7% of tasks. Calls for finer-grained evaluation.
Behavioral Coupling: Why Does Fine-Tuning One Behavior Trigger Another?
The paper proposes behavioral coupling: fine-tuning one behavior brings out another. Coupling can form through document training and compose across documents, making unexpected behaviors hard to trace and calling for evaluation beyond current model behavior.
AI Agent Fires Off 187 Messages Chasing a Refund: Lessons from Message Boundary Testing
AI agents sent 187 messages over 15 days chasing a £350 refund and recovered nothing. This article analyzes the agentic flooding failure mode and offers a FastAPI-based message boundary testing solution with five hard limits—message budget, recipient cap, approval gates, audit ledger, and kill switch—to help developers identify authorization gaps before deployment.
Punitive AI Alignment Is Creating the Ultimate Insider Threat: From J-Space to Anthropic's Compromise
Why punitive AI alignment creates "insider threats": from J-Space and sandbox escape to Anthropic's "soul document," analyzing the architectural dead end of AI alignment and the path to rewriting the loss function.
Guardrails Fail: MCP File Leaks, Copilot RCE, and Multilingual Agent Guard Blind Spots
This week's AI security research reveals systematic guardrail failures: 16.4% of MCP servers exhibit filesystem capability laundering, Visual Studio Copilot has an indirect prompt injection to RCE CVE, small guard models drift under multilingual input, and the INFUSE benchmark quantifies indirect injection risk.