Berkeley CS294 Course: A Complete Guide to Agentic AI Security — Attacks and Defenses

Article Body

A lecture on agentic AI security and defense from UC Berkeley's Fall 2025 CS294-196 Agentic AI course has attracted widespread attention. The lecture systematically dissects the attack surface, risk assessment methods, and defensive strategies facing agentic AI systems, providing an important security framework for this rapidly evolving field.

Agentic AI Security and Safety: Why It Demands Attention Now

Agentic AI refers to AI systems capable of autonomously perceiving their environment, forming plans, and executing multi-step tasks — distinct from traditional single-turn question-answering LLMs. The core architecture typically includes a perception module, a planning module (e.g., ReAct, Chain-of-Thought reasoning), a memory system, and tool-calling capabilities. Since 2023, with the release of interfaces like OpenAI Function Calling and Anthropic Tool Use, and the emergence of frameworks such as LangChain and AutoGPT, agentic AI has moved from the lab into production environments. In 2025, OpenAI launched Operator and Anthropic introduced Claude Computer Use, marking a new phase in which agentic AI can operate real computers — transforming security from an academic discussion into an urgent engineering reality.

2025 has been dubbed the "Year of Agents," with agentic AI experiencing explosive growth. Yet as AI takes control of more and more systems, the incentives for attackers grow in parallel. The lecture cited the International AI Safety Report, co-authored by approximately 100 leading AI researchers from 30 countries, which highlights the broad spectrum of AI risks.

The lecture begins by clarifying two key concepts: AI Safety focuses on preventing systems from causing harm to the outside world, while AI Security focuses on protecting systems from malicious external actors. The two are not independent — safety mechanisms themselves must remain resilient against adversarial attacks.

History has repeatedly shown that attackers keep pace with new technology, and sometimes get ahead of it. As AI capabilities continue to advance, the consequences of misuse will grow increasingly severe. Examining agentic AI security through an adversarial lens is no longer optional — it is essential.

Agentic AI System Architecture and Attack Surface Analysis

The Hybrid Nature of Agentic Systems

Unlike traditional software systems, agentic AI systems are fundamentally hybrid or composite systems. Traditional systems are composed primarily of symbolic components written by programmers, whereas agentic AI systems incorporate neural components such as LLMs — in addition to conventional software code — enabling complex reasoning and planning.

A typical agentic system workflow looks like this: the host deploys the model and system → the user sends a request → the system processes the request and invokes the model → the model interacts with the rest of the system through its outputs → the system interacts with the external world (reading information, executing actions) → the system returns a response to the user. Every step in this chain can become an attack entry point.

Greater Flexibility Means a Larger Attack Surface

The lecture introduces an important classification framework that describes the agentic design space along multiple dimensions: workflow (from static to fully dynamic and autonomous), input space, action space (from no actions to read/write/execute), toolset (from no tools to dynamically discovered tools), memory (from stateless to persistent memory), and autonomy level (from human-in-the-loop to fully automated).

A core insight is this: the greater an agent's flexibility along any given dimension, the larger its attack surface. There is a fundamental tension between security and functionality that must be addressed at the system design stage.

Expanding Security Goals: Beyond the Traditional CIA Triad

The CIA triad — Confidentiality, Integrity, and Availability — is the classic framework that has underpinned information security since the 1970s, first systematized by the U.S. Department of Defense in the Trusted Computer System Evaluation Criteria (TCSEC, commonly known as the "Orange Book"). Confidentiality ensures information is accessible only to authorized parties; Integrity ensures data is not tampered with by unauthorized actors; Availability ensures systems are operational when needed.

Compared to traditional systems, agentic AI systems introduce additional protection objectives beyond the classic CIA triad. API keys, system prompts (which represent core intellectual property for AI-powered products), user interaction histories, and proprietary model parameters all become sensitive assets requiring protection. Furthermore, an agent's "goal alignment" —