Pair-Aware Optimization: Hybrid Strategies Boost Jailbreak Attack Effectiveness on LLM Agents

LLM agents are being deployed in increasingly complex task scenarios, but their security remains an unresolved issue. A recent study delved into the combined effects of different optimization strategies in jailbreak attacks, proposing a new framework called Pair-Aware Optimization that offers a fresh perspective on understanding agent security weaknesses.

The Optimization Dilemma of Jailbreak Attacks: First-Order vs. Zeroth-Order Methods

Existing optimization-based jailbreak attacks fall mainly into two categories: first-order (FO) methods that leverage gradient information, and zeroth-order (ZO) methods that rely on function value feedback. Each has its strengths and weaknesses—FO methods are typically more precise but require white-box access, while ZO methods are more general but less search-efficient. However, the complementarity of these two approaches and the impact of optimization order on attack effectiveness had not been systematically studied before.

The research team systematically examined four strategies: pure ZO, pure FO, ZO-then-FO, and FO-then-ZO. They built a ZO-FO framework that first explores adversarial prefixes through ZO optimization, then refines the corresponding suffixes through FO optimization. Experiments showed that hybrid optimization significantly improves attack effectiveness compared to single-optimizer methods, and that optimization order has a substantial impact on performance.

Adversarial Prefixes and Suffixes: In optimization-based jailbreak attacks, attackers typically construct input as a combination of an "adversarial prefix" and a "suffix." The prefix is an optimizable text segment inserted before the user instruction, designed to steer the model into a state receptive to subsequent malicious instructions; the suffix is appended after the instruction to further reinforce the attack. Together they form the complete adversarial prompt. The prefix usually plays an exploratory role—finding an initial direction that can activate model vulnerabilities; the suffix plays a refinement role—precisely adjusting under the determined prefix to maximize attack success rate. This division of labor allows prefix and suffix optimization to be decoupled, but it also implies a strong coupling effect between them, meaning optimizing either part alone may fail to achieve the joint optimum.

Pair-Aware Optimization: Coordinating Exploration and Refinement

How to coordinate prefix exploration and suffix refinement is the core challenge of the hybrid framework. The proposed Pair-Aware optimization strategy retains multiple prefix candidates, conditionally refines their suffixes, and selects among the generated "prefix-suffix" pairs. This pair-aware mechanism avoids premature convergence to suboptimal solutions, allowing exploration and refinement to interact beneficially.

The insight behind this design is that jailbreak success often depends on the joint effect of prefix and suffix, and optimizing either part alone may fall into local optima. By maintaining candidate pairs and delaying selection, Pair-Aware optimization achieves more comprehensive coverage of the search space.

Conditional Refinement and Delayed Selection: The core mechanism of Pair-Aware optimization lies in "conditional refinement"—for each retained prefix candidate, rather than immediately deciding its quality, it independently optimizes its suffix under that prefix condition to generate complete "prefix-suffix" pairs. Selection then occurs among all generated pairs rather than prematurely eliminating candidates during optimization. This delayed selection strategy resembles the multi-candidate maintenance idea in beam search: by preserving search diversity, it avoids discarding potentially better solutions due to early evaluation noise. In jailbreak scenarios, early individual scoring of prefixes may not accurately predict their attack effectiveness when combined with suffixes, so pair-wise evaluation is more reliable than individual prefix evaluation.

Re-examining Agent Jailbreak from a Stochastic Generation Perspective

The study further examines agent jailbreak from a stochastic generation perspective. The same adversarial input can trigger diverse reasoning trajectories and actions, and a single greedy output cannot capture this diversity. To address this, the team adopted consistent stochastic decoding strategies in both optimization and evaluation phases.

This shift in perspective is critical: traditional jailbreak evaluation often relies on single deterministic outputs, which may underestimate or overestimate the true attack success rate. Evaluation under stochastic decoding better reflects real deployment scenarios and more accurately captures attack robustness.

Stochastic Decoding and Temperature Sampling: Large language models typically use stochastic decoding strategies (such as temperature sampling and top-p sampling) when generating text, randomly sampling the next token from a probability distribution rather than always selecting the highest-probability token. This means the model may produce different output trajectories even with identical input. In agent scenarios, this randomness is further amplified: the model not only generates text but may also trigger tool calls, environment interactions, and other actions, forming diverse reasoning-action sequences. Traditional jailbreak evaluation using only greedy decoding (always selecting the highest-probability token) yields a deterministic single output that cannot reflect the model's true behavior distribution under stochastic sampling. Adopting consistent stochastic decoding for evaluation means sampling multiple times from the same adversarial input and computing attack success rate statistics, thereby more accurately estimating the expected attack effectiveness and robustness.

Experimental Validation and Key Findings

On two types of benchmarks—malicious instructions and malicious environments—hybrid optimization outperformed single-optimizer methods. The impact of optimization order was also significant—the ZO-then-FO strategy performed better in most scenarios, suggesting an intrinsic connection between the "broad exploration first, precise refinement second" search logic and attack success.

The study also found that Pair-Aware optimization reduces the number of queries required for optimization while maintaining attack success rate, which has practical significance for real-world attack scenarios.

Implications for Agent Security

The value of this work lies not only in proposing a stronger attack method but also in revealing methodological issues in LLM agent security evaluation. The effectiveness of hybrid optimization strategies indicates that single optimization paradigms may systematically underestimate agent vulnerabilities. For defenders, this means more robust protection mechanisms need to be designed against compositional attacks.

Meanwhile, the introduction of stochastic decoding evaluation reminds security researchers that jailbreak evaluation protocols themselves need standardization and diversification to truly reflect the attack surface. Agent security is a dynamic game process, with both attack and defense optimization strategies continuously evolving.

One-line Summary

This study proposes the Pair-Aware optimization framework, revealing the critical impact of hybrid optimization and stochastic decoding on agent jailbreak evaluation.

Paragraph Summary

This study systematically explores the combined effects of first-order (FO) and zeroth-order (ZO) optimization strategies in LLM agent jailbreak attacks, builds a ZO-FO hybrid framework, and compares four strategies: pure ZO, pure FO, ZO-then-FO, and FO-then-ZO. Experiments find that hybrid optimization significantly outperforms single methods, with ZO-then-FO performing best in most scenarios. The study further proposes the Pair-Aware optimization strategy, which coordinates exploration and refinement by retaining multiple prefix candidates, conditionally refining suffixes, and delaying selection to avoid premature convergence. Additionally, it re-examines jailbreak evaluation from a stochastic generation perspective, advocating consistent stochastic decoding to capture diverse reasoning trajectories. Experiments show the method improves attack success rate while reducing query counts, offering important insights for agent security evaluation methodology.