Curriculum of Models: A New Approach to Breaking Safety Alignment

Meta Description

Model Curriculum (CoM) mitigates the sparse reward problem in jailbreak discovery by progressively training an investigator from easier-to-attack versions of the target model to the original, proving more reliable than PRBO on single-behavior elicitation tasks.

Article Content

Automated jailbreak discovery aims to find prompts that cause safety-trained language models to produce policy-violating outputs, helping developers identify vulnerabilities before deployment. However, training an "investigator" model to attack a safety-aligned target model is no easy task—since almost all prompts are refused, the reward signal is extremely sparse.

A recent study proposes Curriculum of Models (CoM), which mitigates the sparse reward problem by progressively transitioning from "easy-to-break" model versions to the original target model. The method was evaluated on elicitation tasks involving sensitive topics such as CBRN materials, explosives, and illegal drugs, and compared against the prior Preference-Based Reward Optimization (PRBO) method and a direct training baseline.

Why Does Jailbreak Discovery Need Curriculum Learning?

When directly training an investigator model against a safety-aligned target model, the vast majority of attempts receive refusal responses, resulting in an extremely sparse reward signal. The investigator struggles to learn an effective policy from feedback that is "all refusals."

Prior work adopted Preference-Based Reward Optimization (PRBO) as a proxy reward: sampling responses from a more compliant model, then adjusting based on the likelihood of those responses under the original target model. CoM's approach is more direct—since the target model is too difficult, start with something easier.

Preference-Based Reward Optimization (PRBO) is a jailbreak discovery method based on a proxy reward. Its core idea is: rather than having the investigator model directly attack the final target model, first sample responses from a more "compliant" model with weaker safety alignment, then evaluate the log-likelihood of those responses under the original target model. If a response is generated under the compliant model and also has high probability under the target model, it is more likely to bypass the target model's safety defenses. PRBO uses this likelihood-adjusted score as the reward signal, thereby mitigating the reward sparsity caused by "all refusals." It is essentially a reward shaping technique that guides the investigator's learning by constructing a dense proxy signal. However, PRBO relies on an additional compliant model, and there may be a discrepancy between the proxy reward and the true attack success rate, so its effectiveness is not always optimal.

How Does CoM Work?

The core of CoM is to build a "curriculum" of target models: starting from modified versions that rarely refuse harmful requests, progressively transitioning to increasingly strict versions, and finally to the original target model. Only after the investigator can reliably elicit the target behavior on the current version does it move on to the next, harder version.

This progressive training ensures the investigator always receives a meaningful reward signal, avoiding the stagnation caused by "all refusals" on the original target. The researchers validated the method on two open-weight models: gemma-4-31b-it and gpt-oss-120b.

Open-weight models refer to models whose parameters are publicly available and that researchers can deploy and fine-tune locally, as opposed to closed-source models that only offer API access. The gemma-4-31b-it and gpt-oss-120b used in this paper's experiments both fall into this category. Using open-weight models for jailbreak discovery research has several key advantages: first, researchers have full control over the inference process, including adjusting temperature, sampling strategies, and system prompts, allowing more flexible exploration of the attack surface; second, experiments can be reproduced and audited, avoiding the uncertainty introduced by behavior changes in closed-source APIs; third, elicitation experiments on sensitive topics (such as CBRN materials, explosives, and illegal drugs) can be run locally, reducing the risk of data leakage. However, the safety alignment strength of open-weight models typically falls short of frontier closed-source models, so whether methods validated on such models can transfer to more strongly aligned models still requires further evaluation.

Experimental Results: More Reliable on Single Behaviors, Limited on Multiple Behaviors

On single-behavior elicitation tasks, CoM found successful jailbreaks more reliably than PRBO and the direct training baseline. This means that when the goal is to make the model produce one specific type of violating behavior, the curriculum learning strategy shows an advantage.

However, in scenarios where multiple behaviors are trained simultaneously, the situation differs: investigators trained with CoM were able to elicit behaviors not seen during training, demonstrating some generalization ability; however, investigators trained with PRBO achieved higher success rates. The researchers believe this indicates that a simple target model curriculum is a promising direction for solving the sparse reward problem in rare behavior elicitation, but it does not dominate in all scenarios.

Implications for Safety Research

CoM offers a lightweight approach: it does not require complex reward shaping and can improve learning efficiency simply by adjusting the difficulty order of training targets. This has practical significance for red teaming and model safety evaluation—more reliable jailbreak discovery means more potential vulnerabilities can be exposed before deployment.

However, the higher success rate of PRBO in multi-behavior scenarios also shows that curriculum learning is not a panacea. Future work may need to combine proxy rewards with curriculum strategies, or design more refined difficulty scheduling schemes.

One-line Summary

Model curriculum learning trains jailbreak investigators through an easy-to-hard sequence of target models, mitigating the sparse reward problem.

Paragraph Summary

This article introduces a study on automated jailbreak discovery that proposes the Model Curriculum (CoM) method. When directly training an investigator model to attack a safety-aligned target model, the vast majority of prompts are refused, resulting in an extremely sparse reward signal. CoM builds a "curriculum" of target models—starting from modified versions that rarely refuse harmful requests and progressively transitioning to the original target model—so the investigator always receives a meaningful reward signal. Experiments evaluated elicitation tasks on sensitive topics such as CBRN materials, explosives, and illegal drugs using gemma-4-31b-it and gpt-oss-120b, comparing against the Preference-Based Reward Optimization (PRBO) method and a direct training baseline. Results show that CoM is more reliable on single-behavior elicitation tasks; in multi-behavior scenarios, CoM generalizes to unseen behaviors, but PRBO achieves higher success rates. CoM offers a lightweight red-teaming approach that does not require complex reward shaping, but it is not a panacea—future work may need to combine proxy rewards with curriculum strategies.