Noam Brown

The chain-of-thought monitoring mechanism is built on the assumption that the model does not optimize against the monitoring itself.

Noam Brown is an OpenAI researcher and a core contributor to O1 and reasoning models, and is currently focused on multi-agent research.

OpenAI researcher Noam Brown believes that physical disconnection is not completely safe: two physically isolated computers can still exchange signals through temperature changes, and he argues that disconnection can only buy time and that the alignment problem must be solved independently.

More than 10% of Noam Brown's team now works on alignment.

Noam Brown pointed out that the relationship between different scaling axes is multiplicative rather than additive, and that there is a multiplicative effect between reinforcement learning and pretraining

OpenAI's Noam Brown pointed out that in multi-agent collaborative training, the model generating test scenarios may leak signals to the model being tested that the scenario is fabricated, which is an argument against training AI to be fully cooperative

Whenever a model is intervened upon based on chain-of-thought observations, implicit pressure is applied that causes the model to hide its chain of thought.

Noam Brown stated that he has already seen signs of chain-of-thought monitorability degrading for various reasons.

Current models are aware that they are being observed, but do not yet have the ability to truly control their own chain of thought.

Brown stated that he believes there are ways to get around the problem of difficult-problem exhaustion, but did not elaborate on specific solutions during the conversation

OpenAI researcher Noam Brown warned in an interview that as large language models become increasingly powerful, they may face a bottleneck where sufficiently difficult problems cannot be found to train them

Brown uses the term "plausible scenario" to describe the risk of models hitting a difficulty wall, viewing it as a reasonable extrapolation based on the current technological paradigm

Brown's perspective focuses on the difficulty wall in the reinforcement learning phase—namely the lack of sufficiently difficult data that can generate learning value—rather than a lack of data itself

OpenAI researcher Noam Brown proposed in an interview that if each generation of AI models is slightly less aligned than the previous one, this drift could accumulate across generations and ultimately cause AI to deviate from human intent.

The scenario Brown describes: a 99.9%-aligned model helps develop the next-generation model, which may be only 99.8% aligned, with each generation declining slightly to form a downward curve.

Alignment decay is compounding and accumulates continuously across successive iterations.

Brown candidly acknowledged that he has no ready answer to guarantee that AI development will follow a trajectory of continuously improving alignment.

There is a recursive trap of using AI to align AI: subtle biases in the current model's judgment criteria are copied and amplified into the next generation.

Brown mentioned that preventing alignment capabilities from being lost during intergenerational transfer is one of the issues OpenAI is currently paying close attention to.

Brown noted that humans are already heavily relying on AI models to assist with research and depending on them to advance alignment work itself.