Manacá-1B-Instruct-2: Reproducible Safety Alignment for Brazilian Portuguese LLMs and a DPO Negative Result

Open-source language models have long under-served Brazilian Portuguese (PT-BR), and the few models that do undergo safety alignment rarely disclose which attempts failed, nor do they offer statistically validated head-to-head comparisons. Manacá-1B-Instruct-2 addresses exactly this gap: it is the second-generation post-training alignment version of Manacá-1B (1.72B parameters, trained from scratch for PT-BR), with an SFT corpus of 291,223 examples. The team publicly documents three failed alignment attempts, a controlled data-mixture sweep, and a silent failure mode in evaluation methodology — a practice of "writing negative results into the paper" that is still uncommon in alignment research today.

Across-the-board DPO negative results: reward goes up, generation doesn't change

The most cautionary part of the paper is three DPO (Direct Preference Optimization) attempts, all of which ended in negative results. One of them exhibited a classic "metric illusion": during training the reward margin shot up to 1.0, making it look like preferences had been learned, yet free generation outputs showed no change whatsoever.

This suggests that reward metrics and actual behavior can become completely decoupled — the training signal is satisfied internally but fails to transfer to the text distribution at inference time. For teams that rely on reward curves to judge whether training succeeded, this is a signal worth taking seriously: the reward margin itself does not constitute evidence of effective alignment; validation via free generation is mandatory.

If the DPO route doesn't work, where does safety capability come from? The answer is a dedicated safety-SFT, styled after the LLM-jp AnswerCarefully dataset construction approach, iterated over three rounds until the safety-helpfulness frontier converged. This also suggests, by contrast, that for small-scale native-language models of this kind, carefully constructed SFT data may be more directly effective than preference optimization.

DPO (Direct Preference Optimization) fundamentally bypasses an explicit reward model by constructing a closed-form objective directly from preference pairs (chosen/rejected) to optimize the policy model. Its loss function drives the probability of chosen responses up relative to rejected ones through an implicit reward difference based on log-ratios, so the reward margin in training logs is essentially a measure of how well the model fits the preference pairs, not a measure of true human preferences. When chosen and rejected are semantically very similar, or when the preference data distribution differs systematically from the free-generation distribution at inference time, the model can satisfy the objective purely by adjusting token-level probabilities without exhibiting any behavioral change in open-ended generation — exactly the mechanism behind the decoupling observed in this paper.

Controlled mixture sweep: over-refusal has a knee point

Safety SFT typically requires mixing in a certain proportion of general-purpose data to prevent the model from becoming a refusal machine that won't answer anything. The team conducted a controlled sweep of the general-data proportion, from 10% to 60%, producing a non-monotonic over-refusal curve with a knee point in the 40%–50% range.

The value of this finding lies in turning "how much general data to mix in" from a matter of intuition into a measurable curve: when the proportion is too low, general capabilities get squeezed by safety data and refusal rates rise; when it's too high, safety capability is diluted. The knee point provides a reusable mixture range, with implications for safety alignment in the same language or other low-resource languages.

The paper also includes n-gram decontamination checks and a Chinchilla-anchored compute budget table, attributing the gap in general capabilities to pretraining scale rather than the alignment method itself. This attribution matters — it avoids pinning the blame for pretraining underfitting on the alignment pipeline.

Over-refusal refers to a model refusing harmless requests that should normally be answered, typically stemming from distributional bias in refusal samples within safety SFT data — for example, refusal rationales that are too broad and cover intents that don't actually fall within the safety scope. Evaluating over-refusal requires constructing a set of prompts that are clearly harmless but may superficially trigger safety keywords (e.g., "how to cut vegetables with a knife"), and measuring the model's refusal rate on these samples. This forms a pair of tension metrics with safety rate: safety rate measures the model's ability to block harmful requests, while over-refusal rate measures the degree to which the model mistakenly rejects harmless requests, and together they characterize the safety-helpfulness frontier. n-gram decontamination checks are used to prevent evaluation set content from leaking into training data via n-gram overlap, which would artificially inflate evaluation scores.

Statistically rigorous head-to-head comparison: confidence intervals and paired tests

The final model was compared against four external PT-BR models on the same set of evaluation items, using Wilson confidence intervals and paired McNemar tests with Holm-Bonferroni correction. The results are remarkably clean: the lower bound of Manacá-1B-Instruct-2's safety rate interval is 94.1%, higher than the upper bounds of all four baseline models.

For safety evaluation, reporting only a point estimate makes it easy to be challenged as noise. Providing an interval lower bound and comparing it against other models' upper bounds is effectively using statistical rigor to prove the gap isn't due to random fluctuation. This approach is worth copying by other teams working on low-resource language alignment: when sample sizes are limited, paired tests with multiple-comparison correction are more convincing than single-point accuracy.

Wilson confidence intervals are an interval estimation method for binomial proportions (such as safety rate) that is more robust than the normal approximation when sample sizes are small or proportions are near 0/1, avoiding out-of-range or overly narrow intervals. The paired McNemar test is used to compare the performance difference between two models on the same set of evaluation items: it focuses only on items where the two disagree (one correct, one incorrect), testing whether this disagreement significantly favors one side, making it more sensitive than independent-sample tests. When multiple pairwise comparisons are performed simultaneously, the false positive rate accumulates; Holm-Bonferroni correction controls the family-wise error rate by sorting p-values and progressively tightening significance thresholds. The combination of these three is standard practice for small-sample evaluation in low-resource settings: interval lower-bound vs. upper-bound comparison provides conservative evidence of a gap, paired tests leverage item-level correspondence to boost power, and multiple-comparison correction prevents spurious significance.

Silent failure of the LLM judge: effective samples quietly trimmed

The paper closes with a methodological pitfall. Among the six safety questions in MT-Bench-PT, on two of them the LLM judge refused to score any response, regardless of what was being evaluated. This doesn't throw an error and doesn't leave obvious traces in logs, but it silently reduces the effective sample size N for that category — and all previous reports from the project had been unaware of this.

The insidiousness of this failure mode is that the pipeline runs to completion as usual, scores are output as usual, and conclusions look complete, while in reality the denominator has shrunk. Anyone using LLM-as-a-judge for safety evaluation should explicitly check the judge's refusal rate, rather than only looking at final scores.

All code, raw data, logs, and the scripts that generate every reported number have been made public — reproducibility is handled quite thoroughly.

LLM-as-a-judge has become a mainstream paradigm for open-ended generation evaluation, but its reliability depends on the judge model providing a valid judgment for every sample. When the judge model uniformly refuses to evaluate certain samples due to safety policies, format parsing failures, or prompt design flaws, score computation still proceeds as usual, except that the denominator N is silently reduced, causing confidence intervals for that category to be erroneously computed and compared. Detecting this failure requires explicitly logging the judge's refusal rate and parsing failure rate, and disclosing the effective sample size for each category in reports; a stricter approach is to pre-set a minimum effective sample threshold for each evaluation category and refuse to output that category's score when it falls below the threshold. For judge models accessed via API calls, one should also be aware that backend version updates may change refusal behavior without notice.

Why this work is worth paying attention to

Manacá-1B-Instruct-2's contribution is not in model scale but in transparency: three DPO negative results, an over-refusal curve, and a silent judge failure mode — these are "failure records" that would normally be deleted, yet they are exactly the signposts other teams need most. For safety alignment in low-resource languages, knowing which roads don't work is just as important as knowing which ones do.