Evaluating Compression of Arabic Legal LLMs: A Context-Sensitivity Perspective

Introduction

Deploying large language models in resource-constrained legal scenarios inevitably requires model compression. But the impact of compression on domain performance may depend on the evaluation protocol itself. A study on the Saudi Arabic legal benchmark ArabLegalEval systematically examined three stages—domain adaptation, quantization, and knowledge distillation—across three models: FANAR-9B, JAIS-8B, and Qwen2.5-7B. It compared two evaluation settings: "with-context" evaluation, where supporting legal provisions are provided, and "no-context" evaluation, where no external information is given.

Remove the Context, and Accuracy Drops by Up to 27 Percentage Points

The study first reveals an easily overlooked phenomenon: without context, the accuracy gap between base models can reach up to 27 percentage points. This means many models perform acceptably when given legal provisions as "hints," but once they must answer independently, their true legal reasoning ability is exposed.

This finding directly supports the paper's core argument—no-context evaluation should be a valuable complement to with-context testing. When assessing the impact of adaptation and compression on Arabic legal models, both protocols are indispensable: the former tests the model's ability to apply information under auxiliary support, while the latter is closer to real deployment scenarios where the model must make autonomous judgments.

The difference between with-context and no-context evaluation essentially reflects the model's adaptability to "open-book" versus "closed-book" answering modes. In the with-context setting, the model can rely on retrieved legal provisions for matching and citation, testing information utilization and reading comprehension. The no-context setting requires the model to reason solely from legal knowledge stored in its parameters, more closely resembling its true performance when deployed independently. The larger the gap between the two, the stronger the model's dependence on external hints, and the more likely its internal knowledge representation is incomplete. In legal scenarios, this distinction is particularly important because incorrect or missing legal citations can carry substantive risks, and with-context evaluation alone can easily overestimate a model's practical usability. Therefore, combining both protocols provides a more comprehensive picture of the model's capability boundaries.

Quantization: Big Memory Savings, Minor Accuracy Perturbation

Among compression methods, 4-bit quantization proves the most robust. The study shows that quantization reduces model memory usage by 64% to 71%, while the change in no-context multiple-choice accuracy ranges only from -3.2 to +1.6 percentage points. For resource-constrained legal deployment environments, this trade-off is quite worthwhile: memory usage shrinks dramatically while accuracy loss remains within a very small range.

Quantization's strong performance may stem from the fact that it primarily affects the granularity of numerical representation without altering the legal knowledge structure the model has learned. For legal question-answering tasks that rely mainly on retrieval and provision matching, this perturbation is often within acceptable limits.

4-bit quantization typically refers to compressing model weights from 16-bit floating point (FP16) or 32-bit floating point (FP32) to 4-bit integer (INT4) representation. This process requires first determining a scale factor and zero-point to map floating-point values to the 4-bit integer range, then dequantizing back to approximate floating-point values during inference. Common approaches include post-training quantization methods such as GPTQ and AWQ, which protect important weights by minimizing quantization error. Since 4 bits can only represent 16 discrete values, quantization inevitably introduces numerical noise, but legal question-answering tasks typically rely on semantic matching rather than high-precision numerical computation, so tolerance for such noise is relatively high. Memory gains mainly come from weight storage compression; actual inference must also account for activation values and KV cache overhead, so end-to-end memory reduction ratios are usually slightly lower than theoretical values.

Distillation: The Smaller the Student, the Greater the Variance

In contrast to quantization's stable performance, knowledge distillation results are more variable. The general trend is that the smaller the student model, the more pronounced the accuracy drop.

But the study also provides a positive case: the Qwen2.5-7B student model distilled from FANAR achieved 76.6% accuracy in no-context evaluation, compared to its teacher model's 77.2%. The gap is only 0.6 percentage points, indicating that when model sizes are similar, distillation can effectively preserve the teacher's legal domain capabilities.

This result also suggests that distillation effectiveness is highly dependent on capacity matching between teacher and student. When student size shrinks significantly, the loss of legal domain knowledge becomes much harder to control.

Knowledge distillation's core idea is to have a small student model mimic the output distribution of a large teacher model, rather than learning only from hard labels. In the legal domain, the student model learns not only from the teacher's predictions of correct answers but also from the teacher's probability assignments over incorrect options—"dark knowledge" such as which legal provisions are more easily confused. Distillation effectiveness is constrained by three factors: temperature parameter, loss function weighting, and student capacity. When the student's parameter count is far smaller than the teacher's, its representation space is insufficient to accommodate the teacher's legal knowledge structure, leading to a "capacity bottleneck" and significant degradation in domain performance after distillation. When teacher and student sizes are similar, distillation serves more as domain knowledge transfer and output smoothing, which is why the FANAR-to-Qwen2.5-7B case shows only a 0.6 percentage point gap.

Why Evaluation Protocols Matter

The methodological value of this study lies in examining "how to evaluate" and "what to compress" within the same framework. If only with-context evaluation is used, performance changes from quantization or distillation may be masked by auxiliary information; introducing no-context evaluation allows the model's own legal reasoning ability to be clearly measured.

For practitioners of Arabic legal AI, this means that before compressing a model, they must first clarify how much external support the model will receive in the deployment scenario. If the actual application relies on retrieval augmentation or provision injection, with-context evaluation is more informative; if the model must independently make legal judgments, no-context evaluation results deserve more attention.

Practical Implications

Overall, 4-bit quantization is currently the more reliable compression path for resource-constrained legal scenarios, offering significant memory savings at limited accuracy cost; distillation requires careful selection of student model size, as large capacity gaps can lead to noticeable loss of legal domain capabilities. More importantly, evaluation of any compression scheme should not rely on a single evaluation protocol—only the combination of with-context and no-context evaluation can paint a complete picture of the model's capabilities in real legal work.

One-line Summary

Study reveals no-context evaluation exposes true legal reasoning ability; 4-bit quantization is robust while distillation is volatile.

Paragraph Summary

This study systematically examines the impact of domain adaptation, 4-bit quantization, and knowledge distillation on three Arabic legal models—FANAR-9B, JAIS-8B, and Qwen2.5-7B—comparing with-context and no-context evaluation protocols. Key findings: without legal provision support, base model accuracy gaps reach up to 27 percentage points, showing that no-context evaluation exposes the true level of independent legal reasoning. 4-bit quantization reduces memory by 64%-71% with no-context accuracy changes of only -3.2 to +1.6 percentage points, making it the most robust compression method for resource-constrained scenarios. Knowledge distillation is more volatile, with smaller student models showing greater accuracy drops; however, the Qwen2.5-7B student distilled from FANAR achieves 76.6%, close to the teacher's 77.2%, indicating that capacity-matched distillation can effectively preserve domain capabilities. The study advocates that no-context evaluation should complement with-context testing to jointly assess compression's impact on legal models.