ApexW Optimizer Benchmark: Matches 8-bit AdamW in Unsloth with Lower Memory Usage

A developer shared initial benchmark tests of the new optimizer ApexW on Reddit. The tests were based on the Unsloth framework and a Tesla T4 GPU, comparing standard AdamW, 8-bit AdamW, and two variants of ApexW. Results show that ApexW 8bit has a slight edge in final loss and optimizer state memory usage, but overall training speed is comparable to the other optimizers.

Test Setup: Controlled Comparison on a Small Model

For a fair comparison, the developer fixed the following conditions: base model is unsloth/Llama-3.2-3B-Instruct-bnb-4bit (Llama 3.2 3B instruction-tuned, 4bit quantized), framework is Unsloth, GPU is NVIDIA Tesla T4, dataset contains 1,000 samples, training for 2 epochs and 250 steps, batch size 2, gradient accumulation 4, effective batch size 8, trainable parameters approximately 24.3M (0.75% of total parameters).

The optimizers tested include: PyTorch standard AdamW, BitsAndBytes 8-bit AdamW, ApexW Compact, and ApexW 8bit. Each optimizer was run at two learning rates (2e-4 and 5e-5).

Unsloth Framework: Unsloth is an open-source framework focused on efficient fine-tuning of large language models (LLMs). Through hand-written GPU kernels, dynamic quantization, and memory optimization techniques, it significantly reduces training VRAM usage and improves training speed. It is particularly suitable for parameter-efficient fine-tuning (such as LoRA) on consumer-grade GPUs (like T4). Unsloth supports seamless integration with the Hugging Face ecosystem and has built-in support for loading and training 4-bit quantized models (bnb-4bit), making it possible to fine-tune 3B parameter models with limited VRAM. This test leverages Unsloth's optimization capabilities to fine-tune Llama-3.2-3B-Instruct on a Tesla T4 (16GB VRAM).

LoRA and Trainable Parameters: LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that freezes the main weights of the pre-trained model and only injects low-rank decomposition matrices alongside specific linear layers in Transformer layers for training. The trainable parameters of approximately 24.3M (0.75% of total parameters) refer to the LoRA adapter parameters. Optimizer state memory primarily applies to these trainable parameters and gradients, so the values are much smaller than full-parameter fine-tuning. Understanding this helps explain why the absolute difference in optimizer state memory (e.g., 47MB vs 185MB) is a small fraction of overall VRAM, but the relative proportion is still significant.

Results: ApexW 8bit Has Lower Memory and Slightly Better Loss

In terms of peak VRAM, AdamW is 2.877 GB, 8-bit AdamW and ApexW 8bit are both 2.742 GB, and ApexW Compact is 2.787 GB. The difference in optimizer state memory is more pronounced: AdamW uses 185.501 MB, 8-bit AdamW uses 47.865 MB, ApexW Compact uses 92.751 MB, while ApexW 8bit uses only 47.101 MB, slightly lower than 8-bit AdamW.

Training speed (steps per second) is very close: AdamW is 0.391, 8-bit AdamW is 0.390, ApexW Compact is 0.392, ApexW 8bit is 0.390 (at learning rate 2e-4); the trend is similar at learning rate 5e-5, with differences all between 0.001-0.002.

For final loss, at learning rate 2e-4, ApexW 8bit reported the lowest 1.0829, while 8-bit AdamW is 1.0835 and AdamW is 1.0845; at learning rate 5e-5, ApexW 8bit is 1.1683, 8-bit AdamW is 1.1684, and AdamW is 1.1689. Although the advantage is small, ApexW 8bit is slightly better than the comparison objects at both learning rates.

The developer himself emphasizes that these are preliminary results and cannot yet prove that ApexW has general superiority.

8-bit Optimizers and Block-wise Quantization: 8-bit AdamW (from the BitsAndBytes library) uses block-wise quantization to compress optimizer states (such as Adam's first and second moments) from 32-bit floating-point numbers to 8-bit integer storage, significantly reducing memory usage while maintaining numerical stability through dynamic scaling. ApexW 8bit clearly adopts a similar quantization strategy, but its specific implementation details are not disclosed. The difference in optimizer state memory (e.g., ApexW 8bit's 47.101 MB vs 8-bit AdamW's 47.865 MB) may stem from differences in quantization granularity, metadata overhead, or state sharding. Note that such small differences may be affected by measurement error or implementation details and need to be verified at larger scales.

Peak VRAM and Optimizer State Memory: Peak GPU Memory refers to the maximum GPU memory usage during training, including model weights, gradients, optimizer states, activations, and temporary buffers. Optimizer state memory is only a part of this. In LoRA fine-tuning, because there are few trainable parameters, the absolute value of optimizer state memory is small, so the difference in peak VRAM between different optimizers is less pronounced than in full-parameter fine-tuning. In this article, the peak VRAM difference between AdamW and 8-bit optimizers is about 135MB, which basically matches the optimizer state memory difference (about 138MB), indicating that other components (such as activations) have similar VRAM usage.

Developer Calls for Independent Reproduction and More Rigorous Testing

The developer openly invites the community to reproduce and extend the tests, hoping to verify whether the results are robust. He specifically wants:

• Test ApexW on larger datasets;
• Conduct longer training (1K to 10K+ steps);
• Try different model sizes and architectures;
• Experiment on different GPUs;
• Compare validation loss, convergence, and stability;
• Look for edge cases, numerical issues, or performance regressions.

He is particularly interested in independent reproduction and says he is open to accepting if the results do not hold up.

Statistical Significance and Benchmarking Rigor: The final loss differences reported in the article are extremely small (e.g., 1.0829 vs 1.0835). In a single run with a small-scale dataset (1000 samples), such differences may stem from random seeds, data order, or numerical noise and may not be statistically significant. Rigorous benchmarking typically requires multiple repeated experiments and reporting mean and standard deviation, or using statistical tests to determine whether differences are significant. In addition, the difference in training speed (0.390 vs 0.392 steps/sec) is within measurement error and has limited practical significance. The developer's call for independent reproduction is precisely to rule out chance factors and suggests observing whether trends are stable over longer training and larger datasets.

How to View This Benchmark

From the data, ApexW 8bit achieves optimizer state memory comparable to or slightly lower than 8-bit AdamW, with a slight advantage in final loss; the Compact version has intermediate memory usage but far lower than standard AdamW. However, whether these differences are statistically significant and whether they can be maintained at larger scales still requires further verification.

For practitioners concerned with training efficiency, this test provides a new option, but before adopting in production, it is advisable to wait for more independent reproduction results, especially performance on full-parameter fine-tuning or larger models.

Currently, ApexW's code and more details are not provided in the post. Interested readers can follow updates to the original post.

One-line Summary

Developer's preliminary tests show ApexW 8bit optimizer state memory is slightly lower than 8-bit AdamW, with a marginal advantage in final loss but comparable speed, requiring independent reproduction.

Paragraph Summary

A developer shared initial benchmark tests of the new optimizer ApexW on Reddit, based on the Unsloth framework and Tesla T4 GPU, comparing standard AdamW, 8-bit AdamW, and two variants of ApexW. The tests fixed conditions such as a small model (Llama-3.2-3B-Instruct 4bit quantized), 1000 samples, and 2 epochs. Results show that ApexW 8bit has optimizer state memory of 47.101 MB, slightly lower than 8-bit AdamW's 47.865 MB and far lower than standard AdamW's 185.501 MB; final loss is slightly lower than the comparison objects at both learning rates, but the gap is very small; training speed is almost identical. The developer emphasizes these are preliminary results and calls for independent community reproduction, testing on larger datasets, longer training, and different hardware to verify robustness.