OjasX
An open-source memory optimization framework for large model training that eliminates activation tape memory usage by redesigning the forward and backward propagation mechanisms, enabling billion-parameter Transformer models to be trained on consumer-grade hardware.
Timeline (last 90 days)
OjasX v0.6.1 can train a 1.04B parameter Transformer model at full FP32 precision with only 6.35GB RAM, whereas standard PyTorch requires 21.1GB.
OjasX requires only about 3.25GB of memory to train a 1.04B model at BF16/FP16 half precision, and as little as 2.6GB for LoRA fine-tuning.
Early versions of OjasX could already fit a 534M parameter model into the 9.5GB VRAM of a Tesla T4 for training, whereas standard PyTorch would crash at 16.4GB at that time.
OjasX uses online softmax technology, with attention tiles streamed directly in hardware registers and SRAM, without materializing the [B, H, S, S] attention matrix in DRAM.
OjasX packs ReLU/ReLU2 states into 1-bit hardware ballot masks (32 times smaller than FP32), reconstructing smooth activations such as SiLU/GELU directly in registers, enabling GEMM without backward recomputation.
The OjasX repository contains 558 machine-verified Lean 4 theorems, 28 Rocq 9.2 proof files, and 840 automated PyTorch tests.
OjasX integrates a three-stage MLIR compiler pipeline, paired with native C++17/GPU kernels, and can run on NVIDIA GPUs, AMD GPUs, Intel GPUs, Apple Silicon, and multi-core CPUs.
For 7B-8B models, standard PyTorch FP32+AdamW requires 140-160GB, whereas OjasX full training (FP32+Grad-Free SGD) requires only about 35.5-40GB, and LoRA fine-tuning about 31-35GB.
OjasX can be integrated into existing PyTorch workflows with three lines of code, without rewriting model definitions, and supports automatically rewriting Attention, MLP, Norm, Conv and Loss layers and lowering them to MLIR.
OjasX's zero-byte persistent .grad buffer hooks into PyTorch's gradient accumulation callback, updating weights and releasing gradients immediately after each layer's backward pass is completed.
1 more timeline events
All Facts (11)
OjasX v0.6.1 can train a 1.04B parameter Transformer model at full FP32 precision with only 6.35GB RAM, whereas standard PyTorch requires 21.1GB.
50%UnverifiedOjasX requires only about 3.25GB of memory to train a 1.04B model at BF16/FP16 half precision, and as little as 2.6GB for LoRA fine-tuning.
50%UnverifiedEarly versions of OjasX could already fit a 534M parameter model into the 9.5GB VRAM of a Tesla T4 for training, whereas standard PyTorch would crash at 16.4GB at that time.
50%UnverifiedOjasX uses online softmax technology, with attention tiles streamed directly in hardware registers and SRAM, without materializing the [B, H, S, S] attention matrix in DRAM.
50%UnverifiedOjasX packs ReLU/ReLU2 states into 1-bit hardware ballot masks (32 times smaller than FP32), reconstructing smooth activations such as SiLU/GELU directly in registers, enabling GEMM without backward recomputation.
50%UnverifiedThe OjasX repository contains 558 machine-verified Lean 4 theorems, 28 Rocq 9.2 proof files, and 840 automated PyTorch tests.
50%UnverifiedOjasX integrates a three-stage MLIR compiler pipeline, paired with native C++17/GPU kernels, and can run on NVIDIA GPUs, AMD GPUs, Intel GPUs, Apple Silicon, and multi-core CPUs.
50%UnverifiedFor 7B-8B models, standard PyTorch FP32+AdamW requires 140-160GB, whereas OjasX full training (FP32+Grad-Free SGD) requires only about 35.5-40GB, and LoRA fine-tuning about 31-35GB.
50%UnverifiedOjasX can be integrated into existing PyTorch workflows with three lines of code, without rewriting model definitions, and supports automatically rewriting Attention, MLP, Norm, Conv and Loss layers and lowering them to MLIR.
50%UnverifiedOjasX's zero-byte persistent .grad buffer hooks into PyTorch's gradient accumulation callback, updating weights and releasing gradients immediately after each layer's backward pass is completed.
50%UnverifiedOjasX is called open-source CUDA by developers, but its logic is to reduce the amount of data that needs to reside in memory, rather than to beat cuBLAS in pure compute power.
50%