Spark X

An open-source small language model with approximately 4 billion parameters, employing a sparse attention architecture that interleaves local sliding-window attention layers with global attention layers. It natively supports an ultra-long context window of approximately one million tokens and is capable of running on standard consumer-grade hardware.