Aleph Alpha Open-Sources Kolibri-1: 78B Parameters, Only 3.46B Activated, with 1M Token Context Support

European AI company Aleph Alpha has released Kolibri-1 on Hugging Face under the Apache 2.0 license. This model uses a Mixture of Experts (MoE) architecture with 78B total parameters, but activates only 3.46B during inference — while supporting context lengths of up to 1 million tokens. For developers interested in efficient large models and long-context capabilities, this is an open-source option worth paying attention to.

Sparse Activation: 78B Total, Only 3.46B Active

Kolibri-1's most striking design choice is its extremely high parameter sparsity. The model has 78B total parameters, but each forward pass activates only about 3.46B — less than 5% of the total. The core idea behind this MoE architecture is that the model contains a large number of "expert" sub-networks, but only a small subset is routed and activated for each input token.

The immediate benefit is a dramatic reduction in inference cost. Compared to traditional dense models, MoE models can significantly cut computational overhead at runtime for equivalent performance. For teams with limited compute budgets or those looking to deploy locally, this "large capacity, low activation" combination offers a much better cost-efficiency tradeoff.

That said, 78B total parameters still means loading the model requires considerable VRAM or system memory — sparse activation reduces computation, not storage footprint.

The Mixture of Experts (MoE) architecture is not a new concept — its roots trace back to the 1990s — but it has only seen large-scale adoption in large language models in recent years. A typical MoE layer consists of several "expert" feed-forward networks and a "router" that computes weights for each input token and assigns it to the Top-K highest-scoring experts. In Kolibri-1's case, 78B parameters are distributed across many expert networks, but only those corresponding to 3.46B parameters are activated per inference step. The remaining expert weights reside in memory but don't participate in matrix computations for the current token. This mechanism lets the model retain a large knowledge capacity while compressing the floating-point operations (FLOPs) per forward pass to roughly the level of a dense model with equivalent active parameters. Popular recent open-source models like Mixtral 8×7B and DeepSeek^Ad-V2 follow a similar approach, validating MoE's practical value in balancing efficiency and capacity.

1 Million Token Context Window

Kolibri-1 claims support for context lengths of up to 1 million tokens. Ultra-long context has been one of the key battlegrounds in the large model space, directly determining how long a document, codebase, or conversation history the model can handle in a single pass.

A million-token window means the model can ingest entire books, large code repositories, or massive retrieval corpora in one shot — a capability particularly critical for RAG (Retrieval-Augmented Generation), long-document analysis, and code understanding. Of course, real-world effectiveness depends on the specific task. Context window size is an upper bound; what truly matters is the model's ability to retrieve and reason over information across extremely long contexts.

Achieving ultra-long context capabilities presents two core engineering challenges. The first is positional encoding extrapolation: traditional absolute positional encodings struggle to generalize beyond training lengths, which is why current mainstream approaches use RoPE (Rotary Position Embedding) paired with extrapolation techniques like YaRN or LongRoPE to extend effective range to the million-token scale. The second is the computational complexity of attention: standard self-attention scales quadratically with sequence length, making full attention over 1 million tokens impractical. Real implementations typically rely on variants like sliding window attention, sparse attention, or chunked attention to keep costs manageable. Furthermore, there's a significant gap between "supporting 1M token context" and "performing well in 1M token context" — the industry-standard "Needle-in-a-Haystack" test is a key benchmark for measuring a model's ability to locate critical information across ultra-long contexts, and should not be skipped when evaluating models of this type.

Apache 2.0 License: Business-Friendly

Kolibri-1 is released under the Apache 2.0 license, one of the most developer- and enterprise-friendly open-source licenses available. Unlike some "open weights" models that come with usage restrictions, Apache 2.0 allows free commercial use, modification, and redistribution — with no monthly active user limits or additional licensing requirements.

This is particularly meaningful for the European AI ecosystem. Aleph Alpha, as a German AI company, has long emphasized data sovereignty and European AI independence. Open-sourcing an efficient, long-context model under a permissive license lowers the barrier to enterprise adoption and gives European and global developers a credible alternative to mainstream US-based providers.

Technical Positioning and Key Highlights

Kolibri-1's architecture reflects an important trend in modern large model development: pursuing high model capacity while controlling actual inference costs through sparse activation. This aligns with the direction multiple vendors have been exploring through MoE architectures in recent years.

According to the release, Aleph Alpha has also published a technical report with detailed information on the model's training and architecture. Developers looking to evaluate the model in depth are encouraged to review both the technical report and the model card on Hugging Face, with particular attention to:

• Actual inference resource requirements: What the 78B total parameters demand from deployment hardware
• Real long-context performance: Retrieval and reasoning quality within the 1M token window
• Multilingual and European language support: A potential advantage given the company's European origins
• Benchmark results: Comparative performance against open-source models of similar scale

Summary

Kolibri-1 combines high total parameter count, low activation ratio, ultra-long context support, and a business-friendly license in a single package — making it a distinctive new entry in the open-source LLM landscape. It reflects Aleph Alpha's commitment to European AI autonomy while riding the wave of MoE-based efficient inference. For teams seeking a commercially usable, long-context open-source model, it's worth adding to your evaluation list. How it performs in practice remains to be seen as the community puts it through real-world tasks.

Sources: Aleph Alpha's official Hugging Face page and technical report (circulated via Reddit community discussions). This article is based on publicly available information; specific performance metrics should be verified against the official technical report.