30-Person Korean Team's Breakthrough: How Motif Cracked the Open-Source Top Five with Under 1,000 GPUs

Korean AI startup Motif shared a remarkably instructive story during an NVIDIA Nemotron Labs livestream: a 30-person team using fewer than 1,000 GPUs trained a model that ranked in the top five on the Artificial Analysis Intelligence Index for open-source models. In an era of LLM competitions defined by tens-of-thousands-of-GPU clusters and hundreds of engineers, that resource footprint is almost laughably modest — yet Motif delivered a globally competitive result.

The KAI Initiative: A Nation's Bet on Foundation Models

Motif's success is inseparable from the KAI (Korean AI) initiative, the government-backed program that provides its backdrop. According to Chan-Lan, the Korean representative on the Nemotron team, the KAI Foundation Model project was launched by South Korea's Ministry of Science and ICT in partnership with affiliated institutions. Its core goals are to strengthen Korea's capacity to independently develop competitive foundation models and to build a sustainable domestic AI ecosystem.

In practice, the Korean government has invested in large-scale AI compute infrastructure built on NVIDIA GPUs, and through domestic infrastructure and cloud service partners, allocates that compute to a curated set of model-building teams. The intent is clear: let these teams focus on training and improving models without having to build infrastructure from scratch. More importantly, the resulting models don't just serve a handful of flagship projects — they target public-sector applications and create shared capabilities for Korea's entire AI ecosystem, spanning model development, infrastructure, startups, enterprise researchers, and public services.

Chan-Lan recalled the shared moment that ignited Korea's collective passion for AI — the AlphaGo match against Lee Sedol, played on Korean soil. "Ten years ago, all Koreans watched that match on TV. We all understood how significant AI's impact would be." That national consensus is the underlying driver for government investment across the full chain, from infrastructure to model deployment. In his view, a model like KAI — an entire nation collaborating to build the best AI — has no real parallel anywhere in the world.

The KAI initiative reflects a broader global trend of "national team" approaches to training sovereign AI. Unlike the EU's BLOOM project, the UAE's Falcon, or France's Mistral, KAI places greater emphasis on centralized compute allocation and publicly shared outcomes, rather than simply backing commercial companies. The model's advantage is that it avoids duplicated infrastructure investment and lowers the barrier to entry for startups. The potential challenge lies in designing fair allocation mechanisms and appropriate incentives. Korea's choice to build its infrastructure on NVIDIA GPUs — rather than domestically developed chips — also reflects a pragmatic trade-off between hardware sovereignty and rapid deployment: rather than spending years building a homegrown compute foundation, the priority is to become competitive on proven hardware first, then gradually localize the ecosystem.

"Training Models Is Like Cooking": An Engineering-First Philosophy

Motif's technical lead, Dongbin Oh, shared the team's core methodology. He cited a metaphor from chief engineer Dongsuk Kim: training a large model is like cooking. Cooking certainly involves science — the Maillard reaction, molecular gastronomy, all very "scientific" — but most of it depends on engineering: picking quality ingredients at the market, tasting as you go, managing countless details.

"We believe training LRMs (Large Reasoning Models) is similarly more about engineering than science," Dongbin said. That includes acquiring high-quality datasets, ensuring zero configuration errors, monitoring for anomalies, and — most critically — maximizing GPU utilization through system-level optimizations like kernel tuning and parallelization.

On the data side, Dongbin acknowledged that NVIDIA's open-source Nemotron pretraining dataset was "extremely useful" and has effectively democratized LRM pretraining. Motif leaned heavily on it, supplemented by internal data derived largely from public datasets and Common Crawl data processed with basic filtering and deduplication. He was candid that the pretraining data side had "nothing particularly special" about it.

Motif's incremental scaling strategy is also worth noting: start with a very small parameter count (2.6B) to validate hypotheses and recipes, then — once validated — secure more resources and move to the next size (12.7B), scaling up gradually. This "prove it first, then invest more" approach is well-suited to resource-constrained small teams.

Motif's progressive size-scaling strategy (2.6B → 12.7B) is consistent with the academic literature on scaling laws. The Chinchilla laws suggest that for a given compute budget, model parameter count and training token count should follow roughly a 1:20 ratio, rather than simply maximizing parameter count. For small teams, validating data recipes, hyperparameter configurations, and training stability on smaller models enables fast, low-cost "ablation experiments" — keeping the cost of failure manageable before transferring successful findings to larger models. This approach assumes that patterns transfer across scales, which doesn't always hold perfectly, but offers the best risk-adjusted returns under resource constraints. Common Crawl is an internet-scale web crawl dataset that, after deduplication, language identification, and quality filtering, serves as one of the primary sources for most open-source pretraining corpora today.

"Trust Nothing, Verify Everything": A Small Team's Survival Rule

A mental model Dongbin returned to repeatedly was "trust nothing and verify everything." This is tightly coupled to monitoring: the team tracks metrics like MoE expert load imbalance, abnormal activations, and weights, and adjusts training settings immediately when anomalies surface.

He gave a concrete example: Kim encountered an abnormal activation problem early in pretraining. After relentlessly tracing the root cause, he discovered that weight initialization was producing anomalous weight values (bizarre numbers like 2 or -2), which then triggered the abnormal activations in the training dynamics. Fixing the initialization resolved the issue.

This "verify everything" approach comes at a real cost. Dongbin acknowledged that in an era where everyone relies on agentic coding, the team also has to process hundreds of AI-generated PRs — and bugs lurking within them can waste weeks of compute time, translating to millions of dollars in losses and potentially fatal consequences for a small team. Motif's engineers therefore reviewed the codebase, training code, model code, and configurations again and again, going through each of hundreds of AI-generated PRs by hand.

A compelling example: even widely used, mature frameworks like Torch Titan can contain unexpected bugs. Dongbin pointed out that Torch Titan's implementation of DeepSeek^Ad-style expert bias normalizes updates differently than the original paper describes — a discrepancy the coding agent initially missed, caught only through manual review. "This level of dedication requires a higher degree of commitment. We believe that commitment has made an enormous difference."

Post-Training: A Failure-Driven Feedback Loop

For post-training, Motif's core strategy is to build a feedback loop: analyze the model's current state, identify weaknesses and strengths — quantified through benchmarks like terminal bench and HLE as well as internal scenarios.

The team found that Motif's failure patterns were systematic — for example, the model would conclude without a definitive answer, or in terminal agent scenarios wouldn't know how to recover from errors. They then generated datasets specifically targeting those weaknesses and trained on them. Dongbin calls this approach "failure-driven SFT."

For reinforcement learning, Motif followed the general RL strategy from the Nemotron Ultra technical report: train multiple teacher models, then distill all teachers using MOPD. On the tooling side, they used NeMo RL — a simple bridge between their own codebase and NeMo RL was all it took to avoid reinventing a lot of wheels.

The trade-offs are also worth noting. In their first training run, Motif focused heavily on agentic capabilities, because they believe that's the most promising direction for real-world applications. For this, they didn't even prioritize Korean language ability or the "human-likeness" of general conversation — even though as a KAI company they could have emphasized localization. "We only focused on the capabilities we believed were most important for how people actually use models."

"Failure-driven SFT" is fundamentally a data flywheel strategy: systematically identify the model's weak scenarios through benchmark testing and human evaluation, then synthesize or collect high-quality supervised data targeting those scenarios, and use SFT (Supervised Fine-Tuning) to patch them directly. This contrasts with the earlier approach of "general SFT with more data," which often caused regression in areas where the model was already strong. MOPD (Multi-objective Policy Distillation) is a distillation method that merges the capabilities of multiple specialized teacher models into a single student model — different teachers may excel at math reasoning, coding, or long-context understanding respectively, and MOPD uses weighted distillation signals to improve the student model across multiple dimensions simultaneously, avoiding the capability imbalances that single-teacher distillation can produce. This strategy is described in detail in the Nemotron Ultra technical report, and Motif's direct reuse of it saved substantial validation costs.

Differential Attention and Technical Choices: Good Old-Fashioned Science

When asked by an audience member "why differential attention," Dongbin's answer was characteristically grounded. The core idea of differential attention is to reduce attention background noise — analogous to noise-canceling headphones. But the team didn't go with intuition alone: they tested all promising attention variants and ran ablation experiments, and differential attention was the winner.

"We had a lot of options, tested them, and this one was best," host Chris summarized. Dongbin revealed that the team plans to continue using and improving the differential attention family — the "group differential latent attention" in Motif 3 is one example — with a future goal of eventually naming their own "Motif Differential Attention."

Differential Attention was proposed by Microsoft Research in 2024. The core idea is to subtract two independent sets of softmax attention weights from each other, canceling each other's "background noise" so the model focuses more on genuinely relevant tokens. Analogous to how noise-canceling headphones use anti-phase sound waves to eliminate ambient noise, differential attention suppresses low-information-entropy regions in the attention map through this differencing operation. Experiments show significant improvements in long-context understanding, information retrieval, and hallucination reduction, at the cost of a modest increase in parameters and computation (roughly requiring twice the number of attention heads). Motif has further developed this into "Group Differential Latent Attention," combining ideas from DeepSeek's MLA (Multi-head Latent Attention) for KV cache compression, seeking a better balance between noise reduction and inference efficiency.

The Resource Bill: 700+ B200s for Two to Three Months

On the question everyone wanted answered — compute consumption — Dongbin offered an estimate: the entire pretraining infrastructure amounted to around 700-plus GPUs (later confirmed in the chat as approximately 760 B200s), used for roughly two months, with evaluation research and bug-fixing bringing actual pretraining time to about two to three months.

"Not long," the host remarked. "That's exactly the point — this isn't what we'd define as 'a long time' or 'a lot of GPUs.'" Given how recently Motif was founded, completing high-quality work this quickly is the most inspiring part of the whole story.

The model is now accessible via API, Hugging Face, and the team's own Motif Chat service. Summarizing how NVIDIA's Nemotron projects contributed, Dongbin pointed to three things: the Nemotron Ultra technical report provided valuable experimental observations; the NeMo RL framework saved the cost of building and validating from scratch; and the open-source pretraining dataset was something the team relied on heavily.

Looking Ahead: From Agents to Frontier Science

Dongbin acknowledged that Motif 3 still has coverage gaps. The team's next steps are clear: continue scaling the model (they believe larger models unlock broader domain coverage), introduce multimodal capabilities, and expand to wider scenarios — not just agentic coding or science tasks, but an ambition for Motif 4 to handle everything from everyday conversation to frontier scientific discovery.

A 30-person team reaching the open-source top tier with under a thousand GPUs is a story that may not be replicable by every team. But Motif's methodology — engineering over science, verify over trust, focus over breadth — along with the collaborative national ecosystem of KAI behind it, offers a rare and valuable reference point for model building under resource constraints.