Motif

Korean AI startup Motif, with a team of 30 people using fewer than 1,000 GPUs, trained a model that ranks in the top five on the Artificial Analysis Intelligence Index open-source model leaderboard.

Motif adopts a progressive scaling strategy, first validating hypotheses and recipes with a 2.6 billion parameter model, then scaling up to a 12.7 billion parameter model after successful validation.

Motif relies heavily on NVIDIA's open-source Nemotron pretraining dataset, supplemented by internal data from public datasets and processed Common Crawl data that has undergone basic filtering and deduplication.

The Motif team encountered abnormal activation issues in the early stages of pretraining, the root cause being anomalous weight values (such as 2 or -2) caused by weight initialization, which was resolved after fixing the initialization method.

In the post-training phase, Motif adopts a "failure-driven SFT" strategy: identifying model weaknesses through benchmark testing and internal scenarios, then generating targeted datasets for training.

In the reinforcement learning phase, Motif follows the general RL strategy from the Nemotron Ultra technical report: training multiple teacher models, then distilling all teachers using MOPD (Multi-Objective Policy Distillation).

Motif uses the NeMo RL toolkit, requiring only a simple bridge implementation between its own codebase and NeMo RL.

Motif's first training run was highly focused on agentic capabilities, without prioritizing Korean language ability or the anthropomorphic quality of general-purpose conversation.

After testing all promising attention variants through ablation experiments, Motif selected differential attention, and developed "grouped differential latent attention" in Motif 3.