YaRN

Qwen3-27B natively supports a long context of approximately 260,000 tokens, which can be extended to the 1,000,000-token scale through technologies such as YaRN.

YaRN is a context window extension method for Transformer models using RoPE that can extend context without full fine-tuning by applying differentiated scaling strategies to different frequency dimensions.

pnpm achieves globally shared storage of dependencies through hard links and symbolic links, significantly reducing disk usage and installation time compared to npm and yarn.

YaRN (Yet another RoPE extensioN method) is a positional encoding extension technique proposed in 2023 that enables models to handle sequences longer than those seen during training by dynamically adjusting the frequency components of positional encoding.

YaRN enables models to extend their effective context length by several to tens of times without retraining or with only minimal fine-tuning, by applying differentiated scaling strategies to different frequency components in RoPE.

RoPE has been widely adopted by mainstream models such as LLaMA, Qwen, and Mistral, and has given rise to length extrapolation variants such as YaRN and LongRoPE.

Current mainstream long-context solutions generally use RoPE combined with extrapolation techniques such as YaRN and LongRoPE.

YaRN (Yet another RoPE extensioN) is an extension method for Rotary Position Embedding (RoPE), proposed by Bowen Peng et al. in 2023, which applies differentiated scaling strategies to high-frequency, mid-frequency, and low-frequency components respectively, enabling models to extend their effective context length several times over without retraining or with only minimal fine-tuning.

Qwen3.8 Flash Next has a native context window of 262K tokens, which can be extended to 1 million tokens using YaRN technology

Qwen3.8 Flash Next natively supports multimodal processing of text, images, and video, with a native context window of 262,000 tokens, extendable to 1 million tokens via YaRN technology.

The model natively supports a 256K token context and can be scaled to approximately 1 million tokens via 4x scaling using YaRN technology.

YaRN performs grouped scaling on the frequency components of RoPE, keeping high-frequency components unchanged to preserve local positional precision, interpolating low-frequency components to accommodate longer contexts, and introducing a temperature scaling factor.

YaRN is an extension method for Rotary Position Embedding (RoPE), proposed by Bowen Peng et al. in 2023, which extends context length by applying differentiated scaling strategies to different frequency components.

Qwen3 27B natively supports a context of 262,000 tokens and can be extended to 1,000,000 tokens via YaRN extrapolation.

The YaRN method extends the effective context length while preserving the precision of short-range positional relationships by applying differentiated scaling strategies to frequency components of different ranges.

The FEIHOA service extends the context via YaRN technology, achieving context support of 1M (million-level) tokens.

The industry employs techniques such as Rotary Position Embedding (RoPE) and its extrapolation methods (e.g., YaRN, NTK-aware scaling), FlashAttention, and sparse attention to address long-context challenges.