ModernBERT
An improved encoder model in the BERT architecture family released in late 2024. While retaining the advantages of bidirectional attention, it incorporates modern design elements such as Rotary Position Embeddings (RoPE) and alternating local/global attention, making it suitable for lightweight encoder tasks.
Core Facts
Timeline (last 90 days)
A Reddit user fine-tuned ModernBERT with 138,000 financial news headlines, and the training loss kept decreasing but the test set AUC stayed between 0.49 and 0.51.
ModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
ModernBERT is an improved version of the BERT architecture released in late 2024, introducing modern designs such as Rotary Position Embedding (RoPE) and alternating local/global attention.
All Facts (4)
ModernBERT is an improved version of the BERT architecture released in late 2024, introducing modern designs such as Rotary Position Embedding (RoPE) and alternating local/global attention.
65%UnverifiedSparky的创造者Alexis之前的工作是训练大语言模型（encoder-only架构，类似ModernBERT）
85%UnverifiedModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
50%UnverifiedA Reddit user fine-tuned ModernBERT with 138,000 financial news headlines, and the training loss kept decreasing but the test set AUC stayed between 0.49 and 0.51.
50%