ModernBERT Base

ModernBERT-base is a fill-mask model, a modernized bidirectional encoder-only Transformer (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. It leverages recent architectural improvements including Rotary Positional Embeddings (RoPE) for long-context support, local-global alternating attention for efficiency on long inputs, and unpadding with Flash Attention for efficient inference.

Key Strengths

ModernBERT-base (149M parameters) achieves strong results across natural language understanding, general retrieval, long-context retrieval, and code retrieval tasks. On GLUE, it scores 88.4, surpassing similarly-sized encoder models. For general retrieval (BEIR, DPR setting), it achieves 41.6, and for code retrieval, it sets new state-of-the-art results on CodeSearchNet (56.4) and StackQA (73.6). In multi-vector retrieval (ColBERT setting) on long-context out-of-domain data (MLDR_OOD), it reaches 80.2, significantly outperforming prior models.

Evaluation Results

Task	Metric	ModernBERT-base
GLUE	NLU	88.4
BEIR (DPR)	Retrieval	41.6
MLDR_ID (DPR)	Long-context retrieval	44.0
CodeSearchNet	Code retrieval	56.4
StackQA	Code retrieval	73.6
BEIR (ColBERT)	Multi-vector retrieval	51.3
MLDR_OOD (ColBERT)	Long-context multi-vector retrieval	80.2

Architecture and Training

The model uses a Pre-Norm Transformer with GeGLU activations, was pre-trained up to 1,024 tokens then extended to 8,192 tokens, and trained on 8x H100 GPUs. Training data is primarily English and code, so performance may be lower for other languages. The model is released under Apache 2.0 license.