OPT 125M

OPT-125M is a text-generation model based on a decoder-only transformer architecture, pretrained with a causal language modeling objective. It is part of the Open Pre-trained Transformers (OPT) suite developed by Meta AI, which includes models from 125M to 175B parameters. The 125M variant is designed for efficient zero- and few-shot learning, and its larger counterpart (OPT-175B) has been shown to be comparable to GPT-3 in performance while requiring only one-seventh the carbon footprint to develop (Zhang et al., 2022).

Training Data

The model was pretrained on a 180B-token corpus (800GB) combining BookCorpus, CC-Stories, subsets of The Pile, Pushshift.io Reddit data, and CCNewsV2. Texts are tokenized using the GPT-2 byte-level BPE with a vocabulary of 50,272 tokens, and the model processes sequences of up to 2,048 consecutive tokens.

Key Strengths

• Open weights allow full reproducibility and research, addressing limitations of closed API-only models.
• Performance comparable to GPT-3 on many NLP benchmarks (translation, question-answering, cloze tasks) in zero- and few-shot settings.
• Trained with efficient methods, reducing compute requirements relative to similar-scale models.

Limitations

As noted in the original model card, the training data contains unfiltered internet content, leading to potential biases, toxicity, and hallucination in generations. The model is not immune to the known issues of large language models, and its outputs should be evaluated critically.