Pre-training

Overview

Pre-training is the first LLM development phase: self-supervised next-token prediction on trillions of tokens from web pages, books, and code. The model acquires language structure, grammar, world knowledge, and coding ability. The enormous GPU hours required mean almost all organizations start from an existing pre-trained model.

Frontier vs open models

OpenAI, Anthropic, and Google conduct proprietary large-scale pre-training and offer API access. Open-weight models like Llama and Qwen release pre-trained weights publicly, enabling organizations to fine-tune from a capable base without pre-training costs.