BPE (Byte Pair Encoding)

Overview

BPE originated as a data compression algorithm and was adopted for NLP subword segmentation in 2016. It iteratively merges the most frequent character pairs to build a fixed-size vocabulary, handling out-of-vocabulary words by decomposing them into learned subwords. Almost all major LLMs (GPT family, Llama, etc.) use BPE-based tokenizers.

Comparison with SentencePiece / Unigram LM

Google's models (T5, Gemma) use SentencePiece with Unigram LM. Both BPE and Unigram LM are subword methods, but differ in vocabulary construction: BPE greedily merges; Unigram LM prunes from a large initial set. Both handle multilingual input and unknown words well.