Byte-Pair Encoding (BPE)

Training: scan a large corpus, repeatedly find and merge the most frequent pair of adjacent symbols, and record every merge as a rule. Stop when the vocabulary reaches a target size.
Encoding (inference): given new text, apply the learned merge rules in order to produce token IDs.

Master Byte-Pair Encoding (BPE) — the subword tokenization algorithm powering GPT, LLaMA, and most modern LLMs. Learn how it works step by step, why it beats word-level and character-level approaches, and how it handles rare and unknown words.