byte pair encoding
Word Rarity
Rare
Frequency: Zipf frequency 2.94 in modern English
27
Scrabble Score
16
Letters
4
Syllables
13
Unique Chars
Meaning
noun
A lossless data compression algorithm that iteratively replaces the most frequent pair of adjacent bytes in a sequence with a new byte not already present in the data.
Countable; Uncountable
noun
A subword tokenization method that iteratively merges the most frequent pairs of adjacent characters in a corpus to form longer and more meaningful tokens, typically until a predefined vocabulary size is reached.
Countable; Uncountable