Byte-pair encoding builds a tokenizer from data. Start with single characters, count which two neighbours occur together most often, glue them into one new token, and repeat. Frequent words end up as one token, rare ones as a few familiar pieces.
Byte-pair encoding builds a tokenizer from data. Start with single characters, count which two neighbours occur together most often, glue them into one new token, and repeat. Frequent words end up as one token, rare ones as a few familiar pieces.