minbpe
Minbpe is a minimal and clean Python implementation of the Byte Pair Encoding algorithm, which is widely used in large language models. The repository includes basic, regex-based, and GPT-4 compatible tokenizer classes that perform vocabulary training, tokenization of text, and token conversion.
minbpe
55karpathy · AI / LLM / RAG
Common commercial products in this category: ChatGPT, Claude, Gemini
- On the given text, you train the tokenizer vocabulary and merges
- Convert text to token arrays and convert token back to text
- Save the trained tokenizer model to disk and then reload it
- • Last update 2 years ago
- • NeutralHacker News
55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer with PyTorch/CUDA
“This adds PyTorch/CUDA training support to Andrej Karpathy's minbpe. It takes 2min 28sec (148 seconds) on an RTX4090 to train the BasicTokenizer with a vocab_size of 512 on 307MB of Enron emails. The original code takes 2hrs 15min (8076 seconds) on an M2 Air with Python 3.11 to…”
- • NeutralGitHub Issue
Alternative to bpe
“Maybe I am completely wrong, but to me using something like bpe to build an encoding for text feels stupid. Sure, it is a fairly easy way and it will build an encoding that is efficient in terms of sequence length, but is that the only requirement for such an encoding? Would…”
My note
Alternative pages
More in: AI / LLM / RAG
- rea★ 412,812 · today
- ECC★ 293,487 · yesterday
- hermes-agent★ 258,923 · today
- ponytail★ 181,048 · 2 days ago
- graphify★ 139,557 · today
- MoneyPrinterTurbo★ 136,645 · today