Let's build the GPT Tokenizer

Name: Let's build the GPT Tokenizer
Description: Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Andrej Karpathy

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.

8 Hacker News mentions · 2024–2025

Topics

LLMs & generative AI
Computer vision, speech & NLP
Machine learning & deep learning

Watch on YouTube

Explore more videos