Glitch Tokens - Computerphile

Name: Glitch Tokens - Computerphile
Description: Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.

Computerphile

Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.

HN points to it as the accessible explanation of why glitch tokens exist whenever tokenizers or odd GPT behavior come up.

7 Hacker News mentions · 2023–2023

Topics

LLMs & generative AI
Computer vision, speech & NLP

Watch on YouTube

Explore more videos