🌓 Toggle Theme
Tokenizer Visualizer
Interactive companion to
First Came the Tokenizers Blogpost
See how text gets broken down into tokens
Word-level
Character-level
Subword (BPE-style)
Hello world! Let's explore how tokenizers work.
8
Tokens
46
Characters
ML Example
Pangram
With Emojis
Code