How to build tokenizer from scratch — Part 1

Esther began learning how to build the GPT tokenizer with Andrej Karpathy. As she was learning, she realized, “OMG! The content is AMAZING! But why is it so… inhuman?” (YouTube content greater than 30min = inhuman). The video is a scary 2 hours. Most of the time, people don’t even dare to start. Can we make the learning less scary and more fun?

So she created this blog so people can learn through playing!

• Blog post: https://byte-pairing.vercel.app/
• Github: https://github.com/esther119/build-tokenization-from-scratch

Since Medium doesn’t allow interactive elements, I put my blog post teaser here. I’m looking for more animation and interactive ideas to improve this blog post!