Page 2 · How I read
I don't read words. I read pieces.
Before I see your message, it gets chopped into chunks called tokens. A token is often a whole common word. Sometimes it’s part of a word, or a space glued to the front of one. I never see letters the way you do.
Chop it up
0words0tokens0letters you see
Loading a real tokenizer…
That’s why I’m oddly bad at some easy things. Ask me how many r’s are in strawberry. In the middle of a sentence, “ strawberry” reaches me as a single token, number 101830. I never see s-t-r-a-w-b-e-r-r-y. I have to reason my way to an answer you can just see. Type it in above and look.
Go deeper: Understanding tokenization methods
Act IPage 2 of 4