Pinterest and the visual web 🖼
Once upon a time, people browsed the world wide web through human-curated directories, like this one:
Now that we can handle characters, let’s move on to words.
A critical task for query and document understanding is breaking up the text into a sequence of words. We call these words tokens — but as we’ll see in a moment, tokens include strings that aren’t necessarily words that…