Computronium Blog
Published in

Computronium Blog

Text Segmentation

Normalization, Tokenization, Sentence Segmentation + Useful Methods

Photo by Sergey Zolkin on Unsplash

What does normalizing a text do?

We have previously called this method .lower() to turn all of the words lowercase, so that strings like “the” and “The” both become “the”, so we don’t double count them.

What if we wanna do even more?

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store