Installing NLTK(Natural Language Toolkit) in MacOS Catalina 10.15

My first step towards learning about AI platform — Understanding NLTK

I would to explain how I successfully installed NLTK Python3 environment in my iMac.
In general, installation would be by a single command — pip3 install -U nltk. You can follow https://www.nltk.org/install.html.

But in my case, due to missing packages it took me few extra steps as below:

Ruhees-iMac:~ $ pip3 install -U nltk
Collecting nltk
Using cached nltk-3.5.zip (1.4 MB)
Requirement already satisfied, skipping upgrade: click in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from nltk) (7.1.2)
Requirement already satisfied, skipping upgrade: joblib in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from nltk) (0.15.1)
Collecting regex
Using cached regex-2020.6.8.tar.gz (690 kB)
Collecting tqdm
Using cached tqdm-4.46.1-py2.py3-none-any.whl (63 kB)
Using legacy setup.py install for nltk, since package ‘wheel’ is not installed.
Using legacy setup.py install for regex, since package ‘wheel’ is not installed.
Installing collected packages: regex, tqdm, nltk
Running setup.py install for regex … error
ERROR: Command errored out with exit status 1:
command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘“‘“‘/private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-install-2ap7ryph/regex/setup.py’”’”’; __file__=’”’”’/private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-install-2ap7ryph/regex/setup.py’”’”’;f=getattr(tokenize, ‘“‘“‘open’”’”’, open)(__file__);code=f.read().replace(‘“‘“‘\r\n’”’”’, ‘“‘“‘\n’”’”’);f.close();exec(compile(code, __file__, ‘“‘“‘exec’”’”’))’ install — record /private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-record-dchaguzo/install-record.txt — single-version-externally-managed — compile — install-headers /Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/regex
cwd: /private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-install-2ap7ryph/regex/
Complete output (17 lines):
running install
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64–3.8
creating build/lib.macosx-10.9-x86_64–3.8/regex
copying regex_3/__init__.py -> build/lib.macosx-10.9-x86_64–3.8/regex
copying regex_3/regex.py -> build/lib.macosx-10.9-x86_64–3.8/regex
copying regex_3/_regex_core.py -> build/lib.macosx-10.9-x86_64–3.8/regex
copying regex_3/test_regex.py -> build/lib.macosx-10.9-x86_64–3.8/regex
running build_ext
building ‘regex._regex’ extension
creating build/temp.macosx-10.9-x86_64–3.8
creating build/temp.macosx-10.9-x86_64–3.8/regex_3
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c regex_3/_regex.c -o build/temp.macosx-10.9-x86_64–3.8/regex_3/_regex.o
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
error: command ‘gcc’ failed with exit status 1
— — — — — — — — — — — — — — — — — — — —
ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘“‘“‘/private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-install-2ap7ryph/regex/setup.py’”’”’; __file__=’”’”’/private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-install-2ap7ryph/regex/setup.py’”’”’;f=getattr(tokenize, ‘“‘“‘open’”’”’, open)(__file__);code=f.read().replace(‘“‘“‘\r\n’”’”’, ‘“‘“‘\n’”’”’);f.close();exec(compile(code, __file__, ‘“‘“‘exec’”’”’))’ install — record /private/var/folders/hx/vz4vps157vx9f57_9l4ky8w80000gn/T/pip-record-dchaguzo/install-record.txt — single-version-externally-managed — compile — install-headers /Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/regex Check the logs for full command output.

As you can see from above, it complained :

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

It means that you need to install XCode command line !

How to solve?:

1. Run xcode-select --install
2. If that doesn’t work, sudo it. sudo xcode-select --install
3. If all else fails, try to reset xcode-select. sudo xcode-selct --reset
4. Now try to install NLTK again.

Next blocker was, while downloading NLTK packages, it complained due to SSL certificates check failure!

Ruhees-iMac:~$ python3
Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.download()
showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml
True
>>> nltk.download('all')
[nltk_data] Error loading all: <urlopen error [SSL:
[nltk_data] CERTIFICATE_VERIFY_FAILED] certificate verify failed:
[nltk_data] unable to get local issuer certificate (_ssl.c:1108)>
False
>>>

Now there are 2 options :
(a) Install ssl certificates and rerun
(b) Disable the ssl and continue the download

I would try SSL certificates install on my iMac some other time and write a post on it. So I just went ahead with disabling the ssl check and downloaded the packages. This is how I did it :

Open Python and run the below:

import nltk
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
pass
else:
ssl._create_default_https_context = _create_unverified_https_context
nltk.download()

Now try again running pip3 install -U nltk

If it still didn’t work and you came across other issues, please share it 😃
Enjoy Coding !!