I bought a Macbook recently, and I’ve been doing some of the courses from Wash U’s Machine Learning Specialization on Coursera. They’re pretty keen on Python, so I needed to get a proper Python environment setup. On OS X, doing this well is harder than it sounds.
The key is that OS X (El Capitan, at least) already comes with Python (2.6 and 2.7) pre-installed, and if you mess with them, things will break. I wanted to install an up to date version of both Python 2 and Python 3 though, preferably without hosing everything else. What you want for this is a good language version manager, a la RVM or rbenv, considered in the Ruby community to be totally standard practice, but surprisingly less common elsewhere.
Fortunately recently a Python option has been steadily gaining acclaim for this exact problem: Pyenv.
Unfortunately, it doesn’t really work out of the box on OS X. Let’s start with the easy bits:
- If you want to install Pyenv, you’ll want to use Homebrew. If you don’t have Homebrew, check out this guide: https://coolestguidesontheplanet.com/installing-homebrew-on-os-x-el-capitan-10-11-package-manager-for-unix-apps/
- Now, just install with Homebrew: brew install pyenv.
- You’ll also need readline, a standard GNU command-line management library that’s not included by default in OS X: brew install readline
Now, you want to download your python of choice with pyenv, and turn it on. Easy! Just run pyenv install 2.7.11 (or any other version number).
Sadly, this won’t work:
ERROR: The Python zlib extension was not compiled. Missing the zlib?
This is documented in the pyenv FAQ however, which recommends setting CFLAGS=“-I$(xcrun — show-sdk-path)/usr/include” when you’re installing. This sets options used by the Python compiler to build your new Python install, adding the XCode developer tools path first, so that the zlib libraries can be found.
This should now successfully install.
Undocumented though, is that this is not enough to give you a truly working Python install, especially if you want to install packages later on that require native compilation too.
I ended up with a whole stream of errors trying to install Jupyter and it’s dependencies, particularly trying to get Sqlite extensions working happily. You’ll quickly start hitting more errors like:
Symbol not found: _PyUnicodeUCS2_DecodeUTF8
Undefined symbols for architecture x86_64: “_sqlite3_enable_load_extension”
It look quite a bit of googling to get a working answer, and I’ll happily admit this is somewhat cargo-culted from a whole series of StackOverflow answers and incomplete Github issues. C compilation flags are not my strong suit.
On the other hand though, for me, it now works nicely:
Note the slashes at the end of each line: this is one single command. To explain a little of how I understand this to be working:
- CFlags -I is ensuring that readline header files, the correct openssl header files, and the full XCode development environment libraries are available to the compiler.
- LDFlags is then ensuring readline and the correct openssl actual libraries are available to the linker too.
- Python configure opts is telling the compiler to build a version of Python that uses UCS2 (not UCS4), so that it works with pre-compiler binaries pip installs that are expecting this.
For SQLite specifically, I also needed to install SQLite through Homebrew, and the built-in OS X version doesn’t allow extension loading. That, plus the above pyenv setup seems to have got me back on the straight and narrow.
Phew! Again, I’m not an expert in this, but I have just spent a few hours fudging at it to get my Python environment working, and hopefully this’ll give you enough pointers to get there yourself too. Good luck!