Ansible, Github, and a Failed to validate the SSL certificate story

Recently Github changed their cryptography policies for SSL connections, something they had previously announced would be deprecated but took effect last week: https://githubengineering.com/crypto-removal-notice/. Most people around the world probably did not even notice. I, on the other hand, was running the ansible-galaxy install -r provisioning/requirements.yml and all of the sudden it stopped working.

When you’re running a command using a tool that you didn’t touch, didn’t update, did make a single change to, and then it stops working out of the blue from one minute to the other, that can makes you go mad. What happened? What did I do? What changed? And of course, in a situation like this, there won’t be many examples online that are relevant or helpful. This is when I rely on local communities and other developer to help keep me sane.

So let’s go back in time to the beginning and tell you what I did to fix the issue. It started when I ran the ansible-galaxy install -r provisioning/requirements.yml command and got the following error:

ychaker ~/dev/vagrant/vagrant-node$ ansible-galaxy install -r provisioning/requirements.yml -f
- downloading role 'base', owned by bearandgiraffe
- downloading role from https://github.com/bearandgiraffe/ansible-role-base/archive/master.tar.gz
 [ERROR]: failed to download the file: Failed to validate the SSL certificate for github.com:443. Make sure your managed systems have a valid CA certificate installed. You can use validate_certs=False if you
do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/ansible, /usr/local/etc/openssl. The exception msg was: [SSL:
TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590).

This is a weird error to get in a situation like this. I, personally, have no hand in how ansible installs roles, and how it downloads them from github. The fact that the same command had worked 5 minutes prior was the weirdest part. I didn’t change networks, go behind a VPN, or anything of that sort. I didn’t upgrade my OS, or do anything remotely related to something that could affect how ansible is connecting to github, so what gives?

I googled the error and got some results but nothing actually related to what I was seeing. So that was frustrating. I couldn’t find anything to guide me towards an answer. I tried multiple variations, checking for answers related to ansible, or any related to github itself, or the combo of both, or even just SSL things related to OS X. I wasn’t getting any answers. So I posted the error in a local dev group and someone had pointed me to github engineering blog post. That was the first sign towards progress, but it still wasn’t too helpful because the post didn’t mention anything I could personally do to fix the problem. So either the ansible team was not aware of these change and fucked up, or something else was up.

I chose to give the ansible team the benefit of the doubt. So I considered waiting a few minutes because some posts online suggested that this error some times is caused by a network issue, and waiting usually resolves it. I waited 5 minutes, and hour, and even 5 days, but the problem persisted. So that definitely was not the answer.

Continuing to give the ansible team the benefit of the doubt, I considered that I might have a very old version installed so I ran brew upgrade ansible and was disappointed to see that I already had the latest version. So what about openssl? Nope, that’s also on the latest version, at least the latest version brew knows how to install. What about python? That’s also the latest version brew knows how to install. Well… to an extent.

Having worked with python previously, but not since I got the current laptop, I knew that we’ve got python2 and python3 that are both widely in use. So when the version I was getting was 2.7.10, I knew that I was still on the default system version of python that ships with OS X.

This will eventually lead me towards the right path. I won’t bore you with all of the steps that I took in order to get to a place where I no longer was seeing the certificate error, instead I will share straight away the steps that you need to take to get it to work on your end.

First, you need to uninstall ansible, install python3, pip, and then install ansible using python3. Here’s how:

brew uninstall ansible
brew install python3
sudo pip3 install ansible
sudo pip install virtualenv
virtualenv py3-ansible
ansible --version | grep python

At the end, you should see something like this come up in the terminal:

python version = 3.6.4

That’s it!

It did take me a while to get to this solution, but I hope this will help you save a lot of time. I must shout out folks in the #ansible freenode channel (yup, people are still using IRC), who pointed me to this post that was the missing link for me in order to get it all to work together.