Python Development on Mac Big Sur

I don’t like Python. I like developing on a Mac even less. So let’s try both! Whatever doesn’t kill you makes you stronger, right?

I am running Big Sur. I need to set up a Django Project that uses a Postgresql database. I intend to manage postgresql with PgAdmin. I have Anaconda installed and also Brew.

Getting The Tools I Used

• https://www.djangoproject.com/
• https://www.postgresql.org/
• https://www.pgadmin.org/download/pgadmin-4-macos/
• https://brew.sh/
• https://www.anaconda.com/
• https://www.python.org/

Note that I preface terminal commands with “>”

Python

I used anaconda to update python, but got so far- version 3.7.4. Conda cannot currently update beyond that on a mac. Not bad but I had hoped for 3.9.

>conda update python

These extra commands enable conda to get some newer packages, so python did make it to 3.7.9.

• >conda config — add channels conda-forge
• >conda config — set channel_priority strict

The Database

>brew install postgresql

This may also compile Python 3.9 for you- mine did. But where is it? Conda still thinks I am using Python 3.7.4. Running “python –version” also reports 3.7.4. Let’s just assume postgresql knows what it is doing and has its own Python 3.9 running somewhere…

If you get errors about missing directories or not having permissions, run “brew doctor”. At the top of the reams of output will be a couple of key commands you need to run.

Warning: The following directories are not writable by your user:

/usr/local/bin

You should change the ownership of these directories to your user.

>sudo chown -R $(whoami) /usr/local/bin

And make sure that your user has write permission.

>chmod u+w /usr/local/bin

Warning: The following directories do not exist:

/usr/local/Frameworks

/usr/local/sbin

You should create these directories and change their ownership to your account.

>sudo mkdir -p /usr/local/Frameworks /usr/local/sbin

>sudo chown -R $(whoami) /usr/local/Frameworks /usr/local/sbin

By the way, if you see a bunch of warnings, run “brew cleanup” to fix them.

Start the database and connect with PgAdmin

>brew services start postgresql

Next- before you can connect to your database you need to create the user “postgres”

>/usr/local/opt/postgres/bin/createuser -s postgres

Now you can run PgAdmin (make a good password and don’t lose it!) and create a server. What is PgAdmin? It used to be a desktop app- now it is a locally server website, so running PgAdmin will open a browser window. The database you just installed is now running on localhost at the default port of 5432.

How about the code?

I am using the Pycharm IDE. It allows you to directly clone from a git repo and create a project from that code. Since I have a Django (Python) project, I have a requirements.txt file which you must process with pip to install project dependencies. Run:

>pip install -r requirements.txt

(can also do with pip3)

You will also need to ensure your settings.py file and other related parameters match (database is a great example).

More Database Setup

My settings file has things like database name, user, password, host, and port. You can setup your database the rest of the way using the psql command line tool and then verify in PgAdmin. Run “psql postgres” and enter the following at the postgres=# prompts. Put in your database names, etc.

>psql postgres

postgres=# CREATE DATABASE myproject;

CREATE DATABASE

postgres=# CREATE USER myprojectuser WITH PASSWORD ‘putsomethinghere’;

CREATE ROLE

postgres=# ALTER USER myprojectuser CREATEDB;

ALTER ROLE

postgres=# ALTER ROLE myprojectuser SUPERUSER;

ALTER ROLE

postgres=# \q

If you refresh your PgAdmin view of the databases on your server you will see the new one, ‘myproject’. If you explore a little deeper, though, you find no tables under Schemas:public, since we have not made any. Some Django commands will take care of that!

Django manage.py

The manage.py script does many things in Django. Starts your server and application for one. Also creates and updates your database as changes are made. These are called migrations. From your Django project root directory, run:

• >./manage.py makemigrations
• >./manage.py migrate

Have a look at your database now- you should see tables!

Of course, for me I just got errors due to missing imports. I am pretty sure that is simply a code update change I need (maybe a new branch has not been merged to my repo). This brings up another point, though, and that is to make sure your Pycharm project (or whatever IDE) is using the virtual environment, “mine”, so you have all the right dependencies available to your project. Since you already made the environment, you can select it the Python interpreter from that location in the Pycharm UI.

Adding conda environment variables for YOUR environment

So you made a virtual environment to work in using conda. How can you get environment variables specific to that environment to load each time you use it? Read on!

Assume your environment is call “mine”. Run the following:

>conda activate mine

>cd $CONDA_PREFIX

>pwd

(output should show a path with your environment name at the end, like $HOME/anaconda3/envs/mine)

The following set up two files, allowing for environment variables to be set and unset when you activate or deactivate the environment “mine”.

>mkdir -p ./etc/conda/activate.d

>mkdir -p ./etc/conda/deactivate.d

>touch ./etc/conda/activate.d/env_vars.sh

>touch ./etc/conda/deactivate.d/env_vars.sh

Use an editor (I am primitive so I actually use vi) to put in your desired variables and save the file. For example:

# Django

export DJANGO_DEBUG=True

Run this to get your variables to be used:

>conda deactivate

>conda activate mine

>echo $DJANGO_DEBUG

Verify the variable prints out “True”.

Note on DJANGO _SECRET_KEY. There are several invalidate characters that can cause parsing failure of your environment variables. Unfortunately, python command for creating a secret key spew these out, making them unusable as is. The command get_random_secret_key() is simply bad, and uses characters like “)” and “=” which fail when parsing. Some others do, too. Simplest answer is REMOVE ALL PUNCTUATION from your key. Then it will work.

Testing

Running unit tests and coverage reports should be a snap, but there are things to watch out for. First the basics. My Django project is setup by default to use pytest. I am going to use the coverage module for generating reports that show how much of the source code is actually executed during testing. The higher the better, since you do not want to be deploying untested code or relying solely on integration or manual testing. Test early and often!

Add ‘coverage’ to your requirements.txt file!! Re-run it with pip (pip install -r requirements.txt), but make sure you are in the correct virtual environment! I had a problem with this and pip did NOT work right and I wound up getting import errors when running coverage. I used “conda install coverage” and it worked right.

These commands will run tests with coverage and generate an html and text report. Be sure to run from your django project root directory.

>mkdir htmlcov

>coverage erase

>coverage run manage.py test

>coverage report — omit=”*/migrations*,*/test*”

(for text output to console)

>coverage html — omit=”*/migrations*,*/test*”

(for nice html report files saved to ./htmlcov directory you must make)

Note: this editor will not let me type two dashes in a row. In the above commands, there is a space after html, followed by two dashes and omit.

The “omit” argument specifies directories or file name patterns to skip when running coverage. You only care about coverage on production code, so skip database migrations and test files since they will artificially inflate your coverage results.

Django Certificate Problems On Mac

Now this is special. My Django app makes a call to an external web service using HTTPS (the US Postal Service address validator), and gets a certificate error CERTIFICATE_VERIFY_FAILED. Turns out that Django on a Mac using an anaconda virtual environment is MISSING root certificate authorities, so it cannot validate a server certificate. This is the fix (run in your environment):

>conda install -c conda-forge certifi

(It really is ‘certifi’ at the end). Command output explains what it does:

The following NEW packages will be INSTALLED:

ca-certificates conda-forge/osx-64::ca-certificates-2020.12.5-h033912b_0

python_abi conda-forge/osx-64::python_abi-3.7–1_cp37m

The following packages will be UPDATED:

certifi intel::certifi-2020.6.20-py37hefe589e~ → conda-forge::certifi-2020.12.5-py37hf985489_1

openssl intel::openssl-1.1.1h-h1de35cc_0 → conda-forge::openssl-1.1.1i-h35c211d_0

After this, I restarted the Django server and everything worked properly. Later on I saw a certificate error again (after adding the forge channel to conda), so I re-ran “conda install -c conda-forge certifi” to fix it. Again.

Share Your Tips

The Mac is a challenge to develop on. I know since others on my team are using Linux and not having these problems. Do you have any tips to share? Add them to the comments! Now don’t just say “run linux”.