Starting with pyenv & poetry for Python development
Python is one of the most widely used programming languages within the data science community. Although Python has simplicity in coding and a vast set of libraries for data science, it has its complication and frustration managing multiple Python interpreter versions and dependent packages.
Why does this concern? If you are working on multiple Python projects and if these projects are working on a shared environment, updating the dependent library for one project may affect another one.
In my local environment, I started and switched back and forth with Venv, Pip, Conda, Pyenv, Pipenv, and Poetry.
Venv: This is a Python module, is used to create lightweight virtual environments.
Pip: This is the package installer for Python. You can use it to install packages from the Python Package Index and other indexes.
Conda: Conda is a package and environment management system, can create, save, load and switch beet environments in your local machine.
Pyenv: This is a python environment manager, allows to install and run off multiple python installations on the same machine.
Pipenv: This is a package manager is used to managed Python projects dependencies.
Poetry: This is a recent one and popularly known as a simple Python tool for project dependency management.
pyenv
Setup
Follow these steps to setup pyenv in your environment. The instructions are for Mac OS using homebrew. For other operating systems, refer pyenv installation docs.
# install pyenv using homebrew
brew install pyenvThe pyenv environment settings are stored in ~/.pyenv. If you need to change the default environment, set PYENV_ROOT variable to preferred directory location. I kept as default.
export PYENV_ROOT={MY_PREFERRED_LOCATION}
To keep this setting permanat add the export to your shell profile. E.g: .zprofile or .zshrc
export PYENV_ROOT=”$HOME/.pyenv”
export PATH=”$PYENV_ROOT/bin:$PATH”
eval “$(pyenv init — path)”Create new terminal session for changes added to teh environment.
Get start
Below are few CLI commands for managing environment with pyenv
# list all available python versions 3.8 and 3.9
pyenv install — list | grep “ 3\.[89]”# this will display
3.8.0
3.8-dev
3.8.1
3.8.2
3.8.3
3.8.4
3.8.5
3.8.6
3.8.7
3.8.8
3.8.9
3.8.10
3.8.11
3.8.12
3.9.0
3.9-dev
3.9.1
3.9.2
3.9.4
3.9.5
3.9.6
3.9.7# install 3.9.7 python version
pyenv install 3.9.7# list all python versions in the local environment
pyenv versionssystem
3.8.5
3.8.7
3.9.1
3.9.5
3.9.6
* 3.9.7 (set by $HOME/.pyenv/version)# set global python environment
pyenv global 3.9.7# check with python version is in use
pyenv version
This will out put the global python version like below.
3.9.7 (set by $HOME.pyenv/version)
# set local python environment
cd $Home/code/lab
pyenv local 3.8.7# check the version
pyenv version
This will output local Python version created by pyenv. This will create local file .python-version contains the local version number like below
3.8.7 (set by $Home/code/lab/.python-version)
poetry
Setup
The setup instructions are for MacOS environment. For other environment, refer https://python-poetry.org/docs/
# You can install using brew also but it is not recommended by poetry community. Use the following method to setup the poetry.
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -poetry env info
This will install the poetry files in <HOME>.poetry folder. You can add path for poetry binary file in .zprofile or .zshrc
# add following line in .zprofile or .zshrc
export PATH=”$HOME/.poetry/bin:$PATH”Manage poetry configuration
# Check the poetry version
poetry — versionIf you see something like Poetry version <VERSION_NUMBER> then your installation is good and you can start using poetry.
# if you need to update poetry installtion
poetry self updateYou see the update process start downloading new version if there is one or you will see message like “You are using the latest version”.
Manage configuration
# list the configuration settings
poetry config — listThis will out put the poetry configurations in your environment like below.
cache-dir = “<CACH_DIR>”
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = false
virtualenvs.path = “{cache-dir}/virtualenvs” # <CACH_DIR>/virtualenvsUsing Visual Code, creating a virtual environment folder inside the project makes life easier. If your poetry configuration has a false value for “virtualenvs.in-project” settings, you can change the following.
poetry config virtualenvs.in-project true
and confirm the value is changed now using config list command again. On creating virtual environment (described in the next section), poetry will create .venv folder and add the dependent library files here. Visual code should recognize .venv folder and load required libraries for the project source files. By default in MacOS the required files are stored in ~/Library/Caches/pypoetry/virtualenvs and I had challenge this way because my cache cleanup tools started removing them.
poetry config — list
Get start with creating project
You can create new project using
poetry new code_lab
This will create directory name code_lab and create bunch of project files. The Poetry project scaffold includes the following:
pyproject.toml — the definition file for the project. Poetry manages this definition for you. If you know what you’re doing, you can edit this file directly, but most of the time you won’t need to.
README.rst — an empty README file in ReStructuredText format, the file format used for Python documentation. (There is no rule that says you must use .rst format for your docs; you can use Markdown for simpler cases.)
tests — a subdirectory with scaffolding for unit tests. If you aren’t in the habit of writing tests for your new projects, you should be!
And finally, a subdirectory with the project name that contains the code for your project.
If you want to have separate Python version using penv local command and start with simple project directory structure then I do with following
# go to your project workspace if you have one
cd workspce# create your project folder
mkdir code_lab# go to code lab directory
cd code_lab# set the local python version
pyenv local 3.8.12# initialize poetry envrinment
poetry init# This command will guide you through creating pyproject.toml# check the files in the directory
ls -al# this should show pyproject.toml and .python-version
The .python-version file should contain
3.8.12
The pyproject.toml file should contain
[tool.poetry]
name = “code_lab”
version = “0.1.0”
description = “Python 3.8 lab environment for practicing code”
authors = [“Your Name”]
license = “Apache 2.0”[tool.poetry.dependencies]
python = “^3.8”[tool.poetry.dev-dependencies][build-system]
requires = [“poetry-core>=1.0.0”]
build-backend = “poetry.core.masonry.api”# creating shell environment inside the project folder
poetry shell# this will create .venv folder inside the project directory and following files and directories will be populatedbin -> python environment files
lib -> required libraries
pyenv.cfg -> environment configurations
Adding module (E.g: pandas)
This section shows command and output adding pandas module into a poetry project.
# search for package
poetry search pandas# shows list of pandas related libraries in the repo
pandas (1.3.3)
Powerful data structures for data analysis, time series, and statisticspandas3 (0.0.1)
Boto3 extension to help facilitate data science workflows with S3 and Pandaspandas-alchemy (0.0.2)
SQL based, pandas compatible DataFrame & Seriespandas-datacube (0.0.3)
A package allowing to download datacubes into pandas data framespandas-vectors (0.1.1)
convenience functions for dealing with vectors in panda dataframespandas-ga (0.1)
Reading Google Analytics report into Pandas dataframe.............# add package
poetry add pandas# start adding package and it’s dependencies into your current project and create poetry.lock file with package & version information file it is not created before. # If the poetry.lock file exist then “add” option will update the poetry.lock file added package.Updating dependencies
Resolving dependencies… (0.5s)
Writing lock filePackage operations: 5 installs, 0 updates, 0 removals
• Installing six (1.16.0)
• Installing numpy (1.21.1)
• Installing python-dateutil (2.8.2)
• Installing pytz (2021.1)
• Installing pandas (1.3.3)
Module versions
If you need to install specific version, poetry provides multiple versions contraints.
Exact version: with this configuration, if there is new pendulum version release poetry will not update, when you apply “poetry update”
# adding pendulum module version 2.1.2 only
poetry add pendulum=”2.1.2"Caret version: With the caret ^ character, Poetry will update to any new version that does not modify the left-most non-zero digit in the major, minor, patch grouping.
^1.2.3 version allowed: >=1.2.3 <2.0.0
^1.2 version allowed: >=1.2.0 <2.0.0
^1 version allowed: >=1.0.0 <2.0.0
^0.2.3 version allowed: >=0.2.3 <0.3.0
^0.0.3 version allowed: >=0.0.3 <0.0.4
^0.0 version allowed: >=0.0.0 <0.1.0
^0. version allowed: >=0.0.0 <1.0.0
# adding pendulam module between >=2.1.2 and < 3.0.0
poetry add pendulum=”^2.1.2"
Tilde version: The tilde ~ character tells Poetry to allow minor updates. If you specify a major, minor, and patch version, only patch-level changes are allowed. If you specify a major and minor version, again only patch-level changes are allowed. If you specify only a major version, then minor- and patch-level changes are allowed.
~1.2.3. version allowed: >=1.2.3 <1.3.0
~1.2. version allowed: >=1.2.0 <1.3.0
~1. version allowed: >=1.0.0 <2.0.0
# adding pendulum using tilde version between >=2.1.0 and <2.2.0
poetry add pendulum=”~2.1.2"
Wildcard version
This option allows any version number is allowed at the wildcard position.
*. version allowed: >=0.0.0
1.*. version allowed: >=1.0.0 <2.0.0
1.2.*. version allowed: >=1.2.0 <1.3.0
# adding pendulam module using wildcard for any version
poetry add pendulum=”*”
For additional poetry usage the detail documentation is included here https://python-poetry.org/docs/
