Sane Python environment Part 4, virtualenv
Managing Python project environments with virtualenv
2019-11-10

While pyenv provides isolation for the system Python and pipx provides isolation for the tools, we have not talked about projects yet. In this last part, we will explain how to use pyenv virtualenv to isolate each of your pyenv versions.

pyenv virtualenv takes advantage of the Python virtualenv framework, which enables the easy creation of isolated Python environments for each of your projects. Thanks to virtualenv, we can easily enter our environments to run our code, install packages inside them, and even destroy them with just a few commands. These environments are all installed within the ~/.pyenv/versions/PYTHON_VERSION/envs directory corresponding to your Python version (in our tutorial 3.8.1).

Note: This tutorial is part of a series about Python environments. You can go back to the first part with this link: Sane Python environment Part 1, isolation.

Pre-requisites

Before starting this tutorial, make sure you installed pyenv using Part 1 of the tutorial.

Usage of pyenv virtualenv

pyenv virtualenv is already installed by the pyenv-installer, so you don't need any additional installation step to use it. You can directly start setting up a Python project in the next section.

Setting up a simple Python project

In this section, we will create a simple Python project called request-project:

  • mkdir -p ~/sane-python-tutorial/request-project
  • cd ~/sane-python-tutorial/request-project
  • nano script.py, then copy-paste the content below and save

script.py

#!/usr/bin/env python

import requests

response = requests.get("https://api.github.com/events")
if response.status_code != 200:
    raise RuntimeError("Could not request GitHub events")

event_dict_list = response.json()
if len(event_dict_list) == 0:
    raise RuntimeError("GitHub event list was empty")

first_event_date = event_dict_list[0]["created_at"]
print(f"First event date: {first_event_date}")

script.py is very simple. It imports the requests library and performs an HTTP GET request on the GitHub events API, before printing the date of the first event that was returned. However, script.py will not work as long as the requests library has not been installed with Pip. As we don't want to run pip install requests in our global Python environment and pollute it, we are going to install the library in a specific isolated environment, that is dedicated to our project.

Creating a virtualenv for our project

Let's create a Python 3.8.1 virtualenv called request-project-env for our request-project:

➜ pyenv virtualenv 3.8.1 request-project-env
Using base prefix '/home/pierre/.pyenv/versions/3.8.1'
New python executable in /home/pierre/.pyenv/versions/3.8.1/envs/request-project-env/bin/python3.8
Also creating executable in /home/pierre/.pyenv/versions/3.8.1/envs/request-project-env/bin/python
Installing setuptools, pip, wheel...done.
Requirement already satisfied: setuptools in /home/pierre/.pyenv/versions/3.8.1/envs/request-project-env/lib/python3.8/site-packages
Requirement already satisfied: pip in /home/pierre/.pyenv/versions/3.8.1/envs/request-project-env/lib/python3.8/site-packages

# We can see our new virtualenv became available in the list of pyenv versions
➜ pyenv versions
  system (set by /home/pierre/.pyenv/version)
* 3.8.1
  request-project-env

Using the created virtualenv

Once we have our environment, enter the virtualenv we created:

# Enter (activate) our new virtualenv "request-project-env"
➜ pyenv activate request-project-env

# pyenv is now temporarily pointing to "request-project-env" as current version
➜ pyenv versions
  system (set by /home/pierre/.pyenv/version)
  3.8.1
* request-project-env (set by PYENV_VERSION environment variable)

# As request-project-env was just created, it is empty and contains very few packages
➜ pip list
Package    Version
---------- -------
pip        19.3.1 
setuptools 41.6.0 
wheel      0.33.6

Now that we are inside request-project-env, let's install the requests package and run script.py:

# Let's install the "requests" package with Pip
➜ pip install requests
[...] # installing dependencies, etc....

# We can check that our virtualenv was filled with the "requests" package
# and its dependencies
➜ pip list
Package    Version  
---------- ---------
certifi    2019.9.11
chardet    3.0.4    
idna       2.8      
pip        19.3.1   
requests   2.22.0   
setuptools 41.6.0   
urllib3    1.25.6   
wheel      0.33.6

# After checking that we have all our dependencies, we can now run the script
➜ python script.py 
First event date: 2019-11-03T22:32:37Z

# When we are done using our virtualenv, we should exit it and come back
# to our regular global Python
➜ pyenv deactivate

Automating activation and deactivation of virtualenv based on current working directory

In the previous sequence of commands, we went through the activation and deactivation process of the virtualenv manually. Fortunately, pyenv provides a command called pyenv local that automates this process:

➜ cd ~/sane-python-tutorial/request-project

# "pyenv local" registers a "local" virtualenv for the current directory. That is, our current virtualenv will be set automatically whenever we are visiting this directory. This information is stored in the ".python-version" file.
➜ pyenv local request-project-env

# Let's check the claim above by showing the pyenv version, when called
# from the "request-project" directory
➜ pyenv version
request-project-env (set by /home/pierre/sane-python-tutorial/request-project/.python-version)

# Let's move up one directory and check the pyenv version
➜ cd ..
➜ pyenv version
3.8.1 (set by /home/pierre/.pyenv/version)

From now on, the request-project-env will be activated automatically whenever we are inside the request-project directory.

virtualenv usage in practice

While the example provided in the previous section was pretty basic, the virtualenvs (or Python 3 venvs) are nevertheless commonly used in the industry. We will cover a few of its use cases in the following sections.

Running someone else's Python code in isolation in a virtualenv

When cloning a new repo coming from GitHub or your company's servers and running the install steps written in the README, you never know what is going to be installed in your Python environment. To avoid wrecking your global environment, it is always a good idea to create a new specific virtualenv for each Python repository that you are cloning.

Let's take a typical use case, where your company has a legacy app in Python 2.7, and you want to run it without messing with your global environment or your tools. You can solve this problem with pyenv and virtualenv as follows:

  • install Python 2.7 with pyenv
    • pyenv install 2.7.17
  • create a virtualenv based on Python 2.7 called legacy-code-env
    • pyenv virtualenv 2.7.17 legacy-code-env
  • cd into your repository
    • cd my-legacy-code
  • enable auto-activation of the virtualenv when you cd into the repo
    • pyenv local legacy-code-env
  • you can now develop my-legacy-code inside the legacy-code-env virtualenv

Another common use case for one-off virtualenvs is to create temporary environments to run small experiments before destroying them, for example when testing code from StackOverflow before using it in a real code base.

Managing Python project dependencies in a virtualenv with pip-tools

To specify dependencies in a Python project, a common way is to register them in a requirements.txt file. However, these become quite difficult to manage after a while because because of several reasons:

  • nobody on the team remembers where some dependencies came from
  • some dependencies have incompatible versions
  • when a dependency is removed, some packages can become unnecessary, but there is no way to know which ones

We can solve all the problems above with pip-tools. pip-tools enables you to specify your high-level dependencies (e.g. Django), while taking care automatically of the corresponding low-level dependencies (e.g. pytz dependency of Django). It also enables you to sync your environment to a requirements.txt file, regardless of the messy state you might have currently in your environment.

Installing pip-tools in your project

To use pip-tools, first create a new empty Python virtualenv in your project and activate it using pyenv, before installing pip-tools into that virtualenv:

  • pyenv virtualenv 3.8.1 project-env && pyenv local project-env
  • pip install pip-tools

Managing dependencies with pip-compile using a requirements.in file

pip-tools includes a pip-compile tool that takes a requirements.in file as input. This requirements.in file is similar to requirements.txt, but it contains only high-level dependencies. For example, if your project is using the latest version of Django, you could write something like this in requirements.in:

django>=3.0,<3.1

You can see that requirements.in does not contain the dependencies of Django, only Django itself.

Once you have set your dependencies in requirements.in, use pip-compile to "compile" your requirements.in into a requirements.txt file:

➜ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile requirements.in
#
asgiref==3.2.7            # via django
django==3.0.5             # via -r requirements.in
pytz==2019.3              # via django
sqlparse==0.3.1           # via django

The requirements.txt file generated by pip-compile indicates the source of every indirect dependency next to the package name. For example, pytz==2019.3 is a dependency of django. In addition to that, it also pins each dependency to a precise version number, to make sure that the installation of your dependencies is reproducible.

Applying dependencies from generated requirements.txt with pip-sync

Now that you have a requirements.txt, you can apply it to your Python virtual environment using pip-sync:

➜ pip-sync requirements.txt
Collecting asgiref==3.2.3 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 1))
  Using cached https://files.pythonhosted.org/packages/a5/cb/5a235b605a9753ebcb2730c75e610fb51c8cab3f01230080a8229fa36adb/asgiref-3.2.3-py2.py3-none-any.whl
Collecting django==3.0.2 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 2))
  Using cached https://files.pythonhosted.org/packages/55/d1/8ade70e65fa157e1903fe4078305ca53b6819ab212d9fbbe5755afc8ea2e/Django-3.0.2-py3-none-any.whl
Collecting pytz==2019.3 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 3))
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting sqlparse==0.3.0 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 4))
  Using cached https://files.pythonhosted.org/packages/ef/53/900f7d2a54557c6a37886585a91336520e5539e3ae2423ff1102daf4f3a7/sqlparse-0.3.0-py2.py3-none-any.whl
Installing collected packages: asgiref, pytz, sqlparse, django
Successfully installed asgiref-3.2.3 django-3.0.2 pytz-2019.3 sqlparse-0.3.0

pip-sync will make sure your virtual environment corresponds exactly to what is defined in the requirements.txt file by uninstalling and installing the relevant packages.

After your environment has been synced, make sure your project code works properly. If it does, commit requirements.in and requirements.txt into version control.

Modern packaging solutions

In the last few years, new packaging tools have been developed with inspirations from virtualenv, such as Poetry. Pipenv used to be a promising alternative, but it is no longer maintained seriously.

While these tools have become more and more popular, I have found pip-tools to be a blessing thanks to its very good compatibility with standards like Pip and requirements.txt. It has proven sufficent for the vast majority of my use cases, and would strongly recommend you try it first before installing more advanced tools.

Pitfall: In case you decide to install Poetry, please be aware that it should not be installed with pipx (unlike all other tools) because it is not able to pick up the right pyenv version automatically. Instead, you will have to create the project virtualenv yourself and make use of pyenv local to activate it automatically as seen in a previous section. Only after that will you be able to run poetry install.

Recap

In this tutorial, I started by presenting many issues that can happen in your daily life as a Python developer because of difficulties caused by your development environments. Then, I showed that isolation is important at several levels if you are to maintain and sane environment, and that this isolation can be achieved with three main tools: pyenv for managing isolated Python environments, pipx for managing isolated command-line tools, and pyenv virtualenv to manage isolated Python projects.

Thanks to this toolchain, you should be able to maintain a sane and stable development workflow for Python projects. I thank you for reading this to the end, and wish you a very nice day!

References

Here is a curated list of blog posts related to Python environments that you may find interesting, and that inspired me to write this tutorial: