It has been 1.5 years since I published my first tutorial about "sane Python installation management". However, I still hear about Python environment issues on a regular basis from coworkers, particularly regarding local project dependency management, and I felt like an update to the original tutorial would be beneficial.
This new "2020" version of the tutorial will re-use a good amount of material from the original one, but will try to put each tool being used into a broader context, and explain how all tools relate to each other. This new version will walk you through the way to a sane Python development environment, covering the following topics in separate posts:
pyenv, including shielding your system Python
pipx, so they don't break each other when upgraded
EDIT from Tuesday, Nov 12, 2019: Interestingly enough, Jacob Kaplan-Moss actually published an updated version of his article from 2011 at the same time as I published my own update. You can check it out here: My Python development environment, 2020 edition. It also prompted some debates on HackerNews.
EDIT from Tuesday, Jan 7, 2020: The commands and the console output of this tutorial have been updated with Python 3.8.1 (was 3.6.4 before), and some installation steps have been clarified.
EDIT from Saturday, Feb 22, 2020: I recently discovered
pip-tools, which is an awesome program to manage
pip-tools is much closer to the "vanilla"
pip experience, and therefore much more stable and less prone to usability problems. Combined with pyenv, it solved all my use cases for project dependency management.
The key idea behind all tools described above is environment isolation. In this first part, we will explain why isolation is so important in Python based on common issues encountered by developers, and then formalize the points of contact that require isolation.
If you have done any serious Python development that involves installing Python packages from the PyPI, you may recognize yourself in some of these situations:
pip install cool_new_tool, your innocent
old_boring_tool stopped working because
cool_new_tool upgraded some dependencies that
old_boring_tool was using.
pip install cool_library somehow threw an error so you ran it as root with
sudo pip install cool_library. Then,
pip install started throwing permission errors whenever it was used without
sudo, which made installing some other dependencies impossible. Moreover, you were at a serious security risk because you were downloading random packages from the Internet and letting them run as root.
legacy_project in Python 2, but the
new_cool_project based on Python 3 popped up at your company and you had to work on both at the same time. While working on
new_cool_project, you accidentally ran
pip install insted of
pip3 install and you screwed up the development environment of
What is the common problem to all these situations? The answer is the lack of isolation. That is, whenever you make a modification anywhere inside the Python ecosystem of your machine, you are taking the risk to set off a fire someplace else.
When looking at the common issues described in the previous section, we can identify important points of contact where isolation should be enforced:
/usr) that is common to all users.
black should be isolated from each other and from your Python projects under development. Moreover, they should be installed only once per user, and not once per project.
To enforce isolation for each of these points of contact, tools have been developed by the Python community:
pyenv addresses 1. and 2. by managing your Python environments and interpreter versions inside a directory in your
pipx addresses 3. by installing each tool into a separate directory in your
$HOME, including all its dependencies.
pyenv virtualenv addresses 4. by installing each project into a separate directory in your
$HOME, including all its dependencies.
Each of these tools will be described in a separate part of this tutorial. Feel free to jump to Part 2 about
pyenv if you want to follow the tutorial in order.