Deprecation notice

EDIT: Don't read this page, there is a new version with more recent material.

Article from 2018

After a new week during which a friend's system Python installation was destroyed by yet another unfortunate update, I felt like it was a great time to do some research about the tooling that was designed to prevent that from happening.

In this article, I'm introducing two widely used tools to control potential damage from accidental destruction of Python installations, through isolation:

• pyenv: isolates system Python version from development Python versions.
• pipsi: isolates command-line tools from system Python version

Some key ideas behind pyenv and pipsi are that you should not be working directly with your system Python installation, and that your python interpreter and his friends (pip, packages, CLIs) should live in your $HOME directory rather than /usr/local.

If you are not sure why this isolation is helpful, here are a few examples:

• When your package manager or your OS decides to update the system Python, all your tools and development projects could stop working.
• Some of your innocent common command-line tools that happen to be written in Python (youtube-dl, aws-cli, howdoi, thefuck) could stop working if you break your system Python.
• Installing packages in your system Python may tempt you to run some commands as a superuser. That is, you could give malicious Python code from the Internet all the permissions it needs to mess with your computer.
• With isolated Python versions, you don't need to have python2, pip2, python3 and pip3 in cohabitation. You just have one python + pip pair, which points to a single version of Python at a time. You switch between versions by simply pointing your global Python to the version you need for your current task.

Note: This tutorial was written on an Ubuntu machine. If you are using macOS, consider replacing some of the instructions by high-level calls to brew to take advantage of existing Homebrew formulas. For this, install the Xcode Command Line Tools (xcode-select --install) and Homebrew.

Manage different Python installations with pyenv

pyenv is a command-line tool that enables you to manage Python installations in isolation to the Python installation of your operating system.

That is, it enables you to destroy your Python development installation without destroying the Python installation of your operating system. The core ideas behind pyenv are similar to the ones of nvm for JavaScript, or rbenv for Ruby.

Installation

The installation of pyenv requires some dependencies, so please install those first:

• Ubuntu
  • apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev
• macOS
  • brew install openssl readline xz

As pyenv is independent of your OS Python, it does not use Python for installation, and relies on a Bash script instead. This Bash script (bin/pyenv-installer) is provided in the pyenv-installer repo. Install it using the following command:

• curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

After downloading pyenv and installing it to ~/.pyenv, the Bash script will ask you to add a few initialization lines to your interactive Shell config file (e.g. ~/.zshrc or ~/.bashrc) or your login Shell config (~/.zprofile or .bash_profile). If you don't know which one to pick, choose ~/.zshrc if you use ZSH and ~/.bashrc if you use Bash.

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

These initialization lines include a modification of your $PATH, which enforces the precedence of pyenv over the other registered directories. This shim trick enables pyenv to intercept Python-related commands.

Pitfall: When adding new locations to the $PATH in the future for new commands, always make sure pyenv keeps precedence so it can continue to intercept the Python-related commands.

After updating your config, exit your user session and login again. Once you're logged back in, open a new terminal.

When the new terminal shows up, the pyenv command should be available, and you should be able to run pyenv versions. This command will show you all the Python environments installed on your computer. As no new environment was installed yet, you should just see the Python of your OS:

➜ ~ pyenv versions
* system (set by /home/pierre/.pyenv/version)

Usage

Now, let's check which Python versions are available, and install one of them:

➜ ~ pyenv install --list | grep 3.6
  3.3.6
  3.6.0
  3.6-dev
  3.6.1
  3.6.2
  3.6.3
  3.6.4

➜ ~ pyenv install 3.6.4
Downloading Python-3.6.4.tar.xz...
-> https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tar.xz
Installing Python-3.6.4...
Installed Python-3.6.4 to /home/pierre/.pyenv/versions/3.6.4

➜ ~ pyenv versions
* system (set by /home/pierre/.pyenv/version)
  3.6.4

As you can see, our new Python version was installed successfully in ~/.pyenv/versions. There might be a few warnings for some extensions, but you can fix those later by installing the relevant libraries using your system's package manager.

In the output of the pyenv versions command, the little star next to the version name indicates the current global version of Python, which is initially system.

Thanks to pyenv, your system Python is still fully functional and was not destroyed by your new 3.6.4 installation:

➜ ~ python
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

After screaming out of joy for your preserved system, you can switch to your new version of Python using the pyenv global command:

➜ ~ pyenv global 3.6.4

➜ ~ python
Python 3.6.4 (default, Mar  3 2018, 15:29:45) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Please note that you may have to close and re-open the terminal session for the update from the pyenv global command to come into force.

You can then switch back and forth between your versions using the same command:

➜ ~ pyenv global system

➜ ~ python
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

➜ ~ pyenv global 3.6.4

➜ ~ python
Python 3.6.4 (default, Mar  3 2018, 15:29:45) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Finally, be aware that pyenv has the ability to select multiple global Pythons at the same time, even though it is not required for common development use cases. That is, you may see other tutorials on the Internet where there is more than a single star in the output of pyenv versions.

Pitfall: As you can see, the notions of global and system Python do not refer to the same thing in pyenv. Your system Python is the one distributed by your OS, while your global Python is just a pointer to the Python currently selected by pyenv and located in your $HOME. To avoid messing up your system Python, you should always have your global Python pointing to a version installed using pyenv. In our case, the global Python should be pointing to the version 3.6.4 that was installed earlier.

Uninstallation

pyenv is self-contained, so it can be removed fairly easily. You just need to:

1. Optional: in case you already installed pipsi from the next section of the tutorial, make sure to uninstall it first before uninstalling pyenv.
2. in your Shell config, remove the following:
   • pyenv directory from your $PATH
   • pyenv init lines
3. delete the pyenv directory:
   • rm -rf ~/.pyenv

Manage Python-based command-line tools with pipsi

pipsi is a command-line tool that installs Python-based command-line tools in separated virtualenvs, which are isolated from your global Python installation and from each other.

That is, pipsi enables you to destroy your Python command-line tools without destroying your global Python installation.

Installation

Before running the pipsi installation script, make sure your global Python points to a specific version of Python such as 3.6.4, and not to the system version. Indeed, pipsi is installed with a Python script (get-pipsy.py), which makes it dependent on your global Python installation, and you don't want to ruin your system installation. Moreover, pipsi will install all tools using the version of the pyenv it was itself installed with, even if you change the global Python to point to another version.

As pipsi makes extensive use of virtualenv, make sure this package is installed by running pip install virtualenv.

These caveats being said, here is the command to install pipsi:

• curl https://raw.githubusercontent.com/mitsuhiko/pipsi/master/get-pipsi.py | python

Pitfall: Depending on whether your operating system has ~/.local/bin in the $PATH by default or not, get-pipsi.py may warn you that the pipsi command is not in your $PATH yet. If it's not the case, update your config to add it manually.

Usage

Once pipsi is installed, you can use it to install all kinds of Python command line tools, such as Pipenv:

pipsi install pipenv
...

# See how Pipenv was installed in a virtualenv by pipsi
➜ ~ ls -al $(which pipenv)
... /home/pierre/.local/bin/pipenv -> /home/pierre/.local/venvs/pipenv/bin/pipenv

As shown by the ls command, pipsi installs all tools within the ~/.local/venvs directory and symlinks them to ~/.local/bin.

Uninstallation

To remove pipsi, you can use pipsi itself:

• pipsi uninstall pipsi

To go further

In this last section, you will find a very brief overview of Pipenv which is a natural extension of the isolation ideas of pyenv and pipsi to Python project dependency management.

After this overview, you will find a list of links related to this article.

Manage project dependencies with Pipenv

Pipenv is a command-line tool that simplifies Python project dependency management by combining Pip with virtualenv, in such a way that all your projects are isolated from each other and from your global Python.

That is, it enables you to destroy the installation of one of your Python projects without affecting your other Python projects. At the same time, it also provides a simple framework to specify package dependencies in the Pipfile, which takes ideas from package.json for JavaScript or composer.json for PHP.

As Pipenv introduces many more features than pyenv and pipsi, its usage goes beyond the scope of this blog post. For the time being, you can just install Pipenv with pipsi install pipenv, and read the docs of Pipenv for more details. If you just want a summary, there is a great tutorial by Alexander VanTol on RealPython.

Links

Here is a list of blog posts related to pyenv and pipsi that you may find interesting:

• My Python development environment, 2018 edition by Jacob Kaplan-Moss, core contributor to Django. The corresponding Hacker News discussion is also worthwhile.

• Installing things the right way with pipsi and pipenv by Craig Loftus, who shows how much pipsi and pipenv simplify the Python development workflow compared to older tools.

• PyEnv is the new Conda by Bastian Bechtold, who points out the advantages of standard tools compared to the monolithic Conda.

• Pyenv, Python Version Management Made Easier by Senthil Kumar, who gives a more detailed introduction to pyenv.

If you run into issues with the $PATH manipulations of pyenv, these StackOverflow answers might help you:

• How to correctly set the $PATH? by Gilles and Stéphane Chazelas on StackOverflow Unix

• What should/shouldn't go in .zshenv, .zshrc, ...? by Smith Jon on StackOverflow Unix

• Choosing between .bashrc, .profile, etc... by Dan Rabinowitz and Gilles on StackOverflow SuperUser