UPDATE 2020-04-16 This is not how you should do it

These instructions and tools are great if you're going to use Python 2 and Virtualenv. But Python 2 is dead, and there are better ways to do this in Python 3. I'm still exploring the best ways, but I am leaning towards using pyenv + pyenv-virtualenv (or python venvs, or maybe pipenv?). Pyenv takes inspiration from rbenv which is how Ruby does this, so if you know Ruby then you'll understand it. When I understand better, I'll write another blog post

A while ago I noted that I should write about my shell functions for creating and managing Python Virtual Environments. Recently I was helping my boss set up some python virtual environments for some different projects, and I couldn't remember how to use virtualenv directly.

I really wanted to just share my wrappers with him, and I found that they weren't quite ready to share because I hadn't finished documenting them.

This post is to remedy that.

I have a small collection of bash functions which I use for creating and managing Python Virtual Environments with virtualenv. I keep them in my dotfiles. Rather than make you clone my entire dotfiles set up, I have modularised it down to two files which you can source. They are:

10_meta.sh (v1.1.3) - shell utility functions
50_python.sh (v1.1.3) - python utility functions

To use these, download the scripts to a place where you keep shell functions:

cd path/to/your/shell/library
for X in 10_meta.sh 50_python.sh; do
  curl https://raw.githubusercontent.com/sinewalker/dotfiles/1.1.3/source/$X > $X
done

Then, load them into your bash like this (you can add this to your .bashrc somewhere so they are loaded whenever you start bash):

for X in path/to/your/shell/library; do
  source $X
done

Let's explore them.

Make an environment

One of the most useful meta-functions that you just sourced from 10_meta.sh is describe, which I talk about more fully on my post about Rule 6 - Doc Comments.

$ describe mkvenv 
Fn: mkvenv in ./50_python.sh line 60
    "Makes a Python Virtual env in ${VIRTUALENV_BASE}.";

That's interesting, all the python virtual environments will be created in ${VIRTUALENV_BASE}. What's this? Well you can echo it out:

$ echo $VIRTUALENV_BASE
/Users/mjl/lib/python

It's actually set in 50_python.sh on line 53 as the python sub-directory of your ${LIBRARY} directory, if you have one, or of your ${HOME}/lib directory if you don't. The following line just makes sure you have that directory. All the virtual environments will be created in this location.

So mkvenv will create a virual environment for you:

$ mkvenv
Usage: mkvenv <venv> [virtualenv options]
Makes a Python Virtual env in /Users/mjl/lib/python.
$ mkvenv myvenv
New python executable in /Users/mjl/lib/python/myvenv/bin/python2.7
Also creating executable in /Users/mjl/lib/python/myvenv/bin/python
Installing setuptools, pip, wheel...done.

Oh, that made a virtualenv with my system-default Python 2.7.

What if I want Python 3?

Well it's easy, just pass along normal virtualenv arguments after the name of the environment:

$ mkvenv myvenv3 --python=python3
Running virtualenv with interpreter /usr/local/bin/python3
New python executable in /Users/mjl/lib/python/myvenv3/bin/python3.7
Also creating executable in /Users/mjl/lib/python/myvenv3/bin/python
Installing setuptools, pip, wheel...done.

Cool. Let's use it.

Activate an environment

Activating an environment puts it in your shell $PATH and adds it's name to your prompt, so you can tell that it's active. This uses virtualenv activation scripts to do the work, just like normal:

$ activate myvenv3
(myvenv3)$

A nice thing about my activate is that you don't have to remember where you installed the environment and source bin/activate from there. Instead it is just found for you from the relevant environment in ${VIRTUALENV_BASE}. Very handy.

Listing environments

The activate function (and others) all use Bash TAB-completion, if you have that enabled. But maybe you're not sure what environments you have and need to see them all? Use lsvenv to list your environments:

(myenv3)$ lsvenv
edge-config
hax
jupyter
keras
myvenv
myvenv3
nikola
py27
sqlalchemy
te

Hmm, that myvenv is messy, I want to get rid of it.

Removing environments

Use rmvenv to remove an environment:

(myenv3)$ rmvenv myvenv
Remove Venv: myvenv? [y/N] y
$

Deactivating an environment

You can switch directly to another environment and activate will automatically deactivate the current one first. But if you want to deactivate all of the virtualenv stuff without going to a different environment or ending your shell session, just use deactivate like normal:

(myvenv3)$ which python
/Users/mjl/lib/python/myvenv3/bin/python
(myvenv3)$ deactivate 
$ which python
/usr/local/bin/python
$

Virtualenv alternative: Anaconda

I also use Anaconda Pyhton a little bit, which has its own method for managing virtual environments with the conda tool. But activation and deactivation is a but clunky. So my activate and deactivate functions also work with Anaconda.

But now we have two kinds of Python virtual environments. How is that handled? Use sucuri (which is named after an Amazonian word for anaconda) to switch to Anaconda-mode.

$ sucuri 
Anaconda: ACTIVATED 🐍
$

Yes, that's a UTF-8 character. If your terminal doesn't handle that, my function takes care of you and uses ASCII instead.

Note that there's no other indication that Anaconda is active. My complicated bash prompt function does give you some indication by adding a snake. If you're interested, you can go down that rabbit hole. I'll leave my prompts in place for the rest of this blog post.

Anyway, you can list your environments in the same way. This time the conda environmens are listed instead of the virtualenv ones (using conda info — another thing you don't need to remember):

[mjl@milo:~/hax]
[08:32](🐍)β lsvenv 
snowflakes
root
[mjl@milo:~/hax]
[08:32](🐍)β

Activation is the same too:

[mjl@milo:~/hax]
[08:32](🐍)β activate snowflakes 
[mjl@milo:~/hax]
[08:32](🐍-snowflakes)β which python
/Users/mjl/lib/anaconda/envs/snowflakes/bin/python
[mjl@milo:~/hax]
[08:32](🐍-snowflakes)β deactivate 
[mjl@milo:~/hax]
[08:32](🐍)β

In Anaconda-mode, you can use my mkvenv to make a conda environment, but I don't try to wrap all the special conda commands. Use conda if you want to do something more sophisticated, but for basic stuff, this is fine:

[mjl@milo:~/hax]
[08:35](🐍)β mkvenv anotherenv
mkvenv: Warning! Anaconda is active.
This wrapper will use conda to create anotherenv, but it is only very basic.
Fetching package metadata .........
Solving package specifications: 
Package plan for installation in environment /Users/mjl/lib/anaconda/envs/anotherenv:

Proceed ([y]/n)? 

#
# To activate this environment, use:
# > source activate anotherenv
#
# To deactivate this environment, use:
# > source deactivate anotherenv
#

[mjl@milo:~/hax]
[08:35](🐍)β

(Don't "use source activate anotherenv" … pfft. Just activate anotherenv.)

Here's the new environment:

[mjl@milo:~/hax]
[08:35](🐍)β lsvenv
anotherenv
snowflakes
root

Removing it is a bit tricky. My rmvenv just gives up. It does tell you how you might want to do it with conda though:

[mjl@milo:~/hax]
[08:36](🐍)β rmvenv anotherenv
rmvenv: Warning! Anaconda is active.
Consider using 'conda remove -all -n anotherenv' instead.
Aborting.
[mjl@milo:~/hax]
[08:36](🐍) 3 β

At the end of your conda session, deactivate by running sucuri again:

[mjl@milo:~/hax]
[08:32](🐍)β sucuri 
Anaconda: deactivated
[mjl@milo:~/hax]
[08:32]β

That's all, I hope you find these wrappers simple and easy to use. No more fumbling with virtual environment setup and management.

Happy hacking.