Virtualenv Tutorial Part 2

by Szymon Lipiński
tags: python virtualenv

The First Version of This Tutorial

A couple of years ago I wrote a simple article about the virtualenv. A kind of a very basic tutorial showing how to use the Python Virtual Environment with different versions of Python like Python 2, and Python 3.

Since then I’ve changed how I work with the virtualenvs, so here is a more updated version of the tutorial. A better way of managing the environments.

In this tutorial I will show you how to create a Virtual Environment, manage it, install different libraries, and use different Python versions in each of them.

What The Virtualenv Really Is

Virtualenv is a very simple tool for a kind of virtualizing the Python environment. Unfortunately this is not a full virtualization. It allows you just to have a separate python executable file, along with a separate set of libraries.

It doesn’t give you a totally isolated environment. This means that you cannot fully simulate the environment you have on production servers. Usually applications communicate with some outside world. That worlds has some databases, some webservers, some files. Virtualenv doesn’t virtualize the outside world.

The main problem is that some Python libraries are just wrappers of files, and libraries written in C. A good example is psycopg2, it’s a great library which just a Python wrapper to a PostgreSQL’s library. This is great because this way we have quite fast Python library to connect with PostgreSQL.

However those C libraries are compiled with the current C compiler, the other libraries you have, the database version you have installed, and the hardware architecture.

To fully simulate the production environment you should have a computer with the same architecture, operating system version, installed libraries and programs. Usually this is not very easy to be done. You can also use docker for that.

What Virtualenv is great for is having different Python versions and different sets of libraries. I usually have one environment for each project that I’m working on. Fortunately for my projects the outside world doesn’t matter too much. My Python software works on the production machines without any problems, as they are quite similar to the one I’m working on. Usually they even have the same operating system, architecture, and compiler.

This can be not the same in your case, so be careful.

Python Virtualenv Advantages

The main advantages of the Python Virtualenv are:

Why You Really Need The Virtual Environment

There are many reasons why such a software is needed and why you really should use it.

The main one is that for one project you need a different version of a Python package, than you have installed on your machine, or you don’t have a system package for that. The alternatives are simple:

Building good system packages, like debs or rpms is not easy. What’s more, if you have a task of writing a Python program, you probably don’t want to spend the next couple of weeks reading through the documentation of the packages. Even if you want, I’m sure your client will not be happy about that.

On Linux you can have plenty of python programs running in the background, you can even not know about that. Check it with ps uaxf | grep python. On my Ubuntu even the network connection manager is a Python script (I use wicd). Those scripts have their dependencies, which are packed in the system packages. If your project depends on some Python package, and you want to upgrade a global package, then you can have much bigger problem. This new package can be incompatible with e.g. wicd, and there is a chance that your computer can suddenly lose access to the network.

Simpler things are always better. The simplest thing would be to install that package in a local directory and change the path used for finding the packages, so this one would be found instead of the system one. This is exactly what the Virtualenv does.

Using Different Python Versions

Virtualenv also allows for different Python versions. When you build a Python Virtual Environment, then you can choose the exact Python executable file you want to use. Then there is created a separate directory for this environment, and the executable file is simply copied there. All libraries for this Virtual Environment will be installed in the same directory.

Why Not Virtualenv

Virtualenv is great. Unfortunately it is too verbose and too complicated in a day to day work. That’s why I stopped using that a long time ago.

Virtualenv vs Virtualenvwrapper

There is something much better. Much better means that it is much easier to use, while the functionality is basically the same. It is named Virtualenvwrapper and it is a set of tools for managing Python Virtualenv operations.

How to Install Virtualenvwrapper for Python

I tried installing Virtualenvwrapper globally on my machine using the deb archive with simple:

λ ymon ~ → sudo apt-get install virtualenvwrapper

The beginning λ ymon \~ → is my current shell prompt.

Unfortunately that didn’t work properly. It installed a couple of files, but not all magic happened, and some files were missing. Especially painful was the fact, that the virtualenvwrapper.sh and virtualenvwrapper_lazy.sh were not where they should be.

After reading through almost the whole internet, I found a solution. It was simple. I fully removed the Virtualenwrapper installed with the deb file, and installed it globally using sudo pip. This also installed it in the system directories.

λ ymon ~ → sudo apt-get purge virtualenvwrapper
λ ymon ~ → sudo pip install virtualenvwrapper

And suddenly I got the two files available:

λ ymon ~ → which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

λ ymon ~ → which virtualenvwrapper_lazy.sh
/usr/local/bin/virtualenvwrapper_lazy.sh

After this I had to configure a couple of things. First of all I have created a directory for the Python Virtual Environments:

λ ymon ~ → mkdir ~/.virtualenvs

I have also added an environment variable to my shell. I use zshell currently, so I added this to my ~/.zshrc. If you are using a standard shell, like bash, you need to add the configuration to your ~/.bashrc file.

export WORKON_HOME=$HOME/.virtualenvs

I also had to add a command to the ~/.zshrc to run the activation script when my shell starts.

source /usr/local/bin/virtualenvwrapper_lazy.sh

The only thing left was to log out from the current shell and start a new one.

Yes, I know that I could source a config file, or start a new shell from this one. Unfortunately sometimes it doesn’t read all the files, and sometimes the different config files fight with each other to show who is more important. Running a brand new console window ensures me that all the configs were read and executed like they should be.

How to Create a New Virtual Environment

The Python Virtualenvwrapper provides a couple of nice commands which can be used for playing with the Python Virtual Environments.

The first is used to create a new Python Virtual Environment. The name should be chosen wisely, so you will know what the purpuse of the environment is. After a couple of months I have over twenty environments and I choose the working one only by the name.

However for the purpose of this article, I will create an environment named test1.

λ ymon ~ → mkvirtualenv test1
New python executable in test1/bin/python
Installing Setuptools ..............................................................................................................................................................................................................................done.
Installing Pip .....................................................................................................................................................................................................................................................................................................................................done.
(test1)λ ymon ~ →

As you might notice, the command prompt changed, and now it contains the name of the Virtual Environment.

(test1)λ ymon ~ → python --version
Python 2.7.5+
(test1)λ ymon ~ →

As the Python Virtual Environment has been activated, which is indicated by the changed prompt, all the Python packages you install without using sudo are installed locally. When I call the pip command, it installs the packages in the ~/.virtualenvs/test1 directory.

Let’s check the packages we have preinstalled:

(test1)λ ymon ~ → pip freeze
argparse==1.2.1
wsgiref==0.1.2

And let’s install Django

(test1)λ ymon ~ → pip install Django
Downloading/unpacking Django
  Downloading Django-1.6.4.tar.gz (6.6MB): 6.6MB downloaded
  Running setup.py egg_info for package Django

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: Django
  Running setup.py install for Django
    changing mode of build/scripts-2.7/django-admin.py from 644 to 755

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /home/szymon/.virtualenvs/test1/bin/django-admin.py to 755
Successfully installed Django
Cleaning up...

And now the installed packages are:

(test1)λ ymon ~ → pip freeze
Django==1.6.4
argparse==1.2.1
wsgiref==0.1.2

How to Log Out of a Python Virtual Environment

If you want to deactivate current virtual environment, you can use the command deactivate (that’s exactly like in pure virtualenv):

(test1)λ ymon ~ → deactivate
λ ymon ~ →

As you can see, the environment name disappeared from the prompt.

How to Use a Python Virtual Environment

For activating an exising Python Environment you can use the command workon:

λ ymon ~ → workon test1
(test1)λ ymon ~ →

If you write just workon and press TAB, then you should see the list of all available virtual environments.

How to Remove a Python Virtual Environment

You can remove the virtual environment using the command rmvirtualenv. Beware, it removes the whole directory in your virtual environments directory.

    (test1)λ ymon ~ → rmvirtualenv test1
    Removing test1...
    ERROR: You cannot remove the active environment ('test1').
    Either switch to another environment, or run 'deactivate'.

You cannot remove an environment you are using, so let’s deactivate it first and then remove.

(test1)λ ymon ~ → deactivate
λ ymon ~ → rmvirtualenv test1
Removing test1...
λ ymon ~ →

How to Usee Different Python Versions

The environment I’ve already created is using my default python, which is:

(test1)λ ymon ~ → python --version
Python 2.7.5+

Imagine that I want to use a different one, let’s use Python 3.3. First of all I need to find the binary file:

(test1)λ ymon ~ → which python3.3
/usr/bin/python3.3

Then I need to remove my current Virtual Environment (deactivating it first):

(test1)λ ymon ~ → deactivate
λ ymon ~ → rmvirtualenv test1
Removing test1...

Now I can create a new virtual environment with the same name, but using Python3.3.

λ ymon ~ → mkvirtualenv test1 -p /usr/bin/python3.3
Running virtualenv with interpreter /usr/bin/python3.3
Using base prefix '/usr'
New python executable in test1/bin/python3.3
Also creating executable in test1/bin/python
Installing Setuptools ..............................................................................................................................................................................................................................done.
Installing Pip .....................................................................................................................................................................................................................................................................................................................................done.

Let’s just check if I really have this python version in my new environment:

(test1)λ ymon ~ → python --version
Python 3.3.2+

And let’s install Django once again

(test1)λ ymon ~ → pip install Django
Downloading/unpacking Django
  Downloading Django-1.6.4.tar.gz (6.6MB): 6.6MB downloaded
  Running setup.py egg_info for package Django

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: Django
  Running setup.py install for Django
    changing mode of build/scripts-3.3/django-admin.py from 644 to 755

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /home/szymon/.virtualenvs/test1/bin/django-admin.py to 755
Successfully installed Django
Cleaning up...

Check the installed libraries:

(test1)λ ymon ~ → pip freeze
Django==1.6.4