Virtual Environments using Apache and mod_python

After searching for a while on The Internet I couldn't find a good solution for my problem: running two versions of Trac on the same Apache web server, using mod_python.

Some experience with python, and Apache configuration is assumed!

The problem

Setting up the new airadier.com development site (the one you're reading this post on), I decided to go for the recently released Trac 0.11. The same server was already running an installation of Trac version 0.10.4. So, I needed to be able to run both versions at the same time, each with their own modules and plugins (I didn't went to go through the upgrade process for the current site).

First approach

At first, I tried moving the currently intalled Trac folder and plugin eggs to an isolated folder, that is:

/usr/local/trac0.10

which contained:

trac/
trac-0.10.4.egg-info
TracAccountManager-0.1.3dev_r2548-py2.5.egg/
TracPageToPDF-0.2-py2.5.egg/
TracProjectMenu-1.0-py2.5.egg/
TracTocMacro-1.0-py2.5.egg/
TracWebAdmin-0.1.2dev-py2.5.egg/

then I made a new folder:

/usr/local/trac0.11

and installed the new version of Trac 0.11 and some plugins. After installing these items, the trac0.11 folder had the following files and folders:

cgi-styler-form.py
cgi-styler.py
clearsilver-0.10.1-py2.5-linux-i686.egg/
easy-install.pth
Genshi-0.5.1-py2.5-linux-i686.egg/
IniAdmin-0.2-py2.5.egg/
pygmentize
Pygments-0.10-py2.5.egg/
SilverCity-0.9.7-py2.5-linux-i686.egg/
source2html.py
textile-2.0.11-py2.5.egg/
Trac-0.11stable_r7327-py2.5.egg/
TracAccountManager-0.2.1dev_r3857-py2.5.egg/
trac-admin
tracd
TracFullBlogPlugin-0.1-py2.5.egg/
tracsuperuser-0.2-py2.5.egg/

Installing to an isolated folder

To install python packages or eggs to an isolated folder (e.g. /usr/local/trac0.11), I recommend using easy_install. To install an egg file, just do:

PYTHONPATH=/path/to/virtualenvironment easy_install -d /path/to/virtualenvironment -Z egg_to_install.egg

make sure you have write permissions in the destination folder, you might need to use sudo in front of easy_install:

PYTHONPATH=/path/to/virtualenvironment sudo easy_install -d /path/to/virtualenvironment -Z egg_to_install.egg

to install a ready to build package (that is, a folder containing the setuptools setup.py file), just do:

PYTHONPATH=/path/to/virtualenvironment easy_install -d /path/to/virtualenvironment -Z package_folder/

again, don't forget about write permissions.

Configuing Apache

Next step, I filled the virtual site configuration for Apache, using a Location directive (as instructed by the Trac installation guide for mod_pythoon):

Site for Trac 0.10.4:

        ...

        <Location /trac>
                SetHandler mod_python
                PythonPath "['/usr/local/trac0.10',] + sys.path"
                PythonHandler trac.web.modpython_frontend
                PythonOption TracEnvParentDir /path_to_trac0.10_envs/
                PythonOption TracUriRoot /trac
        </Location>

        ...

Virtual Site configuration for Trac 0.11:

        ...

        <Location /trac>
                SetHandler mod_python
                PythonPath "['/usr/local/trac0.11',] + sys.path"
                PythonHandler trac.web.modpython_frontend
                PythonOption TracEnvParentDir /path_to_trac0.11_envs/
                PythonOption TracUriRoot /trac
        </LocationMatch>

        ...

That way the PythonPath should be different for the different virtual hosts.

Why it didn't work

The previous approach didn't work at all. I got mod_python complaining about the "trac" module couldn't be found. Why? Finally I noticed. Apparently, working from command line, everything was correct:

$ PYTHONPATH=/usr/local/trac0.11/ python
iPython 2.5.1 (r251:54863, Mar  7 2008, 03:41:45) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
m>>> import trac
>>> dir(trac)
['__builtins__', '__doc__', '__file__', '__name__', '__path__', '__version__']
>>> trac.__version__
'0.11stable-r7327'
>>>

$ PYTHONPATH=/usr/local/trac0.10/ python
Python 2.5.1 (r251:54863, Mar  7 2008, 03:41:45) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import trac
>>> trac.__version__
'0.10.4'

The key was the difference between the PYTHONPATH environment variable, and the PythonPath option for mod_python. The first is set before the python interpreter is loaded. When the interpreter stars, the folders in PYTHONPATH are added to sys.path. Then, every folder in sys.path is scanned for eggs and .pth files, and files or folders listed in there are added to the sys.path too. These are the contents of /usr/local/trac0.11/easy_install.pth:

import sys; sys.__plen = len(sys.path)
./Trac-0.11stable_r7327-py2.5.egg
./Genshi-0.5.1-py2.5-linux-i686.egg
./TracAccountManager-0.2.1dev_r3857-py2.5.egg
./Pygments-0.10-py2.5.egg
./clearsilver-0.10.1-py2.5-linux-i686.egg
./textile-2.0.11-py2.5.egg
./SilverCity-0.9.7-py2.5-linux-i686.egg
./TracFullBlogPlugin-0.1-py2.5.egg
./IniAdmin-0.2-py2.5.egg
./tracsuperuser-0.2-py2.5.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

And this are the contents of sys.path when starting the python interpreter from the command line:

$ PYTHONPATH=/usr/local/trac0.11/ python
Python 2.5.1 (r251:54863, Mar  7 2008, 03:41:45) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['',
'/usr/local/trac0.11/Trac-0.11stable_r7327-py2.5.egg',
'/usr/local/trac0.11/Genshi-0.5.1-py2.5-linux-i686.egg',
'/usr/local/trac0.11/TracAccountManager-0.2.1dev_r3857-py2.5.egg',
'/usr/local/trac0.11/Pygments-0.10-py2.5.egg',
'/usr/local/trac0.11/clearsilver-0.10.1-py2.5-linux-i686.egg',
'/usr/local/trac0.11/textile-2.0.11-py2.5.egg',
'/usr/local/trac0.11/SilverCity-0.9.7-py2.5-linux-i686.egg',
'/usr/local/trac0.11/TracFullBlogPlugin-0.1-py2.5.egg',
'/usr/local/trac0.11/IniAdmin-0.2-py2.5.egg',
'/usr/local/trac0.11/tracsuperuser-0.2-py2.5.egg', 
'/usr/local/trac0.11', 
'/usr/lib/python25.zip', 
'/usr/lib/python2.5', 
'/usr/lib/python2.5/plat-linux2', 
'/usr/lib/python2.5/lib-tk', 
'/usr/lib/python2.5/lib-dynload', 
'/usr/local/lib/python2.5/site-packages', 
'/usr/lib/python2.5/site-packages', 
'/usr/lib/python2.5/site-packages/PIL', 
'/var/lib/python-support/python2.5']
>>> 

However (I found out this using the PythonHandler mod_python.testhandler instead of the trac.web.modpython_frontend handler), the sys.path in the mod_python interpreter didn't include all this .egg folder (so, module trac, in folder /usr/local/trac0.11/Trac-0.11stable_r7327-py2.5.egg couldn't be imported).

I guess mod_python loads the python interpreter first, and then adds evaluates the PythonPath option, which adds /usr/local/trac0.11 folder to sys.path, but doesn't check it for .pth files.

The fix

Right now I managed to get it working by wrapping the trac mod_python handler with an special handler that will correctly set-up the python path. This handler is a small python module located in:

/usr/local/trac0.11/handler_wrapper.py

import site
from mod_python import apache
import sys

def handler(req):
        sitedirs = eval(req.get_options()['SiteDirs'])
        for sitedir in sitedirs:
                site.addsitedir(sitedir)
        handler_name = req.get_options()['WrappedHandler']
        mod_handler = __import__(handler_name)
        for comp in handler_name.split('.')[1:]:
                mod_handler = getattr(mod_handler, comp)
        return mod_handler.handler(req)

This handler will take the SiteDirs option from mod_python (see later), and for each specified dir, scan it for .pth files using site.addsitedir from the site module.

After this is done, the wrapped real handler, defined with the WrappedHandler option is imported and called.

The result is sys.path is correctly configured before the real handler is imported and executed, and everything works as expected.

To make this work, a slight modification is needed in the Apache virtual site configuration. The configuration for the 0.11 site is shown below:

        <Location /trac>
                SetHandler mod_python
                PythonPath "['/usr/local/trac0.11'] + sys.path"
                PythonHandler handler_wrapper
                PythonOption WrappedHandler trac.web.modpython_frontend
                PythonOption SiteDirs ['/usr/local/trac0.11']
                PythonOption TracEnvParentDir /path_to_trac0.11_envs/
                PythonOption TracUriRoot /
        </Location>