PYTHONPATH Considered Harmful

(Thanks to Tim D. Smith and Augie Fackler for reviewing a draft. Any mistakes that remain are mine.)

The environment variable PYTHONPATH seems harmless enough. The official documentation refers to its function as "Augment the default search path for module files." However, in practice, setting this variable in a shell can lead to many confusing things.

For one, most directories are poorly suited to be on the Python search path. Consider, for example, the root directory of a typical Python project: it contains setup.py -- and so, if it were added to the current search path, import setup would become possible. (This is one reason to have src/ directories.) Often, directories added unwisely to the Python search path cause files to be imported from paths they do not expect to, and surprisingly conflict.

In addition, it now means that commands such as pip freeze do not describe reality correctly: just because foo==1.2.4 appears in the output, does not mean that import foo will import version 1.2.4 -- and only careful checking of the __file__ attribute will notice it.

Specifically, any bug reporting template that asks the output of pip freeze to be attached is rendered nearly useless.

Worse, active commands in pip would seem to have no effect: pip install --upgrade or pip uninstall would have no effect on the valiant foo package, safely ensconced in a directory on PYTHONPATH.

Finally, PYTHONPATH changes paths for both Python 2 and 3. This means code intended for one interpreter will end up in the other.

Manipulating PYTHONPATH should be reserved for specialized launchers which need to start a Python program despite a complicated system configuration. In those cases, PYTHONPATH should be set inside the launcher, so as not to effect any code outside it.

As always, when needing specialized launchers in order to run code, consider that "starting up should not be the most complicated thing that your program does". It is quite possible that other, existing tools, are already good enough. Maybe virtualenv, pip install -e or Pex. can solve the problem?