Pytest reorders order of python import lookups

Data Management
#1

When attempting to run pytest on daf_butler I had several errors related to using features that were only in sqlalchemy 1.2.X, and the version in use was 1.1.X.

I verified that with eups, the correct version of sqlalchemy was installed and setup. I then verified that PYTHONPATH was correctly set.

It turns out that I had a version of sqlalchemy 1.1.X installed in site-packages through conda. However, site-packages should be among the last thing searched if PYTHONPATH is set.

I did some digging, but the problem went away when I used the python shell for debugging.

This lead me to looking at the value of sys.path (the list of paths searched and their search order) in a pytest session and in a python shell. Needless to say they were different.

It turns out that pytest has a site.py file that programatically redefines sys.path, and in its re-declaration, it puts site-packages near the top of the list, before anything declared in the PYTHONPATH variable.

The consequence of this is that any package that is setup, through our eups method or any of the other normal python ways, that has a package of the same name in site-packages will not be used.

I don’t have time to dig into this any further at the moment, but the behavior is worrying to me. I’m not sure I trust a package that thinks it knows the order things should happen in better than what I as a user have specified. Then again it may simply be a bug or oversight in pytest.

I mostly wanted to make people aware of this issue, and prompt anyone who has more time to look into this.

#2

Thank you! This explains why @brianv0 is having trouble with pytest picking up the conda pytest version rather than the EUPS one when upgrading all our pytest infrastructure.

#3

Interesting. I didn’t see anything related in the existing pytest issues, so it’s probably worth filing one with them, as this is definitely unexpected behavior.

#4

The site.py file is created by setuptools with content from https://github.com/pypa/setuptools/blob/master/setuptools/site-patch.py

So I don’t think you can blame pytest; it doesn’t seem to be doing anything particularly unusual.

#5

I think that also explains why I always get warnings like this despite psutil not having anything inside it that is relevant:

python_psutil/5.4.3+2/lib/python/site.py:26: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp  # Avoid import loop in Python >= 3.3
#6

To be fair I didn’t verify that site.py in pytest was 100% what was rewriting the path (partly because I didnt have the time to grok all the 2 letter variables), but I did verify something in py.test was reworking the paths, and this seemed like a likely culprit. If someone has more time to trace it all the way down, or I will try to get back to it as some point.

My original point stats that when using pytest, sys.path is not in the same order it would be with normal python execution, and this leads to unexpected behavior.

#7

It looks like this may be related:

#8

At least they now seem to agree that PYTHONPATH behavior is in fact well-defined and should take precedence over other packages. @brianv0 can you make a comment on that ticket so that it gets a bit of a kick?

#9

In case it helps anyone else who has been having problems with builds (@erykoff?), I’ve run into some problems building packages today because of this; I’m frequently seeing older versions of many packages from my base conda environment being chosen instead of newer versions in my activated conda environment or EUPS. I’ve seen that with pytest, py, and sqlalchemy so far.

I’ve addressed that by removing everything I could from my base conda environment (a very tedious process as I couldn’t find a way to say “remove everything that isn’t a dependency of conda itself”). I suspect any stacks (including shared stacks) that were created before the adoption of a custom lsst-scipipe for our conda dependencies are similarly broken, as they’ll have those old versions of many of our dependencies.

I think we have to recommend that anyone who has a stack from those days either start clean or do something like my manual removal of packages from the base environment.

#10

Interesting. Though I don’t think this is the exact problem I’m seeing, since the required version of py which is needed by the new pytest simply isn’t installed in the shared stack at the moment. Though since @yusra might not be seeing the same problems I am, maybe it could be related, but then I have even less of an idea of what’s going on.