Difficulty installing on Python 3.6

Support
#1

Attempt to install halted at:
eups distrib install -t v13_0 lsst_distrib

An error comes up related to test failures due to “numpy boolean subtract.” Is this error is due to using Python 3.6? Colleagues who have used Python 2.7 have not run into this issue.

Stacktrace below:

***** error: from /home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0/build.log:
File “tests/testSimpleTable.py”, line 346, in testExtract
self.assertFloatsEqual(d[“a_b_c2”], catalog.get(“a_b_c2”)[idx])
File “/home/mnewsome/envs/lsst_maf/lsst/Linux64/utils/13.0/python/lsst/utils/tests.py”, line 652, in assertFloatsEqual
return assertFloatsAlmostEqual(testCase, lhs, rhs, rtol=0, atol=0, **kwargs)
File “/home/mnewsome/envs/lsst_maf/lsst/Linux64/utils/13.0/python/lsst/utils/tests.py”, line 568, in assertFloatsAlmostEqual
diff = lhs - rhs
TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead.

Ran 23 tests in 0.387s

FAILED (errors=1)
The following tests failed:
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0/afw-13.0/tests/.tests/testSourceTable.py.failed
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0/afw-13.0/tests/.tests/testAstropyTableViews.py.failed
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0/afw-13.0/tests/.tests/testSimpleTable.py.failed
3 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

  • exit -4
#2

Probably is py3.6 + numpy 1.12 related. Do you need v13.0 of the stack or can you use a more recent weekly? I believe w.2017.23 has worked for me.

#3

Thanks for the reply! I gave the w_2017_23 version a try and while it got further along, essentially the same problem happened at a later point. At this point I believe it is likely a problem with py3.6 as I have been able to successfully run the installation in a clean directory using the Miniconda as recommended for the package. This is still concerning as one should be able to run their own python environment without running into these repetitive roadblocks. Below is the stacktrace just in case someone wants to see the nature of the area.

***** error: from /home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/build.log:
self.catalog.columns[“a_flag”])
File “/home/mnewsome/envs/lsst_maf/lsst/Linux64/utils/13.0-4-g2675101+1/python/lsst/utils/tests.py”, line 638, in assertFloatsEqual
return assertFloatsAlmostEqual(testCase, lhs, rhs, rtol=0, atol=0, **kwargs)
File “/home/mnewsome/envs/lsst_maf/lsst/Linux64/utils/13.0-4-g2675101+1/python/lsst/utils/tests.py”, line 554, in assertFloatsAlmostEqual
diff = lhs - rhs
TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead.

Ran 20 tests in 0.255s

FAILED (errors=1)
The following tests failed:
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/afw-13.0-60-gfc1c281/tests/.tests/testAstropyTableViews.py.failed
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/afw-13.0-60-gfc1c281/tests/.tests/testPupilFactory.py.failed
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/afw-13.0-60-gfc1c281/tests/.tests/testSimpleTable.py.failed
/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/afw-13.0-60-gfc1c281/tests/.tests/testSourceTable.py.failed
4 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

  • exit -4
    eups distrib: Failed to build afw-13.0-60-gfc1c281.eupspkg: Command:
    source /home/mnewsome/envs/lsst_maf/lsst/eups/bin/setups.sh; export EUPS_PATH=/home/mnewsome/envs/lsst_maf/lsst; (/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/build.sh) >> /home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/build.log 2>&1 4>/home/mnewsome/envs/lsst_maf/lsst/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/build.msg
    exited with code 252
#4

I’m running python 3.5 myself (in an LSST miniconda environment). I believe @natelust might be running in 3.6?

What version of numpy do you have installed? Given the error, I suspect the incompatibility may be with numpy rather than python. Although, if we really are doing subtraction on booleans, or using assertFloatsAlmostEqual on boolean arrays, we shouldn’t be.

#5

Not, I believe, Python 3.6 (which is working fine for me), but rather NumPy [1.13.0] (https://github.com/numpy/numpy/releases/tag/v1.13.0).

As @parejkoj says, we are clearly doing the wrong thing there though — we should ticket and fix it. In the short term, though, the easiest workaround might be to drop back to an older NumPy.

#6

DM-10926.

#7

I ended up starting from scratch in a clean directory and just allowing the installation to also include its Miniconda package as recommended, and it has been alright so far. I do agree with the concerns over numpy, though. Thank you for submitting the ticket!

#8

Just as verification I built the stack with python 3.6 today (I updated my python version with conda install python last week) everything built fine asside from an esutils problem (related to updating python but lsstsw didn’t know the package needed rebuilt)

#9

I’ve been running py3.6 and updated numpy packages myself lately, and have been running into some of these problems. I didn’t have numpy 1.13 though. It looks like py3.6 + numpy are getting stricter about checking types, etc., which I think people were aware of coming but it’s hard to find all of the broken cases.

It might actually be a useful idea for square to run an occasionally lsstsw build with the newest python + numpy / etc. packages just to look for broken things like this – just to get the developers of the packages a chance to know what’s coming. (this is however, where I usually hear … but that’s why we release our own python + set of requirements!)

FWIW, I do believe you can build sims_maf using sims master and the w.2017.23 tag, with py3.6 + numpy 1.12.