EUPS build installation failure: ap_verify test failure

fraserw · April 18, 2020, 6:49pm

Happy isolation!

Having trouble with the eups build step in installing the new pipeline. Full build below. Anyone have an idea how to get around this?

FWIW, the same error appears when trying to build v18.1 as well.

Thanks again for the help!

newinstall and eups build output:

fraserw@lsst-test:~/lsst_stack$ curl -OL https://raw.githubusercontent.com/lsst/lsst/19.0.0/scripts/newinstall.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 30621 100 30621 0 0 854k 0 --:–:-- --:–:-- --:–:-- 854k
fraserw@lsst-test:~/lsst_stack$ bash newinstall.sh -ct

LSST Software Stack Builder

curl: (23) Failed writing body (1303 != 1371)
!!! This script differs from the official version on the distribution
server. If this is not intentional, get the current version from here:
https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh
Detected git version 2.17.1. OK.

Unable to locate python.

Please set up a compatible python interpreter, prepend it to your PATH,
and rerun this script. Alternatively, we can set up the Miniconda Python
distribution for you.

In addition to Python >= 3.6, some LSST packages depend on recent versions
of numpy, matplotlib, and scipy. If you do not have all of these, the
installation may fail. Using the Miniconda Python distribution will ensure
all these are set up.

Would you like us to install the Miniconda Python distribution (if unsure,
say yes)?yes
::: Deploying Miniconda3-4.7.10-Linux-x86_64.sh
################################################################################################################################## 100.0%
PREFIX=/home/fraserw/lsst_stack/python/miniconda3-4.7.10
Unpacking payload …
Collecting package metadata (current_repodata.json): done
Solving environment: done

Package Plan

environment location: /home/fraserw/lsst_stack/python/miniconda3-4.7.10

added / updated specs:
- _libgcc_mutex==0.1=main
- asn1crypto==0.24.0=py37_0
- bzip2==1.0.8=h7b6447c_0
- ca-certificates==2019.5.15=0
- certifi==2019.6.16=py37_0
- cffi==1.12.3=py37h2e261b9_0
- chardet==3.0.4=py37_1
- conda-package-handling==1.3.11=py37_0
- conda==4.7.10=py37_0
- cryptography==2.7=py37h1ba5d50_0
- idna==2.8=py37_0
- libarchive==3.3.3=h5d8350f_5
- libedit==3.1.20181209=hc058e9b_0
- libffi==3.2.1=hd88cf55_4
- libgcc-ng==9.1.0=hdf63c60_0
- libstdcxx-ng==9.1.0=hdf63c60_0
- libxml2==2.9.9=hea5a465_1
- lz4-c==1.8.1.2=h14c3975_0
- lzo==2.10=h49e0be7_2
- ncurses==6.1=he6710b0_1
- openssl==1.1.1c=h7b6447c_1
- pip==19.1.1=py37_0
- pycosat==0.6.3=py37h14c3975_0
- pycparser==2.19=py37_0
- pyopenssl==19.0.0=py37_0
- pysocks==1.7.0=py37_0
- python-libarchive-c==2.8=py37_11
- python==3.7.3=h0371630_0
- readline==7.0=h7b6447c_5
- requests==2.22.0=py37_0
- ruamel_yaml==0.15.46=py37h14c3975_0
- setuptools==41.0.1=py37_0
- six==1.12.0=py37_0
- sqlite==3.29.0=h7b6447c_0
- tk==8.6.8=hbc83047_0
- tqdm==4.32.1=py_0
- urllib3==1.24.2=py37_0
- wheel==0.33.4=py37_0
- xz==5.2.4=h14c3975_4
- yaml==0.1.7=had09818_2
- zlib==1.2.11=h7b6447c_3
- zstd==1.3.7=h0b5b093_0

The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
asn1crypto pkgs/main/linux-64::asn1crypto-0.24.0-py37_0
bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
ca-certificates pkgs/main/linux-64::ca-certificates-2019.5.15-0
certifi pkgs/main/linux-64::certifi-2019.6.16-py37_0
cffi pkgs/main/linux-64::cffi-1.12.3-py37h2e261b9_0
chardet pkgs/main/linux-64::chardet-3.0.4-py37_1
conda pkgs/main/linux-64::conda-4.7.10-py37_0
conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.3.11-py37_0
cryptography pkgs/main/linux-64::cryptography-2.7-py37h1ba5d50_0
idna pkgs/main/linux-64::idna-2.8-py37_0
libarchive pkgs/main/linux-64::libarchive-3.3.3-h5d8350f_5
libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
libffi pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
libxml2 pkgs/main/linux-64::libxml2-2.9.9-hea5a465_1
lz4-c pkgs/main/linux-64::lz4-c-1.8.1.2-h14c3975_0
lzo pkgs/main/linux-64::lzo-2.10-h49e0be7_2
ncurses pkgs/main/linux-64::ncurses-6.1-he6710b0_1
openssl pkgs/main/linux-64::openssl-1.1.1c-h7b6447c_1
pip pkgs/main/linux-64::pip-19.1.1-py37_0
pycosat pkgs/main/linux-64::pycosat-0.6.3-py37h14c3975_0
pycparser pkgs/main/linux-64::pycparser-2.19-py37_0
pyopenssl pkgs/main/linux-64::pyopenssl-19.0.0-py37_0
pysocks pkgs/main/linux-64::pysocks-1.7.0-py37_0
python pkgs/main/linux-64::python-3.7.3-h0371630_0
python-libarchive~ pkgs/main/linux-64::python-libarchive-c-2.8-py37_11
readline pkgs/main/linux-64::readline-7.0-h7b6447c_5
requests pkgs/main/linux-64::requests-2.22.0-py37_0
ruamel_yaml pkgs/main/linux-64::ruamel_yaml-0.15.46-py37h14c3975_0
setuptools pkgs/main/linux-64::setuptools-41.0.1-py37_0
six pkgs/main/linux-64::six-1.12.0-py37_0
sqlite pkgs/main/linux-64::sqlite-3.29.0-h7b6447c_0
tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0
tqdm pkgs/main/noarch::tqdm-4.32.1-py_0
urllib3 pkgs/main/linux-64::urllib3-1.24.2-py37_0
wheel pkgs/main/linux-64::wheel-0.33.4-py37_0
xz pkgs/main/linux-64::xz-5.2.4-h14c3975_4
yaml pkgs/main/linux-64::yaml-0.1.7-had09818_2
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
zstd pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0

Preparing transaction: done
Executing transaction: done
installation finished.
################################################################################################################################## 100.0%################################################################################################################################## 100.0%
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.7.10
latest version: 4.8.3

Please update conda by running

$ conda update -n base -c defaults conda

To activate this environment, use

$ conda activate lsst-scipipe-4d7b902

To deactivate an active environment, use

$ conda deactivate

name: lsst-scipipe-4d7b902
channels:

conda-forge
defaults
dependencies:
_libgcc_mutex=0.1=main
apipkg=1.5=py37_0
arrow-cpp=0.13.0=py37h117bdfb_0
asn1crypto=0.24.0=py37_0
astropy=3.2.3=py37h516909a_0
atomicwrites=1.3.0=py_0
attrs=19.1.0=py_0
blas=1.0=openblas
blosc=1.15.0=hd408876_0
boost-cpp=1.67.0=h14c3975_4
bottleneck=1.2.1=py37h035aef0_1
brotli=1.0.7=he6710b0_0
bzip2=1.0.6=h14c3975_5
ca-certificates=2019.1.23=0
certifi=2019.3.9=py37_0
cffi=1.12.2=py37h2e261b9_1
chardet=3.0.4=py37_1
coverage=4.5.4=py37h7b6447c_0
cryptography=2.6.1=py37h1ba5d50_0
cycler=0.10.0=py37_0
cython=0.29.6=py37he6710b0_0
dbus=1.13.6=h746ee38_0
double-conversion=3.1.5=he6710b0_1
execnet=1.7.1=py_0
expat=2.2.6=he6710b0_0
fontconfig=2.13.0=h9420a91_0
freetype=2.9.1=h8a8886c_1
future=0.17.1=py37_0
gflags=2.2.2=he6710b0_0
glib=2.56.2=hd408876_0
glog=0.4.0=he6710b0_0
gst-plugins-base=1.14.0=hbbd80ab_1
gstreamer=1.14.0=hb453b48_1
h5py=2.9.0=py37h7918eee_0
hdf5=1.10.4=hb1b8bf9_0
icu=58.2=h9c2bf20_1
idna=2.8=py37_0
jpeg=9b=h024ee3a_2
kiwisolver=1.0.1=py37hf484d3e_0
libboost=1.67.0=h46d08c1_4
libedit=3.1.20181209=hc058e9b_0
libevent=2.1.8=h1ba5d50_0
libffi=3.2.1=hd88cf55_4
libgcc-ng=8.2.0=hdf63c60_1
libgfortran-ng=7.3.0=hdf63c60_0
libopenblas=0.3.3=h5a2b251_3
libpng=1.6.36=hbc83047_0
libprotobuf=3.6.0=hdbcaa40_0
libstdcxx-ng=8.2.0=hdf63c60_1
libuuid=1.0.3=h1bed415_2
libxcb=1.13=h1bed415_1
libxml2=2.9.9=he19cac6_0
lz4-c=1.8.1.2=h14c3975_0
lzo=2.10=h49e0be7_2
matplotlib=3.0.3=py37h5429711_0
more-itertools=6.0.0=py37_0
ncurses=6.1=he6710b0_1
nomkl=3.0=0
numexpr=2.6.9=py37h2ffa06c_0
numpy=1.16.2=py37h99e49ec_0
numpy-base=1.16.2=py37h2f8d375_0
openssl=1.1.1b=h7b6447c_1
pandas=0.24.2=py37he6710b0_0
pcre=8.43=he6710b0_0
pip=19.0.3=py37_0
pluggy=0.9.0=py37_0
psutil=5.6.1=py37h7b6447c_0
py=1.8.0=py37_0
pyarrow=0.13.0=py37he6710b0_0
pycparser=2.19=py37_0
pyopenssl=19.0.0=py37_0
pyparsing=2.3.1=py37_0
pyqt=5.9.2=py37h05f1152_2
pysocks=1.6.8=py37_0
pytables=3.5.1=py37h71ec239_0
pytest=4.5.0=py37_0
pytest-arraydiff=0.3=py37h39e3cac_0
pytest-astropy=0.5.0=py37_0
pytest-cov=2.7.1=py_0
pytest-doctestplus=0.3.0=py37_0
pytest-forked=1.0.2=py37_0
pytest-openfiles=0.3.2=py37_0
pytest-remotedata=0.3.1=py37_0
pytest-xdist=1.29.0=py_0
python=3.7.2=h0371630_0
python-dateutil=2.8.0=py37_0
pytz=2018.9=py37_0
pyyaml=5.1=py37h7b6447c_0
qt=5.9.7=h5867ecd_1
re2=2019.08.01=he6710b0_0
readline=7.0=h7b6447c_5
requests=2.21.0=py37_0
scikit-learn=0.20.3=py37h22eb022_0
scipy=1.2.1=py37he2b7bc3_0
setuptools=40.8.0=py37_0
sip=4.19.8=py37hf484d3e_0
six=1.12.0=py37_0
snappy=1.1.7=hbae5bb6_3
sqlalchemy=1.3.1=py37h7b6447c_0
sqlite=3.27.2=h7b6447c_0
thrift-cpp=0.11.0=h02b749d_3
tk=8.6.8=hbc83047_0
tornado=6.0.1=py37h7b6447c_0
urllib3=1.24.1=py37_0
wcwidth=0.1.7=py37_0
wheel=0.33.1=py37_0
wrapt=1.11.1=py37h7b6447c_0
xz=5.2.4=h14c3975_4
yaml=0.1.7=had09818_2
zlib=1.2.11=h7b6447c_3
zstd=1.3.7=h0b5b093_0
pip:
- deprecated==1.2.5
  prefix: /home/fraserw/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902

newinstall.sh: line 268: /etc/redhat-release: No such file or directory
Configured EUPS_PKGROOT:
Using python at /home/fraserw/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/bin/python to install EUPS
Installing EUPS (2.1.5)… done.
Creating startup scripts (bash) … done.
Creating startup scripts (ksh) … done.
Creating startup scripts (csh) … done.
Creating startup scripts (zsh) … done.

Bootstrap complete. To continue installing (and to use) the LSST stack type
one of:

source “/home/fraserw/lsst_stack/loadLSST.bash” # for bash
source “/home/fraserw/lsst_stack/loadLSST.csh” # for csh
source “/home/fraserw/lsst_stack/loadLSST.ksh” # for ksh
source “/home/fraserw/lsst_stack/loadLSST.zsh” # for zsh

Individual LSST packages may now be installed with the usual eups distrib install command. For example, to install the science pipeline elements
of the LSST stack, use:

eups distrib install lsst_apps

Next, read the documentation at:

https://pipelines.lsst.io

and feel free to ask any questions via the LSST Community forum:

https://community.lsst.org/c/support

                                   Thanks!
                                            -- The LSST Software Teams
                                                   http://dm.lsst.org/

fraserw@lsst-test:~/lsst_stack$ source loadLSST.bash
(lsst-scipipe-4d7b902) fraserw@lsst-test:~/lsst_stack$ eups distrib install -t v19_0_0 lsst_distrib

[ 1/121 ] apr 1.5.2 done.
[ 2/121 ] cfitsio 3360.lsst6 done.
[ 3/121 ] doxygen 1.8.13.lsst3 done.
[ 4/121 ] eigen 3.3.7 done.
[ 5/121 ] fftw 3.3.4.lsst2 done.
[ 6/121 ] gsl 2.6 done.
[ 7/121 ] libyaml 0.1.7 done.
[ 8/121 ] minuit2 5.34.14 done.
[ 9/121 ] mpich 3.2.1 done.
[ 10/121 ] obs_decam_data 19.0.0 done.
[ 11/121 ] obs_lsst_data 19.0.0 done.
[ 12/121 ] obs_subaru_data 19.0.0 done.
[ 13/121 ] obs_test_data 19.0.0 done.
[ 14/121 ] python 3.7 …
Using /home/fraserw/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/bin/python (Python 3.7.2).
[ 14/121 ] python 3.7 done.
[ 15/121 ] starlink_ast lsst-dev-gd6cc4e835a done.
[ 16/121 ] xpa 2.1.15.lsst3 done.
[ 17/121 ] apr_util 1.5.4.lsst1 done.
[ 18/121 ] astropy 3.1.2 …
Using externally provided astropy v3.2.3.
[ 18/121 ] astropy 3.1.2 done.
[ 19/121 ] boost 1.69.lsst1 done.
[ 20/121 ] mpi 0.0.1+3 done.
[ 21/121 ] numpy 1.16.2 …
Using externally provided numpy v1.16.2.
[ 21/121 ] numpy 1.16.2 done.
[ 22/121 ] pycodestyle 2.5.0+2 done.
[ 23/121 ] pyflakes 2.1.0+2 done.
[ 24/121 ] pykg_config 1.3.0+6 done.
[ 25/121 ] pytest_session2file 0.1.9-1-gc58ee16 done.
[ 26/121 ] python_future 0.16.0+5 done.
[ 27/121 ] python_mccabe 0.6.1-1-g86ed08f done.
[ 28/121 ] python_psutil 5.4.3+4 done.
[ 29/121 ] pyyaml 5.1+2 done.
[ 30/121 ] requests 2.9.1.lsst1+6 done.
[ 31/121 ] scons 3.0.0.lsst1+5 done.
[ 32/121 ] sqlalchemy 1.3.8 done.
[ 33/121 ] ws4py 0.4.2+5 done.
[ 34/121 ] coord 1.1.lsst+2 done.
[ 35/121 ] esutil 0.6.3 done.
[ 36/121 ] firefly_client lsst-dev-g97c9674977 done.
[ 37/121 ] flake8 3.7.5+3 done.
[ 38/121 ] healpy 1.10.3.lsst2+3 done.
[ 39/121 ] log4cxx 0.10.0.lsst7+1 done.
[ 40/121 ] matplotlib 3.0.3 …
Using externally provided matplotlib v3.0.3.
[ 40/121 ] matplotlib 3.0.3 done.
[ 41/121 ] mpi4py 3.0.0+2 done.
[ 42/121 ] pybind11 2.2.4.lsst1+2 done.
[ 43/121 ] scipy 1.2.1 …
Using externally provided scipy v1.2.1.
[ 43/121 ] scipy 1.2.1 done.
[ 44/121 ] treecorr 3.2.3.lsst4+3 done.
[ 45/121 ] autograd 1.1.54170B done.
[ 46/121 ] fgcm lsst-dev-g321fba8858 done.
[ 47/121 ] galsim 2.2.1.lsst done.
[ 48/121 ] lmfit 0.9.3+11 done.
[ 49/121 ] ndarray 1.5.3+2 done.
[ 50/121 ] pep8_naming 0.8.2+3 done.
[ 51/121 ] proxmin lsst-dev-gf7baf2a6dd done.
[ 52/121 ] pytest_flake8 1.0.4-1-g8dd8c06 done.
[ 53/121 ] scarlet lsst-dev-g169b417050 done.
[ 54/121 ] sconsUtils 19.0.0 done.
[ 55/121 ] astrometry_net_data 10.0-1-gc676a60+5 done.
[ 56/121 ] astshim 19.0.0 done.
[ 57/121 ] base 19.0.0 done.
[ 58/121 ] jointcal_cholmod master-g25991af22f+10 done.
[ 59/121 ] psfex 19.0.0 done.
[ 60/121 ] verify_metrics 19.0.0 done.
[ 61/121 ] pex_exceptions 19.0.0 done.
[ 62/121 ] sphgeom 19.0.0 done.
[ 63/121 ] utils 19.0.0 done.
[ 64/121 ] daf_base 19.0.0 done.
[ 65/121 ] geom 19.0.0 done.
[ 66/121 ] log 19.0.0 done.
[ 67/121 ] astro_metadata_translator 0.1.0-5-g1039c3c+8 done.
[ 68/121 ] pex_policy 19.0.0 done.
[ 69/121 ] daf_persistence 19.0.0 done.
[ 70/121 ] pex_config 19.0.0 done.
[ 71/121 ] afw 19.0.0 done.
[ 72/121 ] ctrl_orca 19.0.0 done.
[ 73/121 ] ctrl_execute 19.0.0 done.
[ 74/121 ] daf_butler 19.0.0 done.
[ 75/121 ] dax_ppdb 19.0.0 done.
[ 76/121 ] display_ds9 19.0.0 done.
[ 77/121 ] display_firefly 19.0.0 done.
[ 78/121 ] display_matplotlib 19.0.0 done.
[ 79/121 ] shapelet 19.0.0 done.
[ 80/121 ] ctrl_platform_lsstvc 19.0.0 done.
[ 81/121 ] pipe_base 19.0.0 done.
[ 82/121 ] skymap 19.0.0 done.
[ 83/121 ] coadd_utils 19.0.0 done.
[ 84/121 ] ctrl_mpexec 19.0.0 done.
[ 85/121 ] ctrl_pool 19.0.0 done.
[ 86/121 ] obs_base 19.0.0 done.
[ 87/121 ] verify 19.0.0+2 done.
[ 88/121 ] meas_base 19.0.0 done.
[ 89/121 ] obs_test 19.0.0 done.
[ 90/121 ] meas_algorithms 19.0.0 done.
[ 91/121 ] meas_extensions_simpleShape 19.0.0 done.
[ 92/121 ] ap_association 19.0.0+2 done.
[ 93/121 ] ip_isr 19.0.0 done.
[ 94/121 ] meas_astrom 19.0.0 done.
[ 95/121 ] meas_deblender 19.0.0 done.
[ 96/121 ] meas_extensions_photometryKron 19.0.0 done.
[ 97/121 ] meas_extensions_psfex 19.0.0 done.
[ 98/121 ] meas_extensions_scarlet 19.0.0 done.
[ 99/121 ] meas_extensions_shapeHSM 19.0.0 done.
[ 100/121 ] meas_modelfit 19.0.0 done.
[ 101/121 ] cp_pipe 19.0.0 done.
[ 102/121 ] ip_diffim 19.0.0+2 done.
[ 103/121 ] meas_extensions_convolved 19.0.0 done.
[ 104/121 ] obs_ctio0m9 19.0.0 done.
[ 105/121 ] pipe_tasks 19.0.0+2 done.
[ 106/121 ] obs_cfht 19.0.0+2 done.
[ 107/121 ] obs_decam 19.0.0+2 done.
[ 108/121 ] obs_lsst 19.0.0+2 done.
[ 109/121 ] obs_lsstSim 19.0.0+2 done.
[ 110/121 ] obs_sdss 19.0.0+2 done.
[ 111/121 ] obs_subaru 19.0.0+2 done.
[ 112/121 ] pipe_drivers 19.0.0+2 done.
[ 113/121 ] synpipe 19.0.0+2 done.
[ 114/121 ] lsst_obs 19.0.0+2 done.
[ 115/121 ] validate_drp 19.0.0+2 done.
[ 116/121 ] ap_pipe 19.0.0+2 done.
[ 117/121 ] jointcal 19.0.0+2 done.
[ 118/121 ] ap_verify 19.0.0+2 …

***** error: from /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/build.log:
tests/test_ingestion.py 175 7 54 3 96%
tests/test_workspace.py 54 2 6 1 95%

TOTAL 878 264 158 12 67%
Coverage HTML written to dir tests/.tests/pytest-ap_verify.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-ap_verify.xml-cov-ap_verify.xml

========= 1 failed, 41 passed, 14 skipped, 54 warnings in 9.06 seconds =========
/home/fraserw/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/site-packages/astropy/config/configuration.py:532: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))
Coverage.py warning: No data was collected. (no-data-collected)
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/ap_verify-19.0.0+2/tests/.tests/pytest-ap_verify.xml.failed
The following tests failed:
/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/ap_verify-19.0.0+2/tests/.tests/pytest-ap_verify.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

exit -4
eups distrib: Failed to build ap_verify-19.0.0+2.eupspkg: Command:
source “/home/fraserw/lsst_stack/eups/2.1.5/bin/setups.sh”; export EUPS_PATH="/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902"; (/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/build.sh) >> /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/build.log 2>&1 4>/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/build.msg
exited with code 252
(lsst-scipipe-4d7b902) fraserw@lsst-test:~/lsst_stack$

price · April 20, 2020, 1:11pm

Can you attach /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ap_verify-19.0.0+2/build.log?

fraserw · April 20, 2020, 3:12pm

Sure!

build.log (2.7 MB)

price · April 20, 2020, 3:53pm

The error is:

___________________ IngestionTestSuite.testFindMatchingFiles ___________________
[gw2] linux -- Python 3.7.2 /home/fraserw/lsst_stack/python/miniconda3-4.7.10/en
vs/lsst-scipipe-4d7b902/bin/python3.7

self = <test_ingestion.IngestionTestSuite testMethod=testFindMatchingFiles>

    def testFindMatchingFiles(self):
        """Test that _findMatchingFiles finds the desired files.
        """
        testDir = os.path.join(IngestionTestSuite.testData)
    
        self.assertSetEqual(
            ingestion._findMatchingFiles(testDir, ['raw_*.fits.gz']),
            {os.path.join(testDir, 'raw', f) for f in
             {'raw_v1_fg.fits.gz', 'raw_v2_fg.fits.gz', 'raw_v3_fr.fits.gz'}}
        )
        self.assertSetEqual(
            ingestion._findMatchingFiles(testDir, ['raw_*.fits.gz'], ['*fr*']),
>           {os.path.join(testDir, 'raw', f) for f in {'raw_v1_fg.fits.gz', 'raw_v2_fg.fits.gz'}}
        )
E       AssertionError: Items in the second set but not the first:
E       '/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_test/19.0.0/data/input/raw/raw_v2_fg.fits.gz'
E       '/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_test/19.0.0/data/input/raw/raw_v1_fg.fits.gz'

tests/test_ingestion.py:364: AssertionError

Looking at the code, I believe the original author is @kfindeisen. Maybe he can suggest a solution?

kfindeisen · April 20, 2020, 4:32pm

Oh dear. It’s a bug that appears only if your directory path happens to include the substring “fr”.

I’ll fix it so that the pattern is only matched against filenames.

fraserw · April 20, 2020, 5:44pm

Hi @kfindeisen

Well that’s an unfortunately nefarious bug.

When it does get fixed, will there be special instructions to build off the updated code, rather than with a specific version (eg., eups distrib install -t v19_0_0 lsst_distrib
)?

Thanks

kfindeisen · April 20, 2020, 6:02pm

You will be able to build the corrected code using a daily or weekly build.

I hope to get it merged soon; sorry for the trouble!

price · April 20, 2020, 6:29pm

This bug prevents building a supported release. @swinbank, do you think this justifies a patch release of v19?

kfindeisen · April 20, 2020, 6:46pm

@fraserw The fix has been tested and merged. Starting tomorrow, it should be available through eups distrib install -t d_2020_04_21 lsst_distrib.

Sorry again for the inconvenience. This bug had been around for two years, and you happened to be the first person to trigger it.

swinbank · April 20, 2020, 6:56pm

Yes; RFC-689.

Thanks for the fast action here, @kfindeisen and @price.

fraserw · April 20, 2020, 6:57pm

Thanks @kfindeisen

Any way to build on this today? Really need to get something tested ASAP.

kfindeisen · April 20, 2020, 7:25pm

Short of building in a different directory, I don’t think so. As far as I know eups distrib doesn’t support builds more recent than the latest daily.

ktl · April 20, 2020, 7:28pm

We used to have “emergency” instructions on how to bypass a failing test; let me see if I can dig them up. Basically you’d have to go into the EupsBuildDir and tweak things to get the package installed.

ktl · April 20, 2020, 7:29pm

There’s also running from Docker or CernVM FS if you don’t need to build from source.

ktl · April 20, 2020, 7:37pm

Again, not recommended, but https://confluence.lsstcorp.org/display/DM/Getting+started+with+stack+development#Gettingstartedwithstackdevelopment-Dealingwithtestfailures might help.

fraserw · April 20, 2020, 7:54pm

Yeah I see what you mean. Safer to just wait until tomorrow.

BTW I think this bug is persistent in v18.1 which I tried to install as well. If it’s possible to repair version 18 in the same way, that would be great, as I wish to be able to use 18 for consistency with some projects I’ve been working on.

fraserw · April 21, 2020, 8:06pm

Well shoot. The daily build fails to install daf_butler. build log attached.

@kfindeisen any ideas on how to get around this one?

Thanks

End of the eups build terminal output:

[ 74/121 ] daf_butler 19.0.0-53-gd1c14a24+1 …

***** error: from /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/daf_butler-19.0.0-53-gd1c14a24+1/build.log:
tests/test_time_utils.py 58 2 8 4 91%
tests/test_utils.py 103 2 4 1 97%

TOTAL 15422 2709 4858 631 80%
Coverage HTML written to dir tests/.tests/pytest-daf_butler.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-daf_butler.xml-cov-daf_butler.xml

======== 1 failed, 490 passed, 65 skipped, 13 warnings in 56.86 seconds ========
/home/fraserw/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/site-packages/astropy/config/configuration.py:532: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))
Coverage.py warning: No data was collected. (no-data-collected)
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/daf_butler-19.0.0-53-gd1c14a24+1/daf_butler-19.0.0-53-gd1c14a24+1/tests/.tests/pytest-daf_butler.xml.failed
The following tests failed:
/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/daf_butler-19.0.0-53-gd1c14a24+1/daf_butler-19.0.0-53-gd1c14a24+1/tests/.tests/pytest-daf_butler.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

exit -4
eups distrib: Failed to build daf_butler-19.0.0-53-gd1c14a24+1.eupspkg: Command:
source “/home/fraserw/lsst_stack/eups/2.1.5/bin/setups.sh”; export EUPS_PATH="/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902"; (/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/daf_butler-19.0.0-53-gd1c14a24+1/build.sh) >> /home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/daf_butler-19.0.0-53-gd1c14a24+1/build.log 2>&1 4>/home/fraserw/lsst_stack/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/daf_butler-19.0.0-53-gd1c14a24+1/build.msg

build.log (1.0 MB)

ktl · April 21, 2020, 8:18pm

For the daily, you need a master version of newinstall.sh which will bring in a new scipipe_conda_env environment that includes astropy 4.0 which supports the to_value() method.

fraserw · April 21, 2020, 11:53pm

Oh lord. RTFM right? Sorry to waste time by forgetting that tidbit.