Failure to run demo using v13_0 release

Support
#1

I have a fresh conda install of v13_0 on RHEL6-64. I started by installing a new copy of Miniconda 4.3.1. I chose to install DM in the root environment of my miniconda install, rather than creating a separate conda env. That bit seemed to go just fine.

Now I want to run the demo as described here: https://pipelines.lsst.io/install/demo.html. I set up my environment:

> export PATH=/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/bin:${PATH}
> source scl_source enable devtoolset-3
> source eups-setups.sh
> setup lsst_apps
> PS1="[v13_0] “

Note that I’ve tried both lsst_apps and lsst_distrib as well as devtoolset-3 and devtoolset-4 but the result is the same. I’ve downloaded the demo and do:

[v13_0] cd lsst_dm_stack_demo-13.0
[v13_0] setup obs_sdss
[v13_0] ./bin/demo.sh --small
Setting up: astrometry_net_data             Flavor: Linux64    Version: LOCAL:/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/lsst_dm_stack_demo-13.0/astrometry_net_data
Could not import lsstcppimport; please ensure the base package has been built (not just setup).

Traceback (most recent call last):
  File "/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/opt/lsst/pipe_tasks/bin/processCcd.py", line 23, in <module>
    from lsst.pipe.tasks.processCcd import ProcessCcdTask
  File "/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/opt/lsst/pipe_tasks/python/lsst/pipe/tasks/processCcd.py", line 22, in <module>
    from lsst.ip.isr import IsrTask
  File "/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/opt/lsst/ip_isr/python/lsst/ip/isr/__init__.py", line 24, in <module>
    from .isrLib import *
  File "/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/opt/lsst/ip_isr/python/lsst/ip/isr/isrLib.py", line 27, in <module>
    _isrLib = swig_import_helper()
  File "/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/opt/lsst/ip_isr/python/lsst/ip/isr/isrLib.py", line 26, in swig_import_helper
    return importlib.import_module('_isrLib')
  File "/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named _isrLib

Any suggestions? Thanks!

#2

setup miniconda2?

#3

I don’t recognize that module… but I tried it anyhow:

[v13_0] setup miniconda2
Unable to find an acceptable version of miniconda2
I was unclear, this is Miniconda2 that I installed, so it is a python 2.7 environment:
[v13_0] python --version
Python 2.7.13 :: Continuum Analytics, Inc.

#4

I don’t expect to see this warning on a Linux system.

This suggests that python load paths are correctly set up. Is there an _isrLib.so file in that directory?

#5

Yes, it does appear to be there, with content:

[v13_0] pwd
/nfs/farm/g/lsst/u1/software/rh6-64/DMstack/v13_0/opt/lsst/ip_isr/python/lsst/ip/isr
[v13_0] ls
SConscript    assembleCcdTask.py   isr.py     isrLib.pyc      linearize.py
__init__.py   assembleCcdTask.pyc  isr.pyc    isrLib_wrap.cc  linearize.pyc
__init__.pyc  fringe.py           isrLib.i   isrTask.py      version.py
_isrLib.so    fringe.pyc       isrLib.py  isrTask.pyc     version.pyc
[v13_0] ls -l _isrLib.so
-rwxr-xr-x 2 heather lsst 11668216 Feb 28 21:19 _isrLib.so
[v13_0]

Perhaps I should try this again and try a new environment rather than the root env. I think the paths may be slightly different in that context and perhaps it is causing some of my woes. Just to clarify, in my case, I don’t need a separate conda environment and I was running up against an 80 character limit in the swig module that is being supplied. If i have too long a path, the conda install of swig fails, saying it needs to be rebuilt using conda-build 2. I realize we’re moving away from swig, so it hardly seemed worthwhile to do anything other than shorten my path as much as possible - so I started over using a shortened path and figured while I was at it, just use the conda root environment… Or maybe I need to further shorten my path… what I have above with the subdirectories is a bit long - but conda did not complain.

#6

No change… I reinstalled from scratch starting with a fresh Miniconda2 4.3.1 install, followed by a conda install of lsst-distrib. This time, I just called the env “lsst” as mentioned in the installation instructions. Has anyone else tried this conda install on a RHEL6 system or something equivalent?

bash-4.1$ source activate lsst
bash-4.1$ source eups-setups.sh
bash-4.1$ pwd
/nfs/farm/g/lsst/u1/testDM/lsst_dm_stack_demo-13.0
bash-4.1$ setup lsst_distrib
bash-4.1$ setup obs_sdss
bash-4.1$ ./bin/demo.sh
Setting up: astrometry_net_data             Flavor: Linux64    Version: LOCAL:/nfs/farm/g/lsst/u1/testDM/lsst_dm_stack_demo-13.0/astrometry_net_data
Could not import lsstcppimport; please ensure the base package has been built (not just setup).

Traceback (most recent call last):
  File "/nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/opt/lsst/pipe_tasks/bin/processCcd.py", line 23, in <module>
    from lsst.pipe.tasks.processCcd import ProcessCcdTask
  File "/nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/opt/lsst/pipe_tasks/python/lsst/pipe/tasks/processCcd.py", line 22, in <module>
    from lsst.ip.isr import IsrTask
  File "/nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/opt/lsst/ip_isr/python/lsst/ip/isr/__init__.py", line 24, in <module>
    from .isrLib import *
  File "/nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/opt/lsst/ip_isr/python/lsst/ip/isr/isrLib.py", line 27, in <module>
    _isrLib = swig_import_helper()
  File "/nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/opt/lsst/ip_isr/python/lsst/ip/isr/isrLib.py", line 26, in swig_import_helper
    return importlib.import_module('_isrLib')
  File "/nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named _isrLib
#7

and here is the _isrLib.so:

ls /nfs/farm/g/lsst/u1/testDM/v13/envs/lsst/opt/lsst/ip_isr/python/lsst/ip/isr/
SConscript    assembleCcdTask.py   isr.py     isrLib.pyc      linearize.py
__init__.py   assembleCcdTask.pyc  isr.pyc    isrLib_wrap.cc  linearize.pyc
__init__.pyc  fringe.py           isrLib.i   isrTask.py      version.py
 _isrLib.so    fringe.pyc       isrLib.py  isrTask.pyc     version.pyc
#8

Please could you try:

$ cd $IP_ISR_DIR/python/lsst/ip/isr
$ python -c "import _isrLib"

?

#9

I have a glibc mismatch. I don’t think glibc 2.14 is available for RH6 typically. Any chance the conda binaries could be provided with support for an earlier glibc? Otherwise, I think I’ll try a source build.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./_isrLib.so)

bash-4.1$ ldd --version
ldd (GNU libc) 2.12
#10

@jmatt what OS were the conda packages built on? Were these not done on CentOS5?

#11

They were built on CentOS7 with the system gcc.

#12

Ok. Our release notes should say that we don’t expect the conda binaries to work on CentOS6. Do we say that?

#13

The reference platform for the LSST Science Pipelines is CentOS 7 however individual developers compile on a variety of Linux and macOS operating systems so if you are on a similar platform you should be able to build from source successfully.
We also offer Conda binaries and Docker images if you do not wish to install the Science Pipelines from source.

https://pipelines.lsst.io/install/newinstall.html#source-install-redhat-prereqs
I would have expected the conda binaries to be provided for those who may be unable to build from source. In our case, we’re hoping to continue to use the DMstack at SLAC on the public machines, which are RH6-64, and make a DMstack installation generally available for the camera team. SLAC has not jumped whole-hog onto RH7 quite yet, though of course individual users have CentOS7 and similar OSes.
I suspect I can build from source, but we have continued to hope that the conda installation option would be useful to us.

#14

@heather999 is our Docker release not going to work for you for some reason?

We are moving to a binary distribution solution that will be easier to produce on multiple platforms, but if you have a real issue and can’t build from source, let me know and I’ll generate you RHEL6 conda binaries for 13.0.

#15

Hi @frossie,
I’m building from source now and I fully expect that will work. Even if CentOS7 is the preferred environment, I am hoping that there is a commitment to supporting some of these legacy OSes - particularly RH6 and friends. This is ultimately for a production environment for the camera work, Docker containers are not really feasible in that environment - at least not yet. I am curious to hear what binary distribution DM is moving toward - we’re actually considering conda distributions for another non-LSST project and I’m a little alarmed that this may be more difficult to support than we anticipate.