ImportError of 'undefined symbol'

Charles · September 19, 2019, 3:19pm

Hi,

I installed the hscPipe 7.9.1 with binary distribution (by setup EUPS_PKGROOT ) at Fedora30. It start from scratch. Although I got one library issue and two warnings (see below), the installation seems successful. However, I got library error when I run some test scripts (ingestCalibs.py, hscIngestImages.py, singleFrameDriver.py). I wonder anyone can give a advise to fix it?

Charles

Main error message:

(lsst-scipipe-1172c30) [ytchen@ichiro hscPipe7.9.1]$ ingestCalibs.py
Traceback (most recent call last):
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.9.1-hsc/bin/ingestCalibs.py”, line 2, in
from lsst.pipe.tasks.ingestCalibs import IngestCalibsTask
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.9.1-hsc/python/lsst/pipe/tasks/ingestCalibs.py”, line 5, in
from lsst.afw.fits import readMetadata
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/fits/init.py”, line 1, in
from .fitsLib import *
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/fits/fitsLib.py”, line 1, in
from .fits import *
ImportError: /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/daf_base/7.0-hsc/lib/libdaf_base.so: undefined symbol: _ZN5boost16re_detail_10690012perl_matcherIN9__gnu_cxx17__normal_iteratorIPKcSsEESaINS_9sub_matchIS6_EEENS_12regex_traitsIcNS_16cpp_regex_traitsIcEEEEE14construct_initERKNS_11basic_regexIcSD_EENS_15regex_constants12_match_flagsE

Issue 1:
I fixed it by replacing /array/users/ytchen/hscPipe/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/../lib/libstdc++.so.6 with /usr/lib64/libstdc++.so.6 (after running newinstall.sh).

ImportError while importing test module ‘/array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/tests/test_specFrame.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_specFrame.py:4: in <module>
import astshim as ast
python/astshim/init.py:26: in <module>
from .base import *
E ImportError: /array/users/ytchen/hscPipe/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/…/lib/libstdc++.so.6: version `GLIBCXX_3.4.26’ not found (required by /array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/lib/libastshim.so)

Two warnings when I install hscPipe 7.9.1:

[ 85/97 ] meas_deblender 7.0-hsc+1 (Linux64) … Warning: trouble retrieving meas_deblender-7.0-hsc+1@Linux64.tar.gz: IncompleteRead(1573727 bytes read, 8955030 more expected)
[ 89/97 ] meas_modelfit 7.0-hsc+1 (Linux64) done.
Failed to contact URL http://tigress-web.princeton.edu/~HSC/hscPipe7/Linux64/ ([Errno -2] Name or service not known)

timj · September 19, 2019, 4:07pm

Do you have a boost installed somewhere else on your system that is confusing the linker?

Charles · September 19, 2019, 4:16pm

I am not sure what is the boost you mean. I do find some python stuffs in /usr/include/boost/

price · September 19, 2019, 4:39pm

It sounds like you have ABI incompatibilities. I suggest installing from source rather than using a binary distribution.

Charles · September 19, 2019, 11:15pm

Actually, I encounter a error when I am installing from source code. It look likes a threading issue, but I couldn’t figure it out …

Here is the error message:

[ 55/97 ] base 7.0-hsc …
***** error: from /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log:
warn(ConfigurationMissingWarning(msg))
tests/test_packages.py::PackagesTestCase::testPackages
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/python_future/0.16.0+6/lib/python/future-0.16.0-py3.7.egg/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated, and in 3.8 it will stop working
from collections import Iterable
/array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))
– Docs: http://doc.pytest.org/en/latest/warnings.html
========= 1 failed, 14 passed, 1 skipped, 4 warnings in 70.02 seconds ==========
Coverage.py warning: No data was collected. (no-data-collected)
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
The following tests failed:
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build base-7.0-hsc.eupspkg: Command:
source “/array/users/ytchen/hscPipe7.9.1/eups/2.1.5/bin/setups.sh”; export EUPS_PATH="/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30"; (/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.sh) >> /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log 2>&1 4>/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.msg
exited with code 252

This is the last part of build.log

============================= test session starts ==============================
platform linux – Python 3.7.2, pytest-3.6.2, py-1.7.0, pluggy-0.6.0
rootdir: /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc, inifile: setup.cfg
plugins: flake8-1.0.4, xdist-1.20.1, forked-0.2, session2file-0.1.9, cov-2.5.1, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
passed
gw0 [16] / gw1 [16] / gw2 [16] / gw3 [16] / gw4 [16] / gw5 [16] / gw6 [16] / gw7 [16] / gw8 [16] / gw9 [16] / gw10 [16] / gw11 [16] / gw12 [16] / gw13 [16] / gw14 [16] / gw15 [16] / gw16 [16] / gw17 [16] / gw18 [16] / gw19 [16] / gw20 [16] / gw21 [16] / gw22 [16] / gw23 [16] / gw24 [16] / gw25 [16] / gw26 [16] / gw27 [16] / gw28 [16] / gw29 [16] / gw30 [16] / gw31 [16]
scheduling tests via LoadScheduling
s.[gw15] node down: Not properly terminated
fSlave restarting disabled
…
=================================== FAILURES ===================================
____________________________ tests/test_threads.py _____________________________
[gw15] linux – Python 3.7.2 /array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/python3.7
Slave ‘gw15’ crashed while running ‘tests/test_threads.py::ThreadsTestCase::testApi’
generated xml file: /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml

----------- coverage: platform linux, python 3.7.2-final-0 -----------
Name Stmts Miss Branch BrPart Cover
----------------------------------------------------------------
python/lsst/init.py 2 0 0 0 100%
python/lsst/base/init.py 4 0 0 0 100%
python/lsst/base/packages.py 118 23 40 2 80%
python/lsst/base/version.py 5 0 0 0 100%
python/lsstcppimport.py 1 0 0 0 100%
python/lsstimport.py 50 20 19 7 54%
tests/test_packages.py 50 1 2 1 96%
tests/test_threads.py 12 8 2 1 36%
----------------------------------------------------------------
TOTAL 242 52 63 11 76%
Coverage HTML written to dir tests/.tests/pytest-base.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-base.xml-cov-base.xml
---------------------- coverage: failed slaves -----------------------
The following slaves failed to return coverage data, ensure that pytest-cov is installed on these slaves.
gw15

=============================== warnings summary ===============================
tests/test_packages.py::PackagesTestCase::testPython
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/python_future/0.16.0+6/lib/python/future-0.16.0-py3.7.egg/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated, and in 3.8 it will stop working
from collections import Iterable
/array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))

tests/test_packages.py::PackagesTestCase::testPackages
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/python_future/0.16.0+6/lib/python/future-0.16.0-py3.7.egg/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated, and in 3.8 it will stop working
from collections import Iterable
/array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))

– Docs: http://doc.pytest.org/en/latest/warnings.html
========= 1 failed, 14 passed, 1 skipped, 4 warnings in 70.02 seconds ==========
Coverage.py warning: No data was collected. (no-data-collected)
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
The following tests failed:
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

exit -4

price · September 19, 2019, 11:40pm

The test slave crashed while running the threads test.

Is this an install from scratch?
Have you made any customisations to the stack?
What compiler version are you using?
Do you have Intel MKL and/or OpenBLAS installed? If so, what version(s)?
How much memory do you have on this machine?

Charles · September 20, 2019, 3:14am

is this an install from scratch?

Yes, I start from a empty directory, and run newinstall.sh, then run ‘eups distrib install hscPipe 7.9.1’

Have you made any customisations to the stack?

I only copy the libstdc++.so.6 (system) to minconda.

What compiler version are you using?

gcc 9.2.1

Do you have Intel MKL and/or OpenBLAS installed? If so, what version(s)?

The version of Intel MKL is 2013.0.079.
The version of openBLAS is 0.3.7.

How much memory do you have on this machine?

I submit a interactive job via qsub, with 1 core of 88 cores node and 4GM memory(not sure about this, maybe no limit =1.0T)

timj · September 20, 2019, 3:46pm

If you are building from scratch this doesn’t seem like a good thing to do.

I don’t think anyone has tried to build with gcc9.

base should be able to build with a tiny amount of memory.

The test that failed in base was using 16 cores not 1. Can you try again with setting EUPSPKG_NJOBS=1 environment variable?

Charles · September 22, 2019, 10:02am

I replace libstdc++.so.6 in miniconda with that in system, because I encountered following error message (only post part of long message). Actually it really confuses me, the error message tell that the miniconda don’t include GLIBCXX_3.4.26, which is the request of eups package…

ImportError while importing test module ‘/array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/tests/test_specFrame.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_specFrame.py:4: in <module>
import astshim as ast
python/astshim/init.py:26: in <module>
from .base import *
E ImportError: /array/users/ytchen/hscPipe/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/…/lib/libstdc++.so.6: version `GLIBCXX_3.4.26’ not found (required by /array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/lib/libastshim.so)

Should I use older version? any suggestion for version?

I tried following case, all encountered the same issue:

1 core without EUPSPKG_NJOBS=1
1 core with EUPSPKG_NJOBS=1
16 core without EUPSPKG_NJOBS=1
16 core with EUPSPKG_NJOBS=1

I still saw 32 running jobs of python3.7 (via top) when the installation of base for four cases above.

I naively tried to use the binary installation for base, afw and daf_butler individually, which all get this kind of error. They can pass the installation but I never get the whole installation done. However, I guess that I will encounter ABI incompatibilities somewhere.

Charles · September 23, 2019, 10:45pm

I start over the whole installation with EUPSPKG_NJOBS=1, then get this error message

[ 55/97 ] base 7.0-hsc …

***** error: from /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log:
plugins: flake8-1.0.4, xdist-1.20.1, forked-0.2, session2file-0.1.9, cov-2.5.1, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3
collected 16 items
run-last-failure: run all (no recorded failures)

doc/conf.py . [ 6%]
python/lsstDebug.py . [ 12%]
python/lsstcppimport.py . [ 18%]
python/lsstimport.py . [ 25%]
python/lsst/init.py . [ 31%]
python/lsst/base/init.py . [ 37%]
python/lsst/base/packages.py . [ 43%]
python/lsst/base/version.py . [ 50%]
tests/testModuleImporter2.py . [ 56%]
tests/test_packages.py … [ 87%]
tests/test_threads.py .sh: line 1: 73556 Segmentation fault (core dumped) ( TRAVIS=1 pytest -Wd --lf --lfnf=all --junit-xml=tests/.tests/pytest-base.xml --session2file=tests/.tests/pytest-base.xml.out --junit-prefix=base --cov=. --cov-branch --cov-report=term --cov-report=xml:‘tests/.tests/pytest-base.xml-cov-base.xml’ --cov-report=html:‘tests/.tests/pytest-base.xml-htmlcov’ )
Global pytest run: failed with 139
mv: cannot stat ‘tests/.tests/pytest-base.xml.out’: No such file or directory
scons: *** [tests/.tests/pytest-base.xml] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build base-7.0-hsc.eupspkg: Command:
source “/array/users/ytchen/hscPipe7.9.1/eups/2.1.5/bin/setups.sh”; export EUPS_PATH="/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30"; (/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.sh) >> /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log 2>&1 4>/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.msg
exited with code 252

Charles · September 25, 2019, 6:30am

I notice that these is a note at $HSCPATH/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/test_threads.py. The EUPSPKG_NJOBS=1 could be the cause of the error: tests/test_threads.py .sh: line 1: 73556 Segmentation fault (core dumped) But I still have not solution for solving the installation of base 7.0-hsc…

import unittest

class ThreadsTestCase(unittest.TestCase):

def testApi(self):
    from lsst.base import haveThreads
    self.assertIsInstance(haveThreads(), bool)

    from lsst.base import setNumThreads  # noqa F401
# Raises lsst::base::NoThreadsException which is untranslated
# when threading library is missing, this needs fixing!
#        self.assertIs(setNumThreads(4), None)

    from lsst.base import getNumThreads
    self.assertIsInstance(getNumThreads(), int)

    from lsst.base import disableImplicitThreading
    self.assertIsInstance(disableImplicitThreading(), bool)

price · September 27, 2019, 1:32am

I agree that you shouldn’t be replacing libstdc++.so.6.

Try using gcc version 6 (which I think is what comes with devtoolset-6).

price · September 27, 2019, 1:51am

Check what ABI version is being used in the miniconda (per StackOverflow):

readelf -sV /path/to/lib/libstdc++.so.6 | sed -n 's/.*@@GLIBCXX_//p' | sort -u -V | tail -1

Then look up a compiler in the ABI table under point 4; I think that’s the latest version you can use, but I’m not sure.

For reference, on the LSST development machine, I get GLIBCXX_3.4.25 in the shared stack, and we regularly compile with gcc 6.3.1. The compiler version listed in the ABI table is gcc 8.0.0.

Charles · September 27, 2019, 7:30am

I think you are right about this. The new/older gcc probably causes unexpected issue, at least, I had confirmed older one has issues. Unfortunately, the OS of the purposed cluster I am trying to install hscPipe 7.9.1 is Fedora 30, I haven’t found a easy way to install devtoolset-6 yet, still working on that.

I get GLIBCXX_3.4.27 for gcc 9.2.1, so it’s too new… So this should be the cause of potential following unexpected issue, I guess.

#########################

In the mean time, I had successfully installed hscPipe 7.9.1 at two test PCs:
(1) CentOS 6 (default gcc is 4.4.7) with devtoolset-6
(2) whole new installation of CentOS 7 (default gcc is 4.8.5 ) with devtoolset-6

However, I found a potential issue/bug/prerequisite in (1) case. The old-installed fftw in (1) is version 3.2.1-3.1.el6, which produce the following failure error and crash entire installation during the installation of galsim 1.5.1.lsst1-11-gfba72ad:

/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.3.1/ld: /usr/local/lib/libfftw3.a(hc2rIII_2.o): relocation R_X86_64_32 against `.rodata’ can not be used when making a shared object; recompile with -fPIC

Once I replace /usr/local/lib/libfftw3.a with the one (fftw 3.3.3-8.el7) from (2) , the whole installation is very smooth and successful. However, I am so curious about why eups installation didn’t use the one in miniconda (fftw 3.3.4.lsst2), or the ones in LD_LIBRARY_PATH or LIBRARY_PATH. The failure build log said “Found static library: /usr/local/lib/libfftw3.a” even I added the paths of new version or minconda fftw into LD_LIBRARY_PATH or LIBRARY_PATH. I thought the propose of miniconda is to provide a identical installation environment (except gcc), so why the installation didn’t use the one in miniconda?