Failure of pipelines install on OS X 10.13.1

cnsetzer · December 1, 2017, 2:14pm

I was trying to install the lsst_distrib package of the pipelines and during the install, using the standard eups distrib install -t v14_0 lsst_distrib after loading the loadLSST.bash and excuting the bash newinstall.sh -3 command, I encountered a failure. (As a note I do not use the standard bash newinstall.sh -ct because I want to also install the lsst_sims pacakge as well and the binaries for that are currently broken, as noted in step 1 on https://confluence.lsstcorp.org/display/SIM/Catalogs+and+MAF). In any case, the installation will begin but by the third package, doxygen-1.8.5.lsst1 it fails. The unix terminal output for this failure is the following:

$ eups distrib install -t v14_0 lsst_distrib
   [  1/106 ]  apr 1.5.2 (already installed)                             done. 
   [  2/106 ]  cfitsio 3360.lsst4 (already installed)                    done. 
  [  3/106 ]  doxygen 1.8.5.lsst1 ... 

***** error: from /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/doxygen-1.8.5.lsst1/build.log:
46:
In file included from /Library/Developer/CommandLineTools/usr/include/c++/v1/stdlib.h:88:
/Library/Developer/CommandLineTools/usr/include/c++/v1/__config:163:11: fatal error: 'sys/endian.h' file not found

          ^~~~~~~~~~~~~~
# include <sys/endian.h>
          ^~~~~~~~~~~~~~
# include <sys/endian.h>
          ^~~~~~~~~~~~~~
1 warning and 1 error generated.
make[2]: *** [../objects/qbuffer.o] Error 1
make[2]: *** Waiting for unfinished jobs....
1 warning and 1 error generated.
make[2]: *** [../objects/qdatastream.o] Error 1
1 warning and 1 error generated.
make[2]: *** [../objects/scstring.o] Error 1
1 warning generated.
make[1]: *** [all] Error 2
make: *** [all] Error 2
+ exit -4
eups distrib: Failed to build doxygen-1.8.5.lsst1.eupspkg: Command:
	source "/Users/cnsetzer/lsst_stack/eups/2.1.4/bin/setups.sh"; export EUPS_PATH="/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67"; (/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/doxygen-1.8.5.lsst1/build.sh) >> /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/doxygen-1.8.5.lsst1/build.log 2>&1 4>/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/doxygen-1.8.5.lsst1/build.msg 
exited with code 252

If it is helpful the build.log contains the following:

+ cd /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/doxygen-1.8.5.lsst1
+ _EUPS_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
+ . /Users/cnsetzer/lsst_stack/eups/2.1.4/bin/setups.sh
++ export EUPS_SHELL=sh
++ EUPS_SHELL=sh
++ export EUPS_DIR=/Users/cnsetzer/lsst_stack/eups/2.1.4
++ EUPS_DIR=/Users/cnsetzer/lsst_stack/eups/2.1.4
+++ echo /Users/cnsetzer/lsst_stack/eups/2.1.4
+++ sed -e 's/ /-+-/g'
++ eupslocalpath=/Users/cnsetzer/lsst_stack/eups/2.1.4
+++ /Users/cnsetzer/lsst_stack/python/miniconda3-4.2.12/bin/python -E -c '
from __future__ import print_function
import sys
pp = []
for d in sys.argv[1].split(":"):
    if d and d not in pp:
        pp += [d]
if not sys.argv[2] in pp:
    pp = [sys.argv[2]] + pp
print(":".join(pp))' /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67 /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
++ export EUPS_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
++ EUPS_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
++ _eups_path=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
+++ /Users/cnsetzer/lsst_stack/eups/2.1.4/bin/eups_setup DYLD_LIBRARY_PATH= eups -r /Users/cnsetzer/lsst_stack/eups/2.1.4
++ eval export 'SETUP_EUPS='\''eups' LOCAL:/Users/cnsetzer/lsst_stack/eups/2.1.4 -f DarwinX86 -Z '(none)'\'';' 'unsetup()' '{' eval '`"/Users/cnsetzer/lsst_stack/eups/2.1.4/bin/eups_setup"' 'DYLD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}"' --unsetup '"$@"`' ';' '};' 'setup()' '{' eval '`"/Users/cnsetzer/lsst_stack/eups/2.1.4/bin/eups_setup"' 'DYLD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}"' '"$@"`' ';' '}'
+++ export 'SETUP_EUPS=eups LOCAL:/Users/cnsetzer/lsst_stack/eups/2.1.4 -f DarwinX86 -Z (none)'
+++ SETUP_EUPS='eups LOCAL:/Users/cnsetzer/lsst_stack/eups/2.1.4 -f DarwinX86 -Z (none)'
++ export 'SETUP_EUPS=eups LOCAL:/Users/cnsetzer/lsst_stack/eups/2.1.4 -f (none) -Z (none)'
++ SETUP_EUPS='eups LOCAL:/Users/cnsetzer/lsst_stack/eups/2.1.4 -f (none) -Z (none)'
++ export EUPS_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
++ EUPS_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
++ unset eupslocalpath _eups_path
++ '[' X '!=' X -a -f /Users/cnsetzer/lsst_stack/eups/2.1.4/etc/bash_completion.d/eups ']'
+ '[' X/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67 '!=' X ']'
+ EUPS_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67
+ tar xzvf /var/folders/k9/v7x12qwx0yv_1xv6vtbk9x280000gn/T/eups/distrib/doxygen_7i09fmt_
x doxygen-1.8.5.lsst1/
x doxygen-1.8.5.lsst1/patches/
x doxygen-1.8.5.lsst1/patches/0001-Correct-version-detectection-for-flex-2.6.patch
x doxygen-1.8.5.lsst1/ups/
x doxygen-1.8.5.lsst1/ups/doxygen.table
x doxygen-1.8.5.lsst1/ups/pkginfo
x doxygen-1.8.5.lsst1/upstream/
x doxygen-1.8.5.lsst1/upstream/doxygen-1.8.5.src.tar.gz
++ find . -maxdepth 1 -type d '!' -name '.*'
++ head -n 1
+ PKGDIR=./doxygen-1.8.5.lsst1
+ test '!' -z ./doxygen-1.8.5.lsst1
+ cd ./doxygen-1.8.5.lsst1
+ [[ ! -e ./ups/eupspkg ]]
+ mkdir -p ./ups
+ ln -s /Users/cnsetzer/lsst_stack/eups/2.1.4/lib/eupspkg.sh ups/eupspkg
+ setup --just --type=build apr 1.5.2
++ /Users/cnsetzer/lsst_stack/eups/2.1.4/bin/eups_setup DYLD_LIBRARY_PATH= --just --type=build apr 1.5.2
+ eval export 'SETUP_APR='\''apr' 1.5.2 -f DarwinX86 -Z '/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67'\'';' export 'LSST_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib;' export 'DYLD_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib;' export 'APR_DIR=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2;' export 'PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/bin:/Users/cnsetzer/lsst_stack/eups/2.1.4/bin:/Users/cnsetzer/lsst_stack/python/miniconda3-4.2.12/bin:/Users/cnsetzer/miniconda3/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin;' export LD_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib
++ export 'SETUP_APR=apr 1.5.2 -f DarwinX86 -Z /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67'
++ SETUP_APR='apr 1.5.2 -f DarwinX86 -Z /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67'
++ export LSST_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib
++ LSST_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib
++ export DYLD_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib
++ DYLD_LIBRARY_PATH=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2/lib
++ export APR_DIR=/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/DarwinX86/apr/1.5.2

Please let me know if I can provide more information. Thanks for taking the time to read my issue.

timj · December 1, 2017, 5:47pm

It seems that that old doxygen version 1.8.5 does not build on High Sierra 10.13.1 with the newest command line tools. I’m not entirely sure what the problem is, although it looks like a typo in the Apple include files. You will have to build with a newer version of the stack which comes with 1.8.13. Try tag w_2017_47.

cnsetzer · December 5, 2017, 9:21am

Sorry the delay in reply. This has definitely improved the situation, but now I’m running into an issue installing galsim. It appears there is also an issue with installing this on High Sierra. Would it be worthwhile to get the most recent version v1.5 from their github? The message that is returned is the following:

***** error: from /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/galsim-1.4.3.lsst1-2-g9310893/build.log:
Nor did any of the known variations on this work.
If these are not the correct library name or location, you can tell scons
the correct names to use with the flags EXTRA_LIB_PATH and/or EXTRA_LIBS.
Mac version is 10.13.1

Starting with El Capitan (OSX 10.11), Apple instituted a new policy called
"System Integrity Protection" (SIP) where they strip "dangerous" environment
variables from system calls (including SCons).  So if your system is using
DYLD_LIBRARY_PATH for run-time library resolution, then SCons cannot see it
so that may be why this is failing.  cf. Issues #721 and #725.
You should try executing:

    scons DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH

Please fix the above error(s) and rerun scons.
Note: you may want to look through the file INSTALL.md for advice.
Also, if you are having trouble, please check the INSTALL FAQ at 
   https://github.com/GalSim-developers/GalSim/wiki/Installation%20FAQ

+ exit -4
eups distrib: Failed to build galsim-1.4.3.lsst1-2-g9310893.eupspkg: Command:
	source "/Users/cnsetzer/lsst_stack/eups/2.1.4/bin/setups.sh"; export EUPS_PATH="/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67"; (/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/galsim-1.4.3.lsst1-2-g9310893/build.sh) >> /Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/galsim-1.4.3.lsst1-2-g9310893/build.log 2>&1 4>/Users/cnsetzer/lsst_stack/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/DarwinX86/galsim-1.4.3.lsst1-2-g9310893/build.msg 
exited with code 252

I’ve tried running the suggested command with scons but I receive the following:

cnsetzer lsst_stack $ scons DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH

scons: *** No SConstruct file found.
File "/Users/cnsetzer/lsst_stack/python/miniconda3-4.2.12/lib/python3.6/site-packages/SCons/Script/Main.py", line 924, in _main

I still have a few things I can try, but I will get to them later this week. Thanks for the help thus far.

jmac · December 6, 2017, 12:02am

Hello! Just from my own issues building DMStack & obs_decam on OSX High Sierra, may I suggest the following? Don’t know if they will help or if you’re running into these issues anymore, but you never know…
(1) Download the “master” version of loadLSST.bash, not the v14_0 one. Just copy out /master/ for /v14_0/ in that long path.
(2) Did you try the standard -ct build? Does that work? Also this may be nonsense, but one week w_2017_47 “worked,” with obs_decam, the next week it didn’t. Using w_2017_48 made that problem go away, as far as I know.
(3) Again, possibly totally irrelevant but you may want to use a .bash_profile that doesn’t put your own version of anaconda at the top of the path. I thought this caused an issue at some point but I could be wrong.
(4) Do you have the latest Xcode tools installed?

Sorry if these are wrong or a waste of time, and good luck!

timj · December 6, 2017, 5:34pm

GalSim is broken on anaconda python 3.6.3. @rmjarvis just fixed it today and indicates that this will go into 1.5.1. I have fixed the LSST GalSim with that patch so the next weekly will be buildable (comes out on the weekend).

I have heard reports that forcing a downgrade of the python to 3.6.1 or something might fix the problem for you until the real fix comes out next week.

@danielsf is looking into the upgrade of our GalSim to 1.5.1 but there were some changes to C++ include files between 1.4 and 1.5 and so some fixes are needed in the meas_extensions_shapeHSM package.

cnsetzer · December 6, 2017, 7:08pm

@timj

Great to hear! Thanks for the help. I will wait until then to try the install and let you know if that closes this issue.

cnsetzer · December 12, 2017, 1:14pm

@timj Hi Tim, just checking in with my progress after attempting the last weekly building. Beginning with a clean lsst_stack directory and using the master branch for the newinstall.sh script, I have been able to successfully install galsim. However, if I am trying to install the full lsst_distrib via eups distrib install -t w_2017_49 lsst_distrib command it fails on the install of package [65/114] log 14.0-2-ga5af9b6+5 with the following terminal output of the failure:

  [ 65/114 ]  log 14.0-2-ga5af9b6+5 ... 

***** error: from /Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/build.log:
tests/test_log.py:33: in <module>
    import lsst.log as log
python/lsst/log/__init__.py:25: in <module>
    from .log import *
python/lsst/log/log/__init__.py:25: in <module>
    from .log import *
E   ImportError: dlopen(/Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/log-14.0-2-ga5af9b6+5/python/lsst/log/log/log.so, 2): Symbol not found: _libiconv
E     Referenced from: /Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/apr_util/1.5.4/lib/libaprutil-1.0.dylib
E     Expected in: flat namespace
E    in /Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/apr_util/1.5.4/lib/libaprutil-1.0.dylib
====================== 1 passed, 1 error in 2.33 seconds =======================
Global pytest run: failed
Failed test output:
Global pytest output is in /Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/log-14.0-2-ga5af9b6+5/tests/.tests/pytest-log.xml.failed
The following tests failed:
/Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/log-14.0-2-ga5af9b6+5/tests/.tests/pytest-log.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build log-14.0-2-ga5af9b6+5.eupspkg: Command:
	source "/Users/cnsetzer/lsst_stack/eups/2.1.4/bin/setups.sh"; export EUPS_PATH="/Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6"; (/Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/build.sh) >> /Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/build.log 2>&1 4>/Users/cnsetzer/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/DarwinX86/log-14.0-2-ga5af9b6+5/build.msg 
exited with code 252

I have looked in the build log files and I have not found any additional information beyond the terminal output.

I realize now, I could move this discussion over to the LSSTC Slack #desc-software-help channel if that would be more appropriate. Thank you again for taking the time to help with this.

timj · December 12, 2017, 2:15pm

The libiconv error suggests that you have the miniconda libiconv package installed. This should not be the case if you are using our miniconda and have not installed anything extra. To proceed you need to uninstall libiconv from conda. You can put it back again after the build. We have DM-12105 discussing this.

I’m not part of DESC. If you are on Slack there is the #dm-build-problems channel.

cnsetzer · December 12, 2017, 7:36pm

@timj Yup that took care of it! Thanks for the help, everything is working now.