Issue installing weeklies from source

Support
#1

I’m trying to automate the process of installing from source the weekly version of lsst_distrib at CC-IN2P3. The goal is to have a shared, read-only installation of the most recent weeklies, for which I would like to have a repeatable procedure in place.

Currently I’m facing issues when installing from source following the documentation. I’m currently trying to install the EUPS product lsst_distrib tagged w_2017_5. If I use the master version of newinstall.sh I get an error:

$ curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh
<snip>
$ PYTHON=/usr/bin/python bash newinstall.sh -b

LSST Software Stack Builder
=======================================================================

(23) Failed writing body
!!! This script differs from the official version on the distribution server.
    If this is not intentional, get the current version from here:
    http://sw.lsstcorp.org/eupspkg/newinstall.sh

Installing EUPS (v2.0.2)...  done.
Installing Miniconda2 Python Distribution ... 
  [  1/1  ]  miniconda2 4.2.12 ... 

***** error: from /scratch_ssd/fabio/w.2017.5/EupsBuildDir/Linux64/miniconda2-4.2.12/build.log:
Fetching package metadata .........
.Solving package specifications: .


PackageNotFoundError: Package not found: '' Packages missing in current linux-64 channels: 
  - fontconfig 2.12.1 0
  - glib 2.50.2 0

You can search for packages on anaconda.org with

    anaconda search -t conda glib

(and similarly for the other packages)

You may need to install the anaconda-client command line client with

    conda install anaconda-client

+ exit -5

If I try newinstall.sh tagged w.2017.5 (here) I get the same error, which seems to be linked to miniconda2 4.2.12.

If I try the version of newinstall.sh recommended for installing v12 (here) I don’t see this error. This version of newinstall.sh installs miniconda2 3.19.0.lsst4.

Now my questions:

  • What is the recommended version of newinstall.sh to use for installing a given weekly? Ideally, my installation process should determine that at runtime so that it can be automated.
  • Is there a reproducible way to get an installation of lsst_distrib (both stable and weekly versions) and its dependencies, in particular miniconda? My experience is that in the current situation, depending when you execute newinstall.sh you may get a potentially more recent version of miniconda (or EUPS for that matter) than the one used when the weekly was actually tagged. This makes very difficult to reproduce the results of the installation process over time.

Any guidance on how to properly tackle this would be appreciated.

#2

Hi @FabioHernandez,

This breakage was caused by packages being removed from the public conda channels. I fixed this a week ago for source builds (https://jira.lsstcorp.org/browse/DM-9074) with lsstsw but didn’t realize that the eupspkgs were effected until yesterday evening (https://jira.lsstcorp.org/browse/DM-9163). I am planning to get a fix out today but I am unsure if the right thing to do is to move the weekly tag or generate a new intra-week tag.

To answer your questions:

  • newinstall.sh itself changes very infrequently. There is [ideally] very little in the script that is tied to any particular stack release with the exception of the 3rd party conda packages we’re using as deps. An obvious flaw with this strategy has just been exposed in that particular versions of conda packages may vanish from the public channels without warning. The script lives in the lsst/lsst repo and is tagged for releases. Eg. https://github.com/lsst/lsst/blob/w.2017.5/scripts/newinstall.sh

  • We need to improve how we are handling 3rd party packages for a particular release. I am unsure as what the correct approach for the eupspkgs is at this point. Something we could consider would be to maintain a mirror of the conda public channels for each release. However, I’m concerned this might be problematic for an end-user trying to compose an environment that includes non-DM components.

#3

Thanks for your prompt answer, @josh. For the immediate issue of installing the current weekly (i.e. w_2017_5, I will wait for your fix. As far as I’m concerned, if that make your life easier, I can wait for next week’s tag to be released, so no need for me to have an intra-week tag.

For the more general problem of reproducibility over time of a given installation of the stack and its dependencies, I fully understand this is not an easy problem to solve. However, I do think it is an important one we as a project should consider addressing. That being said, I’m afraid I don’t have suggestions to offer on the best strategy to solve it.

Since I have your attention, would you consider using fixed-length weekly tags, such as w_2017_05 instead of w_2017_5, consistently to what was done for weekly tags in 2016? The motivation is that over time we will have several of those versions installed and it will be easier for the end user to find a given version among the available ones, given the lexicographical order used by default by the ls command and even when browsing the tags repository at https://sw.lsstcorp.org/eupspkg/tags.

1 Like
#4

New versions of the miniconda2/miniconda3 eups products have been tagged that update the list of pinned conda package versions. A new release of newinstall.sh has been made that will pickup the new product versions. These products are not used by lsstsw and are not tagged as part of the release as they are considered external dependencies. This clearly creates a dependency tracking gap.

This problem essentially slipped through the cracks due to optimization. Although each weekly release is built via eups distrib install as part of the docker container build, a base container image is used that already has newinstall.sh + deps present. I have updated our weekly release process so that the newinstall base container will always been rebuilt prior to the weekly release container. This isn’t a proper canary build (one should be implemented: https://jira.lsstcorp.org/browse/DM-9223) but it at least will prevent these problems from escaping notice.

I have also rebuilt the weekly docker container.:

$ docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2017_5
[vagrant@03840abf06cc ~]$ . /opt/lsst/software/stack/loadLSST.bash 
[vagrant@03840abf06cc ~]$ conda list | egrep 'fontconfig|glib'
fontconfig                2.12.1                        1  
glib                      2.50.2                        1
1 Like
#5

I’ve opened a jira issue on changing our weekly tag format to a fixed width: implement a fixed-length weekly tag format

1 Like