Using the stack on lsst-dev

:bangbang: Warning

The material below has largely been rendered obsolete by e.g. DM-5021. It is preserved here for posterity, but please refer to DM-5435 and this topic for details on a replacement, and, in general, refer to the Developer Guide for the most current information.


Many of us use lsst-dev as a convenient machine for developing, or, at least, for occasional testing. However, I, for one, can never remember what the best way to get hold of a recent version of the stack on that machine is; I’m guessing others, and particularly newcomers, have similar problems. Here’s my attempt to record the situation for posterity: others who know better please do correct me.

The Buildbot stack

Everything that Buildbot builds gets installed into ~lsstsw/stack. You can access this stack by running:

$ . ~lsstsw/bin/setup.sh

You can choose a buildbot tag (look at the log – for example this one – and find a line like #BUILD ID: bNNNN) then run:

$ setup -t bNNNN lsst_apps # or whatever product you prefer

However, the current version of eups doesn’t scale well to extremely large numbers of tags, so this gets really slow. A better option might be:

The shared, copied stack

Selected tags generated by Buildbot are copied to a shared stack in /lsst/stack. Access it like so:

$ source /lsst/stack/loadLSST.sh

See this mail to dm-devel for further details, including how to add new tags via the copyTag.sh script.

However, as this stack accumulates more tags, it also slows down. Unfortunately, there’s no easy way to prune the old tags. Instead, you could consider:

A personal, copied stack

The copyTag.sh script described above can be used to copy tags from the Buildbot stack into your own personal stack. An easy way to bootstrap a stack is to run newinstall.sh – see these instructions. For example, the following should work:

$ mkdir /raid/${username}/stack
$ cd /raid/${username}/stack
$ curl -O https://sw.lsstcorp.org/eupspkg/newinstall.sh
$ bash newinstall.sh
(Select options to:
    1. Continue without installing git;
    2. Install Anaconda)
$ . loadLSST.bash
$ /lsst/stack/copyTag.sh bNNNN
$ setup -t bNNNN lsst_apps # or whatever product you prefer

This way, you have complete control of the tags available in your personal stack. When it gets too overcrowded and slow, simply delete it and create a new one.

2 Likes

The linked email doesn’t actually describe how to use copyTag.sh (is it just “copyTag.sh <tag-name>”?), and it’s a pain to access the list archives. Could @ktl perhaps repeat that information here?

The third part of the post above does give an example:

$ /lsst/stack/copyTag.sh bNNNN

But yes, that could have been clearer.

The script has a (minimal) “usage” line that says:

Usage: copyTag.sh [-n] [-e] <eups-tag>

-n shows what will be done
-e allows redeclaration of existing tag

It copies from the buildbot stack into whatever is the first stack in your EUPS_PATH.

Something is wrong here (not sure if it’s new); /lsst/stack.sh does not exist.

Should be /lsst/stack/loadLSST.sh. Will fix.

lsst-dev refers to the machine installed at NCSA
lsst-dev.ncsa.illinois.edu

I add to help searching find this page.

Be warned that DM-5021 has currently rendered the /lsst/stack stack unusable.