Developing Software with the DM Stack

[rowen (Russell Owen)]

This page describes a method of installing and maintaining a DM software stack that allows you to easily keep up to date with the current master of packages and to push your own changes. It was updated 2015-11-04 to support git-lfs packages such as afwdata.

Initial Installation

• Set up a git credential manager, e.g. as per afwdata’s README.md.
• (alternately: edit the file lsstsw/etc/repos.yaml to change all instances of https://github.com/lsst, except afwdata,to git@github.com:lsst. This works fine if you don’t plan to push to afwdata, and doesn’t require using the credential manager.)
• Clone the lsstsw package into the location of your choice: git clone git@github.com:lsst/lsstsw.git
• Install lsstsw and rebuild lsst_apps using these instructions. Note that this pulls over all of afwdata, which is a lot of data. If you already have a cloned copy you may be able to avoid this by making a symlink to it from lsstsw/build/afwdata.
• Eups tag the current build as “current” using command: eups tags --clone bNNN current
  where bNNN is the eups tag that lsstsw used for the packages installed by the rebuild command.

Development Cycle

To work on one or more packages, start by making a fresh build (if you have not gotten one recently) using rebuild:

• rebuild lsst_apps
  If this complains that it cannot clone some packages then you will have to update repos.yaml; the simplest way to do this to specify the -u flag in the rebuild command.
• eups tags --clone bNNN current as described above

Next declare the git clones of the packages you will be working on to eups. The git clones are in $LSSTSW/build and I suggest you declare them with version git and a eups tag of your username (since that tag is permitted by eups without altering the eups configuration). For example, user rowen could issue the command eups declare -r $LSSTSW/build/afw afw git -t rowen.

Now edit your code and rebuild manually, using scons as needed. Having eups-tagged your packages, you can set them up using: setup -r . -t your_username.

Warning: the rebuild command will erase any changes you have not pushed. Do not run the rebuild command while developing and testing your code unless you really need it. Before you run rebuild carefully verify that you have pushed (not just committed) all your changes.

When you are done with your work:

• Clean your commit history using git rebase -i $HEAD~N where N is the number of commits.
• Get your changes reviewed.
• Clean up your code as required by the review and clean up the commit history again.
• Merge your changes to master and push all changes.
• Remove your username tag from the packages. For example to remove your username tag from afw: eups undeclare afw git -t your_username.
• Build a fresh copy of the stack using rebuild lsst_apps followed by eups tags --clone bNNN current.

Caveats

• Be careful not to overwrite your work with a “rebuild” command. Always commit and push before rebuilding, or do your work on clones that are not in lsstsw/build.
• Be sure to clone tags if you want your new build to be current, because otherwise the default “setup” command won’t act as you expect.
• Be careful when trying to run packages in the build directory (rather than installed versions) because “rebuild” only builds what it has to, so some of those directories may not have been built.
• You should usually specify “-u” with the rebuild command to update your repos.yaml file. Otherwise you risk the build failing with a complaint about missing packages.

[jsick (Jonathan Sick)]

How does this ‘lsstsw’ method of curating a development stack differ in practice from using lsst-build (described in https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool)? Should a stack developer always use the lsstsw workflow your describe, or are there cases where lsst-build should be used directly?

[timj (Tim Jenness)]

differ in practice from using lsst-build

Well, I use lsst-build directly but maybe I’m just special. I think it was because I couldn’t find the right documentation when I was trying to find out what to do so I simply looked around the lsstsw checkout and the documentation in the code that I could most easily follow was in the lsst-build command…

I should add that it took me a few times before I realised that this scheme trashes your entire working tree and you have to push to the remote repo. I was somewhat confused the first few times when I had done work on a ticket branch and asked lsstsw to build it, only to find that my branch had disappeared (luckily I knew the SHA1 so I could find it again).

[RHL (Robert Lupton)]

I use a much less magic approach (magic is of course in the eye of the beholder). When I want to get some work done, I build a recent weekly (I’ll use a binary when I can) :

eups distrib install lsst_apps -t w_2015_36

Then if I want to work on e.g. afw,

cd LSST/afw

(where I have a git clone)

git pull
setup -r . -j
scons -Q opt=3 -j 6

and I’m good to go. If I need to work on more than a product or two, I tag them:

eups list -t rhl

(and then eups undeclare -t rhl NAME any that I don’t still need)

eups declare -r . -t rhl

followed by

setup -t rhl -t w_lsst_36 lsst_apps

I should be able to make w_lsst_36 current with a eups tag --clone=w_lsst_36 current command, but there seem to be problems in the table file that the versions are incorrect – it’s on @mjuric’s list. If I could, then either

setup -t rhl -T w_lsst_36 lsst_apps

or a simple

setup -t rhl lsst_apps

would work.

[natepease]

Now edit your code and rebuild manually, using scons as needed. Having eups-tagged your packages, you can set them up using: eups setup -r . -t your_username.

Is this line correct? or is this the correct command? setup -r . -t your_username
(leaves out the eups before setup). - when I leave off eups the command works, otherwise it says “eups setup: Unrecognized command: setup”

[ktl (K-T Lim)]

Is this line correct?

Just testing you . Fixed, and thanks.

[natepease]

thanks! I’m glad to know I’m at least somewhat sane

[rowen (Russell Owen)]

The work flow @RHL advocates also works, but I feel is less suitable for active daily development because lsstsw (or lsstbuild) provides direct support for the following:

• Pull in a needed fix, either from master or one or more ticket branches
• Rebuild a set of interrelated packages after making a fix to one of them

Using eups distrib install is something developers should know how to use, but I don’t recommend it for daily development because it is too limited and offers too great a chance of incompatibility between the master you are working on and the last weekly release. See also What development workflow should be advocated? [w/ Poll] for an extensive discussion.

[RHL (Robert Lupton)]

Does your proposal work with a shared stack?

[rowen (Russell Owen)]

I doubt using lsstsw to build shared stacks for development is a good idea, though it can be done. The biggest issue is that each developer would have to keep packages being worked on out of the main stack area, so that if one developer runs rebuild it does not overwrite the work of other developers. That loses one of the main advantages to lsstsw: being able to build all related packages on one or more tickets. Still, with careful coordination it can work.

For a shared stack that is intended for users instead of developers lsstsw seems even less appropriate, since users should use official releases of the software, if possible.

[RHL (Robert Lupton)]

That was what I thought. I believe that we need a workflow that works with a mostly shared stack. This doesn’t mean that this way of working will be good for all developers/situations, but if it isn’t I think we need to document and support two options.

[rowen (Russell Owen)]

It’s a bit tricky to do it all. Developers often require cutting edge software, and users are usually better served by official (or semi-official) releases. It is hard for me to imagine that developers and users would usefully share a stack.

I can imagine developers being able to share lsstsw stacks if we tweak lsstsw to install in a shared location but offer each developer a personal “build” directory. I have no idea if that would provide enough benefit to be worth doing. It would save some space, but it would also get very cluttered and that clutter would be hard to clean out. And a cluttered stack is hard to manage; eups slows down and even if that were fixed, the output of list commands becomes large enough to be a problem.

[RHL (Robert Lupton)]

It is hard for me to imagine that developers and users would usefully share a stack.

Why? A specific version (or tagged version) is well defined in a shared stack whether or not some developer has other versions installed.

[parejkoj (John Parejko)]

Russell just requested that I post this here for his memory.

When I need to clean a bunch of old builds from my lsstsw, I do the following:

cd ~/lsst/lsstsw/stack/
rm -rf DarwinX86/*
rm -rf ups_db/*

Then I rebuild lsst_apps, and replace the afwdata/BLAH directory with a symlink to my ~/lsst/afwdata checkout (which is kept up to date because it’s symlinked to ~/lsst/lsstsw/stack/afwdata).

This lets you clean up an overly-large eups list and free a bunch of disk space, without having to reinstall everything or damaging your miniconda install/environments.

[rowen (Russell Owen)]

Then I rebuild lsst_apps, and replace the afwdata/BLAH directory with a symlink to my ~/lsst/afwdata checkout (which is kept up to date because it’s symlinked to ~/lsst/lsstsw/stack/afwdata).

One minor addendum: also do the same for testdata_cfht. None of the other testdata packages is large enough to be a concern at this time.