What development workflow should be advocated? [w/ Poll]

jsick · October 14, 2015, 2:03am

In writing docs for our development workflow, I realize that there are at least three workflows for developing the stack

eups distrib install a recent tagged release and then clone packages as needed and do the eups declare and setup. This is described on a community posting and on confluence.
Use lsstsw to clone and build a full stack, and make branches off those clones and eups declare and setup as necessary. This was described in a community post and linked through to a page on confluence
Use lsst-build directly. I know of one person who actually does this, but I don’t think we should encourage it

Of course, these methods do share some commonality, but I’d ideally like settle on a single, clear, nearly fool-proof development workflow document that I can give to new hires and the open source astronomy community. Of course, deeper docs can always capture alternative workflows and tweaks.

I’ll confess that I’ve already started on the lsstsw method, since I like the approachability of having all the code cloned, and it doesn’t rely on a weekly tagged build (which seems unusual coming from a ‘pull master’ GitHub Flow world). At the same time, I realize that lsstsw was really made for the CI environment, and doesn’t have much in the way of human ergonomics.

I also don’t have the day-to-day stack development experience that the UW and Princeton crews have.

So I’ll open it to you: what method should we advocate in a developer workflow guide? Please vote in the poll. If you have a nuanced answer, or have a fourth way, submit a new post outlining it.

Thanks!

#1 eups distrib workflow
#2 lsstsw workflow
#3 lsst-build workflow
other

0 voters

Show results

timj · October 14, 2015, 3:41am

Do we think those have the same answers? lsstsw is for somebody who needs to be living life on master, developing multiple packages together and ensuring that the work is not broken by others and giving you confidence that what you have just pushed to your ticket branch is going to be a state reasonably close to what the CI is going to use to test it. eups distrib is always trailing edge a bit and you have to wait for new versions of the stack to be published before you can try out a feature added in the last week (if we have weeklies made available).

lsstsw has the added bonus of giving you easy access to all the git repos.

jsick · October 14, 2015, 3:51am

In so much as they’re both groups coming in without pre-existing workflows and are wanting the best advice we can give.

I’ve thought that perhaps we need a ‘lsstsw for humans’ that wraps a lot of the various eups-related chores, scons building and test running into a command line app. But I also realize it’s probably a bad idea to introduce yet another layer on our toolchain without a clear endgame.

Improving our build toolchain is something to address in another post; for now we need to doc what we’ve got.

jbosch · October 14, 2015, 4:10am

+1

I think this is a role lsst-build was supposed to fill back when @mjuric started building it, but like so much of our developer infrastructure it never really got developed past “barely good enough”, and since it was developed further as the backend for buildbot, lsstsw actually became easier for developers to use than lsst-build itself.

swinbank · October 14, 2015, 12:16pm

I voted for eups distrib (which puts me in a minority of one, for now at least). However, let me attempt to be more nuanced:

The eups distrib workflow description linked to above is complex, because it discusses tagging, declaring, and so on. Those might be convenient, but, in fact, are not necessary. At it’s heart, this method boils down to three commands: eups distrib install, setup, scons. My contention is that all developers have to be conversant with those before they can effectively work with the stack, regardless of whether they also choose to adopt higher level tooling: eups distrib because it is our primary documented method of installing the stack, while setup and scons are required whatever workflow is adopted (@rowen’s post, linked above, also describes their use).

Of course, there are downsides to a pure eups distrib workflow: I’m quite happy with – indeed, would encourage – the idea that we should describe the fundamentals and also describe higher-level tools that can make the practising developer’s life easier, but I don’t think that means we can avoid covering the basics.

natepease · October 14, 2015, 3:45pm

You do have to consider your target audience though; are you including qserv in this conversation? We discovered recently that the lsstsw tools don’t work for the qserv (or xrootd only?) workflow, at least not without some fancy footwork. @fritzm can provide better details on that I think.

timj · October 14, 2015, 4:11pm

I would like to hear about that because Jenkins uses lsstsw and Qserv has no problems there.

fritzm · October 14, 2015, 4:44pm

I think Nate is referring to the following:

I am not familiar with the lsstsw tool, and so my working model when working with xrootd and qserv has been distrib install + clone + eupspkg + tag + setup.

Nate came back from bootcamp with a cool incantation for directly declaring a git clone to the local eups. That seemed appealing because I could potentially skip the eupspkg parts. But of course, for xrootd and qserv the git tree is quite different from the binary install tree that eups in the end expects. So you can’t do that for those guys – you have to use eupspkg {install,decl} as well.

I’m looking forward to peeping the videos from boot camp to learn more about the lsstsw tools and see how/if they could improve the workflow I’m currently using.

natepease · October 14, 2015, 5:18pm

Thanks for clarifying, Fritz.

WRT the videos, I think I picked up the lsstsw workflow in the tutorial sessions (not during the recorded presentations); I’m not sure there will be a lot in there for you re. this. However, I’m sure you could talk to any number of people including folks at UW, specifically @rowen for more info.

rowen · October 14, 2015, 5:39pm

I completely agree with @timj.

lsstsw has the added benefit of being able to build any ticket branch or combination of ticket branches. This has important advantages:

If one is working on a ticket that spans multiple packages, one can update all dependent packages to the current master (e.g. to pull in a needed fix) and continue one’s work.
• It is easy to test ticket branches, which can be quite important on a system not yet supported by Jenkins.

jbosch · October 15, 2015, 1:17am

I completely agree that developers need to be able to use those tools, and even though this isn’t my primary workflow, I do still use it frequently, especially on the HSC side where I’m more likely to be making a small bugfix to a released version.

The reason I’d advocate lsstsw (or ideally something like it but better) is that I think it’s bad for developers to rely on released versions as a part of their default workflow, because that inevitably leads to version incompatibilities, and that leads to invalid bug reports (e.g. segfaults from ABI incompatibility) and unnecessary blocking (developer A can’t get work done until developer B has time to make a release). When you don’t build the whole stack from source yourself regularly (even if you know how), it becomes much harder and slower to do it when you need it, and you need it pretty often.

RHL · October 16, 2015, 1:10am

For the “work on an issue” sort of application, what you call the “eups distrib workflow” is very convenient, especially in an environment where you share a stack – and that’s going to be ever commoner as we work with larger data volumes that you don’t have on your laptop.

Many (and later most?) users will not need to check out a complete distribution of their own on a shared machine, but that’s what the lsstsw approach assumes.

The latest weekly is already installed for everyone to use (someone runs a cronjob to do the “eups distrib install”?). So a user who wants to work on a package just has to follow the patten which Jonathan references [and which I don’t know how to embed here]

setup lsst_apps -T weekly
git clone git@github.com:lsst/meas_extensions_photometryPetrosian
setup -r . -j
scons

and they are away. The package in question could be level 3 that needs a recent release, or it could be something way down in the stack – although that is more likely to run into API/ABI problems. We should be checking for ABI changes, but that’s another matter (lost somewhere on trac).

ktl · October 16, 2015, 1:17am

It’s not clear that “larger data volumes” necessarily implies shared machines or shared stacks. The NCSA Nebula cluster is going to be able to give each developer a separate machine with access to shared large data. (We can make a shared stack available as well, of course, but it is not mandatory.)

jsick · October 16, 2015, 1:22am

Perhaps be worth having a second workflow document describing how to develop against a shared stack (given it’s current importance pre-Open Stack cluster).

RHL · October 17, 2015, 2:20am

I agree that it need not, but it opens up the possibility of shared stacks and I think we should take it. End users absolutely need it (“Look at this DECam rerun” shouldn’t start with, “but first install the LSST stack”), and I believe that it’s a very valuable mode for many/most developers. There will always be exceptions, but given that we have to support it for end users, let’s take advantage of the possibility.

I think that large data volumes do imply that working on laptops will become less common.

parejkoj · November 4, 2015, 12:33am

I’m not voting yet (too new to have a strong opinion), but I’m about to re-install via the lsstsw method which I have a slight preference for right now.

That said, I don’t understand how two of the points made above are relevant here:

Users and developers are almost certainly going to have different methods. Most users probably shouldn’t even touch eups, and just conda or pip install lsst (or just use the stack that’s already installed on their shared system). On the other hand, developers having a full checkout of the entire stack that they develop inside of seems like a clear workflow to me.
I don’t see how larger data volumes are relevant either: the software stack is separate from the data, and I thought the whole point of the butler was to make the code completely data-location agnostic. In the worst case, afw_data (for example) would live on a network drive, etc. and be the only package that gets setup differently. What am I missing?