Summer 2015 Release - status and discussion

Data Management
,
#1

Here is where we currently are in the release process.

Summary

We have shipped it. Mopping up.

S15-specific Precursor Steps

  1. Tag precursor weekly (w.2015.36 )
  2. Verify OSX builds pass by users unable to build w.2015.35

Release Eng Steps

  1. Eups publish rc1 candidate
  2. Git Tag 11.0-rc1
  3. Wait for first round of blockers to clear (see DM-2680 )
  4. Repeat last 2 steps, -rcN candidates <-- final candidate is _rc3
  5. Confirm DM Externals are at stable tags
  6. Tag DM Auxilliary (non-lsst_distrib) repos
  7. Full OS testing (see https://ls.st/faq )
  8. Git Tag 11.0, rebuild, eups publish, copyTag.sh to update lsst-dev

Documentation Steps

  1. Update Prereqs/Install
  2. Update Known Issues
  3. Gather Release notes
  4. Gather Metrics report
  5. Gather contributed binaries
  6. Email announcement
#2

The w_2015_36 tag builds for me. :thumbsup:

1 Like
#3

The SCONSFLAGS and LSST_CFG_PATH problems in psfex are still there in w.2015.36; the latter is a very unusual case and may be omitted, but the former needs fixing. I think Nate Lust (who doesn’t seem to be on discourse) is looking at it today, but I don’t know the issue number.

#4

I can’t see a ticket on the SCONSFLAGS problem so I can’t comment on whether it’s a blocker for 11.0. I agree that DM-3655 is not a blocker.

#5

@natelust Can you file that ticket?

#6

@frossie Can you /lsst/stack/copyTag.sh release candidates to the /lsst/stack shared stack for ease of use on lsat-dev?

#7

The SCONSFLAG bug in psfex has been filed under DM-3749. I have a fix for that now, and am awaiting a reviewer. @RHL

#8

Modulo DM-3754
which has a fix in Pull Request to ‘eups’

w_2015_36 builds for me on Mac OS X 10.10.5
(Mac Pro – Late 2013)
clang-602.0.53

using Homebrew as the Python package manager.

Builds both ‘lsst_apps’ but also all the way through to ‘lsst_distrib’

#9

@mwv Awesome, will be getting some homebrew-based installation details from you so we can document that.

1 Like
#10

Post mortem from my POV.

From the technical point of view this was the best release in my time here.

  1. The 2-OS CI runs were fantastically useful through the cycle (more as we get capacity) and really cut down on the release debug cycle. As was the faster CI. And the vagrant sandbox is still wonderful, though it needs some TLC.
  2. I am much happier with my release prep process once the first candidate is down. Even putting in the 11th hour fixes was simple. I know how to improve it for next time, too.
  3. The CI hammering of the user-facing (newinstall.sh) process saved us from shipping what seemed to be a successful but in fact very flaky build.
  4. We ran out of time and effort to prep and CI binaries this time, but Fabio did a great job stepping in there. We’re working with him on an ongoing process in parallel with working with our Docker solution that we can scale down to weeklies.

Things that are still a giant pain:

  1. Prepping the documentation in Confluence. Never again - the next official release will be with git-backed documentation. This was by far the most time consuming step (and bugger-all fun either, for all concerned).
  2. Although it was UW and Princeton that bore the brunt, the new characterisation requirement is an example of the thing that makes official releases time consuming. We’ll work to automate these over the next couple of cycles.
  3. The El Capitan issue. Boy we need OSX CI and delcaring it a supported platform - the vast majority of current users are on it, it’s official in every way but name.
  4. The lack of test framework is becoming a tall pole. It’s hard to isolate failing tests in CI, and there are a lot of ad-hoc behaviours (like each test having its own eccentric way of figuring out whether it is running at the DAC). I am of course hopeful that we will do the RFC-69 work to address it this cycle.

I am severely tempted to make a mid-cycle “official” release as an experiment, partly to drive the improvements outlined above (== hold my feet to the fire) and partly to see if that generates a saving when it comes to the cycle release.

DM developers: you are awesome and it is a privilege (not to mention fun) working the releases with you.