Winter / Extra 2016 Release - Status and Discussion

Data Management
,
#1

Here is where we currently are in the release process.

Summary

Release is out: LSST Stack Version 12.0 (Winter 2016 / Extra 2016 Release)

Final build was b2091

W16-specific Precursor Steps

  1. Identify any pre-release blockers (“must-have features”)
  2. Wait for them to clear (see DM-6057 )

Release Engineering Steps

  1. Eups publish rc1 candidate (based on b2060) :white_check_mark:
  2. Git Tag 12.0-rc1 :white_check_mark:
  3. Branch v12 of newinstall.sh :white_check_mark:
  4. Github release lsst_demo v12 :white_check_mark:
  5. Wait for first round of bugs to clear :white_check_mark:
  6. Repeat last 2 steps, -rcN candidates <-- final candidate is rc?
    (no _rc2, all changes involved non-distrib repos)
  7. Confirm DM Externals are at stable tags :white_check_mark:
  8. Tag DM Auxilliary (non-lsst_distrib) repos
  9. Full OS testing (see https://ls.st/faq ) :white_check_mark:
  10. Git Tag 12.0, rebuild, eups publish:white_check_mark:

Binary release steps :white_check_mark:

  1. Produce factory binaries :white_check_mark:
  2. Test factory binaries :white_check_mark:
  3. Gather contributed binaries :white_check_mark:

Documentation Steps :white_check_mark:

  1. Update Prereqs/Install :white_check_mark:
  2. Update Known Issues :white_check_mark:
  3. Gather Release notes :white_check_mark:
  4. Gather Metrics report (unchanged for 12.0)
  5. Email announcement :white_check_mark:
12.0 release (W16/X16)
#2

Frossie,

I’m interested in trying a build of the (future) release candidate to be distributed via CernVM FS. If there is some (even preliminary) documentation on how to build and test the release I’m happy to try it and provide feedback. Alternatively, if you plan to centrally produce a binary release distributable via CernVM FS I would be happy to help with that also.

My goal is that the CernVM FS binaries of the future release be available when the new release is officially announced.

#3

@FabioHernandez v12_0 is on the server if you would like to have a go. A couple of changes, see the instructions here if you need them: https://pipelines.lsst.io/install/newinstall.html#installation-set-up

#4

Thanks @frossie. I managed to install v12.0 on Linux but not on OS X. This Linux version is now online. :grinning:

Here is my feedback. I could not follow the installation instructions exactly as stated in https://pipelines.lsst.io/install/newinstall.html#installation-set-up because the command:

curl -OL http://ls.st/8g2

creates a file named 8g2 and not newinstall.sh as I think is intended. I did instead:

curl -o newinstall.sh -L http://ls.st/8g2

In addition, executing newinstall.sh shows the warning below:

$ source newinstall.sh

LSST Software Stack Builder
=======================================================================

!!! This script differs from the official version on the distribution server.
    If this is not intentional, get the current version from here:
    http://sw.lsstcorp.org/eupspkg/newinstall.sh

<...>

The file in http://sw.lsstcorp.org/eupspkg/newinstall.sh is actually an older version of the one in http://ls.st/8g2.

It is nice to see that now you provide a demo tarball specific for each version.

Finally, I encountered an issue when building v12.0 on OS X Yosemite. The problem seems to be related to the package afw. Here is the trace:

$ eups distrib install -t v12_0 lsst_apps
   <...>
   [ 39/62 ]  pex_config 12.0                                            done. 
   [ 40/62 ]  afw 12.0 ... 

***** error: from /cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0/EupsBuildDir/DarwinX86/afw-12.0/build.log:

The errors for vector batch interpolation with variance

worst mu error  1.30094019438e-05
worst sig2 error  2.09062343838e-07

The errors for vector batch interpolation without variance

worst mu error  1.30094019438e-05

The errors for vector add interpolation

worst mu error  0.000142775502779
worst sig2 error  3.84439639196e-07
The following tests failed:
/cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0/EupsBuildDir/DarwinX86/afw-12.0/afw-12.0/tests/.tests/testGaussianProcess.py.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build afw-12.0.eupspkg: Command:
	source /cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0/eups/bin/setups.sh; export EUPS_PATH=/cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0; (/cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0/EupsBuildDir/DarwinX86/afw-12.0/build.sh) >> /cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0/EupsBuildDir/DarwinX86/afw-12.0/build.log 2>&1 4>/cvmfs/lsst.in2p3.fr/software/darwin-x86_64/lsst-v12.0/EupsBuildDir/DarwinX86/afw-12.0/build.msg 
exited with code 252

I tried also to build using export MACOSX_DEPLOYMENT_TARGET="10.9" and it fails also building afw, but the error message is different. I can provide it if that may help.

#5

Ouch, sorry! That’s what I get for trying to put in a last minute update. I’ll switch to using the full URL and drop the short links. Fixing the doc now.

We’re switching to GitHub for distributing newinstall.sh since it reduces overhead in distributing updated newinstall.sh scripts.

Right; we need to re-engineer this check.

#6

Hi @jsick et al. (& thanks for putting this together!).

Quick feedback – The instructions on how to run the comparison after the demo seem out of date. Running ./bin/compare detected-sources.txt worked for me.

#7

There is still an issue. The updated version of the documentation suggests to use curl -OL https://raw.githubusercontent.com/lsst/lsst/v12/scripts/newinstall.sh but this does not work either:

$ curl --head https://raw.githubusercontent.com/lsst/lsst/v12/scripts/newinstall.sh
HTTP/1.1 404 Not Found
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-GitHub-Request-Id: B91F1117:3854:3507FB1:5763AE62
Content-Length: 15
Accept-Ranges: bytes
Date: Fri, 17 Jun 2016 08:02:32 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-fra1242-FRA
X-Cache: HIT
X-Cache-Hits: 1
Vary: Authorization,Accept-Encoding
Access-Control-Allow-Origin: *
X-Fastly-Request-ID: 8c4160773c2d2068ffd55ebfcb55e9c2d2f6e733
Expires: Fri, 17 Jun 2016 08:07:32 GMT
Source-Age: 52
#8

I managed to build successfully on OS X Yosemite. v12.0 is now available online for both Linux and OS X.

1 Like
#9

Seems that the tagging format got changed when v12 was re-tagged. @frossie, do you know why the tag got changed from v12 to 12.0? Are we settled on this format for branch/tag names for releases?

I’ve updated https://pipelines.lsst.io/install/newinstall.html. Thanks for testing this!

#10

It looks like there is now a 12.0 branch instead of a 12.0 tag. Tags have always been numeric, never with a “v” prefix.

(Also, even when doing quick-patching, a minimal test to make sure that URLs dereference is advisable.)

#11

Yes, it was my bad for not initially checking cURL against the short link. I did check the first patch. Then the tag got switched to a release branch with a different name afterwards, and now I’ve tested the latest patch. Hopefully that clarifies what happened.

Once we’ve got a system these sorts of documentation changes will all be automated from single origins of truth.

#12

Fantastic thank you Fabio