SQuaSH is the best

Data Management
#1

I want to highlight a great success of the SQuaSH tool recently. Following is a screenshot taken today from http://squash.lsst.codes.

Pasted image2030×952 84.1 KB

This figure tells a fantastic story about how metric monitoring will change how we do our work in the future.

The first thing you may notice is the outage starting on 01/04. That’s the moment I broke the CI by merging a change to the reference catalog management. The outage ends on 01/25 when @swinbank fixed my breakage. But wait, there is now a regression in this metric. @cmorrison found and fixed the regression on 02/02.

Massive thanks to the SQuqRE team, but especially to @afausti. I don’t think we would have found the outage or the regression (let alone fixed them) without a tool like this.

8 Likes
#2

Thanks @ksk. And we’re just getting started :slight_smile: Thanks to an awesome collaboration with @parejkoj, we’re refining our validate_base metric framework into something that any DM developer can use to push performance metrics to SQUASH. Soon it’ll be not only the fundamental metrics that @mwv is measuring with validate_drp, but metrics specific to an algorithm to help us measure the health of specific pieces of code. I think this sort of continuous integration will be a game changer for DM.

4 Likes
#3

Thanks @ksk. We are glad that SQuaSH has been useful already! having a working prototype and improving it based on user experience is a successful approach. Our first use case was validade_drp from @mwv that led to the beautiful work by @jsick on the validate_base metric framework. Here is a preview on how the SQuaSH drill down plots can help to understand the way performance metrics are computed and how sensitive are the measurements to selection criteria like the signal-to-noise cut.

Screen Shot 2017-03-02 at 9.56.24 AM.png1556×1402 150 KB

New use cases like the jointcal presented by @parejkoj will help to extend SQuaSH and we look forward to squashfy the HSC validation plots by @laurenam.

This work wouldn’t be possible without SQuaRE’s engineering skills to make everything running in production mode and integrated to our CI system. A special thanks to @josh for doing that.

#4

Hey Simon, first of all thank you for taking the time to give such positive feedback. We really appreciate it. I was hoping SQuaSH would be useful despite its relatively larval state to developers - I know you guys do tons of work and juggle lots of code balls in the air, so you need all the support good test tooling can give you.

As Angelo and Jonathan implies, we’re still doing a lot of rapid development on the SQuaSH design and infrastructure to develop the capabilities that it needs. So we haven’t had the opportunity to work on the developer user experience side as much. Once all the back-end elements are stood up we are hoping to sit with individual/small groups of developers and work with you on your wishlists for improvements.

#5

Nice! @ksk, thanks for pointing out this success story, and many thanks
to all the SQuaSHers, too!