DM Monthly Status Report for July 2021

The DM monthly status report covering July 2021 activities has been posted to DocuShare, collection-8803. For convenience, the High-level Summary is pasted below. Direct link to the full report: DocuShare Authorization Error

High-level Summary

The tiger team led by the DM System Scientist came to a decision about the Rubin Community Alert Brokers program. Seven full stream brokers were selected and two downstream brokers endorsed. All selected brokers have been informed and a press release is being coordinated with the communications team. A DM Technical Note is being prepared following a review of the User Batch requirements by the DM Science Team. The DM system scientist served as a reviewer in the NSF review of US participation in LHC ATLAS & CMS projects.

The release of v22.0.0 of the Science Pipelines. marked the official release of the Gen3 middleware and the start of the deprecation process for the old Gen2 middleware. We have also demonstrated the middleware capability to provide node-local read-only caches of the Butler registry (“Execution Butler”) to our workflow system (PanDA and HTCondor) in order to reduce database load during processing. We also transitioned our PipelineTask framework to use native Python logging.

Among algorithmic improvements, jointcal (the pipeline task performing global astrometric calibration) now supports applying proper motions to matched sources. DM Technical Notes have been published on the status of Scarlet (the blender task) [dmtn-194] and on an approach for masking streaks [dmtn-197]. We also conducted an Image Differencing audit on the status of different configurations of image differencing.

Data Preview 0.1 launched according to plan on June 30th and had its first month of operation on our Google cloud-based Interim Data Facility. While part of the pre-Ops program intended to prepare the science community for LSST data, it also served as a major integration milestone for a number of DM teams and has confirmed the production-readiness of a number of DM services. A major Qserv load-stability issue uncovered by this exercise was successfully investigated and addressed by the team, and a number of improvements were made to the Rubin Science Platform Authentication and Authorization service to resolve behavior that proved confusing to new users.

We demonstrated initial automated calibration production at the summit with OCPS (Observatory Control System Controlled Pipeline System). This is the interface that will be used in commissioning to acquire, process and check daily calibrations. In other summit news, the Long Haul Network backup link is fully operational. Throughput tests are being planned to measure the full bandwidth of the link (40Gb/s); so far only 10Gb/s throughputs have been tested. An initial draft of a DM Technical note (DMTN-198) has been made describing our implementation of our file management system (Data Backbone).

In Data Facility news, emergency security patches were applied to the NCSA teststand cluster and then to the rest of our machines at NCSA a week later. Additional DESC raw files were found at NERSC and ingested into NCSA, and then sent to our Interim Data Facility and the CC-IN2P3 facility in France. Contract and financial work for 2022 was also carried out.