The DM monthly status report covering August has been posted in Docushare, Collection-4640. For your convenience I am pasting high-level summary and project status below. Direct link to the full report (pdf): http://ls.st/kyb.
High-Level Summary
Data Management development continues to track the Fall 2016 rollingwave cycle plan. The
team completed bulk of the effort related to updating the DM software to Python 3. Work on
setting up the Prototype Data Access Center (PDAC) continued. Notable advancements were
made in SuperTask design, algorithms for image differencing and L1/AP database prototyping.
JavaScript and Python APIs for Firefly were completed, and SWIG to pybind11 migration was
thoroughly researched and proposed.
The whole team met at the LSST2016 meeting in Tucson, the DMLT participated in the
NSF/DOE Annual Status Review.
The work on replan continued and is on track for completion by the end of this calendar year.
Project Status
Selected technical accomplishments:
-
The Architecture Team played a major role in the 2016 Project and Community Workshop, including defining the DM session agenda, preparing for and organizing Python 3 and pytest porting, leading discussions on Commissioning and Deep Drilling from the DM perspective, and facilitating cross-team interactions. The team also devoted substantial energy to the NSF/DOE Annual Status Review held at the same time. The DM Technical Control Team was reconstituted and began work. Progress was made on SuperTask designs.
-
The AP Team produced a DMTN describing measurement on difference images using a noise decorrelation technique which greatly improves signal to noise estimates of the detections. The team also implemented an algorithm to infer DCR-less templates from a set of simulated images. Finally, the team began work on the alert distribution system by identifying technologies for implementing a prototype alert distribution system.
-
The DRP Team completed its exploratory work on converting the LSST stack’s C++/Python interface from SWIG to pybind11. A report and summary have been prepared for the community; over the next month, the project will make a decision about its future. Members of the team made major contributions to the effort, led by Tim Jenness of the System Architecture team, to update the stack to run on Python 3 and to use the pytest system for unit test. The team continued its effort to improve design documentation and implement the replan.
-
The SUIT Team attended the LSST2016 all hands and project manager candidates interviews. The team continued the regression tests and bug fixes of the newly migrated JS based Firefly system, finished the new JS and Python APIs for Firefly, and produced two Jupyter widgets using Firefly image and table display.
-
The DAX Team progressed with assembly and preparation of the SDSS Stripe 82 data products to be loaded into the PDAC, and data is now being staged at NCSA. An L1/AP database prototype was stood up at IN2P3 and some initial performance measurements were done, revealing that that some further schema changes may be required to meet baseline performance criteria. Data Butler improvements/enhancements continued, including completion of initial design work on the composite datasets feature. Containerization infrastructure for Qserv and DAX web services was advanced in preparation for PDAC deploy.
-
Activities within the Net&BS Team: Mountain - Base Network: Cobra/Telefonica finalized the phase 1 of the survey and produced a report. There are 14 posts in total that need to be replaced, apart from the 43 Telefonica will install as intermediate posts. We started work on phase 2 and appears it will approved by early September to commence that part of the work. There have been some differences in safety policies but appears that is now a non-issue. DWDM Equipment: Coriant is selected internally and work commenced on the final contracts for Reuna and AURA. Alvaro Soms is working to produce a contract defined by AURA. ADVA has not been informed of the decision in case things go awry with Coriant and we have to backtrack.
-
The NCSA Team:
-
Presented at the NSF/DOE Joint Status Review and attended the LSST2016 Project and Community Workshop
-
Completed the final report from our investigation of eight workflow systems, available as DM Tech Note, “DMTN-025: A survey of workflow management systems”.
-
Deployed the FY16 object storage evaluation infrastructure. Completed an assessment of four object store technologies to evaluated the effectiveness of object storage as a custodial file store within the Data Backbone (link).
Completed the semi-annual refresh of the costing model, with updates to LDM-143 and LDM-144, submitted to the DM TCT for approval via RFC-210. -
Developed a logical design of data flows at the Summit and Base Sites through DPP processing and archiving services, and worked through use case of disconnected observing with representatives from other subsystems in the new Chilean ITC “Tiger Team.”
-
Drafted a concept of operations for Service Monitoring system needed to monitor the DPP services during operation. Supports operational processes and concerns such as incident response, problem management, performance, capacity management, and availability management.
-
The SQuaRE Team Contributed to the LSST2016 Workshop and NSF reviews, and deployed an OSX CI slave farm.