Alert format, serialization, and transport

Science Data Q&A
#1

A question about alert serialization and transport posed in a GitHub issue which we are posting here, as it seems likely to be of broader interest:

The section 3.5 Alerts to DIAsources (Latest Revision: 2019-07-29), describing alert serialization and alert transport, is not consistent with what is described in LDM-612 (Latest Revision: 2019-05-13).

Especially:

  • These alerts will be issued in VOEvent format, and should be readable by VOEvent-compliant clients (LSE-163)

is in conflict with

  • LSST is prototyping a bulk transport system built on the open-source distributed queue system Apache Kafka, with Apache Avro used as a binary serialization format. (LDM-612).

What would be the correct description then?

#2

Two years ago there was a draft IVOA note discussing this issue:

Don’t know the current status. Suggest that at least some relatively small amount of effort is warranted to maintain VOEvent compliance. Considering that the VOEvent working group demonstrated a JSON serialization as early as 2005, this should not be too onerous. (AVRO is layered on JSON.) Also in 2005, a prototype of VOEvent transport was demoed at the NVO summer school using a third-party publish-subscribe paradigm. There’s no reason the Kafka publish-subscribe model shouldn’t work.

Which is to say that requirements on the internal LSST format and protocols can be considered separately from a requirement for easy translation into VOEvent for community interoperability.

#3

I would distinguish between the semantic content of VOEvents, the serialization format, and the transport layer (as @RobSeaman discusses below) . For bulk transport to community brokers LSST is definitely planning to use Kafka in place of VTP. Our prototype alert formats and serialization are not very VOEvent-like, but could be evolved to be more so, and we have had conversations within the IVOA on this topic. Finally I’ll note that we’re committed to providing a formal VOEvent interface from the LSST filtering service, which is user-facing.

Even as it stands, the document conflict is not quite as sharp as it appears: the footnote to the LSE-163 passage above says “Or some other format that is broadly accepted and used by the community at the start of LSST commissioning”, and Avro & Kafka are presently in use by ZTF and several prototype alert brokers.

Finally I’ll note that many of our change-controlled documents would benefit from updates to reflect our most current thinking–this is one such place.

#4

Thank you @MelissaGraham for bringing the issue here, and @ebellm RobSeaman for replies.

I am glad to hear it - I was worried about a change of transport given the big impact on the community broker development.

Concerning the layer [alert system -> community brokers], I am not sure having alerts being more VOEvent-like would be a plus. To my opinion, the real challenge here is to have a high efficiency to transmit data from one side to another - and the combo Kafka & Avro is meant for (both in terms of performance & integration with other big data tools).

However I completely agree with the use of VOEvent-like alerts for the LSST filtering service (and in the exit side of community brokers), as there the integration with the scientific community tools matters the most.