Recently a number of questions have emerged with respect to image metadata, including metadata derived from observatory events and telemetry and metadata computed by DM pipelines. How this metadata is to be captured, stored, and eventually provided to productions and users has not been clearly documented in LDM-230. The relationship between the EFD and the productions is similarly unspecified.
Gregory and I discussed this a little bit today and came up with the following proposal that we think meets the requirements and clarifies what has been written without significantly conflicting.
Comments/criticisms are welcome as always; we will build the results into the developing design documents.
-
For L1, we store in the L1 Database exposure-centric metadata that includes the metadata captured for AP (which might include certain other values not actually used by AP but determined to be of interest) along with metadata computed in AP.
-
Prompt processing image ingest needs to subscribe via the SAL interface to those metadata topics to be captured as early as possible (soon after the start of integration); retrieve a limited amount of history (e.g. the previous two values) via the SAL interface; and gather all updates during exposure until a defined window (in the tens of milliseconds, exact spec to be located) after readout is complete; then feed those captured values to a predefined set of algorithms to determine what goes in the actual metadata. Examples of those algorithms might be “select last”, “compute average”, “interpolate using X method”, or even “just return the whole list”.
-
As much of the EFD as possible (minus redacted information) should be converted to exposure-centric metadata in the DR L2 Database (along with metadata computed in the DRP); the conversion happens before other DRP processing, making all of this metadata available to the rest of the DRP. This conversion needs to happen separately for each DRP because the algorithms used during the conversion may change.
-
Exposure-centric metadata from any of the above sources can be attached to raw or calibrated exposures on export. For each DRP, a default set will be defined (including at a minimum the metadata used for processing that DR) but users can subtract from (or possibly add to) it.