Unified exposure metadata

[rowen (Russell Owen)]

DM-5503 introduces new unified metadata for exposures. The data is obtained using exposure.getInfo().getVisitInfo(). This returns an lsst.afw.image.VisitInfo, which contains the following data (though not all data is available for all cameras):

• exposure ID
• exposure duration (shutter open time)
• dark time: time from CCD flush to readout, including shutter open time
• date of exposure
• universal time (unknown for most cameras)
• earth rotation angle (unknown for most cameras)
• boresight RA/Dec
• boresight AzAlt
• boresight airmass
• boresight rotation angle (not yet set for most cameras)
• rotation type
• observatory longitude, latitude and altitude
• basic weather information

Visit info obtained from an exposure using exposure.getInfo().getVisitInfo().

To make this work, the camera mapper in every obs_ package has a subclass of lsst.obs.base.MakeRawVisitInfo which generates a VisitInfo from the FITS header of a raw image. This is used by the data butler when reading raw images, in order to provide suitable VisitInfo. I adapted all obs_ packages that I knew of, except obs_monocam. However, in most cases I was unable to figure out how the rotator angle (some cameras provide such a value, but it is not sufficiently well documented), so more work can be done here. As far as I can tell no cameras provide a UT1 date or earth rotation angle.

In theory all CCDs in a visit should have the same visit info, but this is not enforced and will not be true if the raw header data used to create the visit info varies from CCD to CCD.

For the most part this change is backwards compatible. However, moving exposure time and date from Calib has been somewhat disruptive:

• When you read old coadds, the exposure time and date are no longer available for each of the exposures that went into the ExposureTable. They cannot be put into the new Calib object, and there was no straightforward way to generate a VisitInfo for this case.
• Much old code set the exposure time by updating the Calib in an existing exposure. That no longer works because VisitInfo is immutable. The existing code was modified to set the exposure time a bit earlier.

[RHL (Robert Lupton)]

When you read old coadds, the exposure time and date are no longer available for each of the exposures that went into the ExposureTable. They cannot be put into the new Calib object, and there was no straightforward way to generate a VisitInfo for this case.

I’m concerned by this. Why can’t we read the old numbers and put them in the new place?

Much old code set the exposure time by updating the Calib in an existing exposure. That no longer works because VisitInfo is immutable. The existing code was modified to set the exposure time a bit earlier.

I’m concerned by this too. A common pattern is to take an exposure, do some processing, estimate the zero point and update. I can see why you might need to copy the old visitInfo, update it, and set the exposure, but in that case why did we need to change the code to set it earlier?

[jbosch (Jim Bosch)]

I’m concerned by this. Why can’t we read the old numbers and put them in the new place?

It’s just hard to implement because we’d need to break through a few levels of encapsulation to do it:

• VisitInfo's persistence doesn’t get access to values that were previously persisted as part of Calib when the persistence framework asks it to reconstruct itself; it only gets access to the values that were persisted as part of VisitInfo.

• ExposureRecord, which holds them both, only gets access to an integer handle for each that it can pass to the persistence framework to reconstruct them.

I advised @rowen that it probably wasn’t worth adding this, because the lack of backwards compatibility should only affect ExposureRecord, not Exposure itself (if this broke Exposure backwards compatibility too that should be considered a bug; though now that I think about it I’m not sure we tested that). As a result, this only affects our ability to get exposure times out of existing CoaddInputs catalogs, which I believe are the only ExposureRecords we persist that are not just an implementation detail.

[rowen (Russell Owen)]

I’m concerned by this too. A common pattern is to take an exposure, do some processing, estimate the zero point and update. I can see why you might need to copy the old visitInfo, update it, and set the exposure, but in that case why did we need to change the code to set it earlier?

The zero point is in the Calib object and can be set at will. But if you find you need to adjust the exposure time, you should probably do it in a different object. VisitInfo is meant to be per visit, not per CCD. The CCD-specific metadata is a task remaining to be done. In particular the airmass, exposure time and exposure date may all be slightly different at each CCD. I had hoped we could manage with a model for these, but perhaps that will not suffice.

[RHL (Robert Lupton)]

That’s OK. But I think we should check the Exposure claim.

[RHL (Robert Lupton)]

That’s OK then.

I don’t think that a model will be good enough, if only because we need to persist the per-CCD data for end users (including databases).