Whilst working on the obs_lsst
metadata translators we’ve been thinking about how exposure Ids are formed. Currently in gen2 butler exposureId seems to be equivalent to CcdExposureId (ie an integer uniquely specifying data from a particular visit/exposure and CCD combination. For obs_lsst
we form these IDs in a number of different ways and all of them now differ from how it’s done in other obs packages.
For this discussion I will use the term detector_exposure_id
to refer to CcdExposureId and exposure_id
to refer to the integer associated with the visit/exposure (but identical for all CCDs).
In the table below nnn
means zero-padded integer and YMDHMSF are derived from a date of ISO format YYYY-MM-DDTHH:MM:SS.FF
Camera | exposure_id | example | detector_exposure_id | example | nbits |
---|---|---|---|---|---|
LSSTCam | YYYYMMDDnnnnnn | 20231231000123 | exposure_id+nnnn | 202312310001230000 | 58 |
AuxTel | YYYYMMDDnnnnnn | 20180920000065 | exposure_id | 20180920000065 | 48 |
TS8 | YYYYMMDDHHMMSSF | 201807241041568 | exposure_id+nn | 20180724104156804 | 55 |
Phosim | RUNNUM | 204595 | exposure_id+nnnn | 2045950038 | 37 |
Imsim | RUNNUM | 3010002 | exposure_id+nnnn | 30100020036 | 37 |
UCDCam | YYYYMMDDHHMMSS | 20181205233148 | exposure_id+n | 201812052331480 | 51 |
Where the number of bits listed is for the maximum value of detector_exposure_id we will encounter.
Before we add obs_lsst into lsst_distrib I would like us to have one last look at how we calculate detector_exposure_id. In particular the max number of bits is much larger than the 32 that we have been using in the past (I’m about to add tests to obs_base that check this for consistency).
For LSSTCam and AuxTel the exposure_id is constructed from the day of observation and a zero-padded sequence number. Currently the sequence number reserves space for 999,999 observations a day. Given we only have 86400 seconds per day that should be changed to support 99,999. I believe that we could in theory take more than 10,000 images in a day so we can’t reserve less space than that. Using YYMMDD format instead of YYYYMMDD will also help. For the detector_exposure_id 4 digits are reserved but this should be 3 (the fourth leading zero was a spacer to make it more readable).
With these changes a detector_exposure_id of 99123199999250 (250 max detectors) requires 47 bits (45 bits in 2032). AuxTel 39.
TS8 is currently configured as if it has no more than 99 detectors but can be read out multiple times per second. Saying that we will never write more than one exposure per second, using the 2-digit year and saying we can have 999 detectors (we are currently debating how to represent this), results in 49 bits still being required. UCDCam is similar but only 3 detectors. TS3 data will be the same but possibly have 3 digits for detector number.
Phosim and Imsim use the run number and the detector number but we only need to use 3 digits for detector number. They more or less fit in 32 bits with space for 250 detectors.
All of these numbers exceed the 32 bits that we assume is more than enough for detector_exposure_id.
Comments welcomed. I intend to make the changes proposed above to at least help the situation.