What is the latest adopted Raw FITS File Format?

Data Management
,
#21

No RA and DEC. Here’s the output (note: I made up some keywords like the detector serial number):
instrument: LSSTCam

telescope: Simonyi Survey Telescope
datetime_begin: 2014-09-12T06:00:00.000
altaz_begin: None
boresight_airmass: None
boresight_rotation_angle: None
boresight_rotation_coord: unknown
dark_time: 30.0 s
datetime_end: 2014-09-12T06:00:30.000
detector_exposure_id: 2014091200001153
detector_group: R34
detector_name: S00
detector_num: 153
detector_serial: e2V-CCD250-041
detector_unique_name: R34_S00
exposure_group: 2014091200001
exposure_id: 2014091200001
exposure_time: 30.0 s
location: (1818938.94323602, -5208470.950938, -3195172.08426757) m
object: UNKNOWN
observation_counter: 1
observation_id: 162700
observation_reason: unknown
observation_type: raw
observing_day: 20140912
physical_filter: g
pressure: None
relative_humidity: None
science_program: unknown
temperature: None
tracking_radec: None
visit_id: 2014091200001

#22

I think your FITS header may be missing the RATEL, DECTEL, and ROTANGLE keywords. The first two are the boresight coordinates and the last one is the rotation angle of the camera.
(These are the keywords used by imSim currently. I think if you are using LsstCamMapper, then the LSE-400 keywords are probably what you would want to use.)

#23

Thanks. my primary header lists:
TELRA = 228.34003
TELDEC = 76.56939

but, I do not have ROTANGLE keyword.

#24

Okay. many thanks for the suggestions. I will make adjustments according to LSE-400

#25

Since there is not LSSTCam on-sky data and no example files, I just checked the code and the header translator currently disables ra/dec calculation. We’ll need to fix this before you can use your simulated data.

#26

How long might this fix take? I am in a bit of a time crunch because a test pilot phase is suppose to start next week for Euclid. I am wondering if I could be provided the FITS format compatible with the imSim mapper.

#27

I am working on this in DM-28612 and will try to have something that goes into the next weekly (published Thursday morning Pacific Time).

1 Like
#28

I am looking at the imSim code and it seems to write the extension hdus as described in LCA-13501 except that the keywords for the pointing are different. @jchiang am I wrong?

#29

Correct. The metadata translator for imsim is different. See https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/translators/imsim.py#L86-L90 – you have to pretend to be imsim in order for that code path to be used.

#30

That’s right. Those are the keywords (RATEL, DECTEL, ROTANGLE) that ImsimMapper expects.

#31

I merged this just now so the next weekly should be able to calculate ra/dec for LSSTCam if you use RASTART/DECSTART or RA/DEC.

1 Like
#32

I just tried running processCcd with the lsstCam mapper, but I get the error

File “src/table/IdFactory.cc”, line 57, in lsst::afw::table::{anonymous}::SourceIdFactory::SourceIdFactory(lsst::afw::table::RecordId, int)
Exposure ID ‘3014091200001153’ is too large. {0}
lsst::pex::exceptions::InvalidParameterError: ‘Exposure ID ‘3014091200001153’ is too large.’

It appears to have combined too many keyword values.

#33

I’m not sure what you mean by combining keyword values.

The identifier passed to a SourceIdFactory should be a CcdExposureId, which contains both the exposure identifier and a detector number. In this case, the exposure identifier is 3014091200001, indicating the first CCS-triggered exposure on observing day 2014-09-12. The detector number is 153, which is reasonable. The identifier is a 64-bit integer, of which only 52 are used here.

Something else must be wrong, perhaps in the number of bits that are being reserved. Unfortunately, I will not be able to look at this further until late next week.

#34

Yes, I meant that it’s combining the exposure time, and the detector number with the visit_id. In the previous version when I ran translate_header, which I showed up above 11d ago, the setting was: exposure_id: 2014091200001

#35

But the exposure id that is causing the error is not coming from translate_header, is it? I would expect that to continue returning a 13-digit number.

Is there a reference manual and/or tutorial for using gen3 butler?
#36

No, it is not. It’s later in the processing. here are a few lines:

/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-6-g4e60332+b9160e71e7/python/lsst/ip/isr/isrTask.py:2466: FutureWarning: Call to deprecated method getFilter. (Replaced by getFilterLabel. Will be removed after v22.) – Deprecated since version v22.0.
filterName = afwImage.Filter(exposure.getFilter().getId()).getName() # Canonical name for filter
/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ip_isr/21.0.0-6-g4e60332+b9160e71e7/python/lsst/ip/isr/isrTask.py:2466: FutureWarning: Replaced by FilterLabel. Will be removed after v22.
filterName = afwImage.Filter(exposure.getFilter().getId()).getName() # Canonical name for filter
WARN 2021-02-11T21:00:22.957+0100 processCcd.isr ({‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153})(isrTask.py:2470)- No rough magnitude zero point set for filter g.
INFO 2021-02-11T21:00:22.957+0100 processCcd.isr ({‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153})(isrTask.py:2478)- Setting rough magnitude zero point: 31.692803
INFO 2021-02-11T21:00:22.975+0100 processCcd.charImage ({‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153})(characterizeImage.py:420)- Processing {‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153}
WARN 2021-02-11T21:00:24.021+0100 processCcd.charImage ({‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153})(characterizeImage.py:567)- Source catalog detected and measured with placeholder or default PSF
FATAL 2021-02-11T21:00:24.023+0100 processCcd ({‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153})(cmdLineTask.py:441)- Failed on dataId={‘dayObs’: ‘2014-09-12’, ‘expId’: 3014091200001, ‘raftName’: ‘R34’, ‘detectorName’: ‘S00’, ‘detector’: 153}: InvalidParameterError:
File “src/table/IdFactory.cc”, line 57, in lsst::afw::table::{anonymous}::SourceIdFactory::SourceIdFactory(lsst::afw::table::RecordId, int)
Exposure ID ‘3014091200001153’ is too large. {0}
lsst::pex::exceptions::InvalidParameterError: ‘Exposure ID ‘3014091200001153’ is too large.’

#37

Thanks for looking into it. I know it’s low priority. If you need more information let me know.

#38

A fatal error like this is not low priority, and it looks like it is demonstrating a problem in code that is not well-tested.

I believe the problem is that this is the first time that anyone has tried to run processCcd on a CCS-triggered image. While the code is correct in saying that the exposure id fits within 52 bits out of 64 and thus 12 bits are available, the code that is failing uses a C++ int64_t type and hence only has 63 bits available before its shift-and-compare fails. OCS-triggered images fit within 51 bits and so do not cause this problem.

The simplest solution would be to claim a size of 53 bits for the exposure id. Modifying the record id type to be uint64_t would be better but might have other consequences.

In the meantime, as a workaround, you could simulate an OCS-triggered image (beginning with 2014) rather than a CCS-triggered one.

#39

Sorry I don’t know what a CCS-triggered or a OCS-triggered image is? For clarity, my purpose is to use the latest LSST stack to process a RAW LsstCam image for the e2V sensor that was simulated using an external code developed within the Euclid simulation group.

#40

I have filed your bug as DM-28796.

The first 8 digits of the exposure id encode the observation day as well as the controller that caused the image to be taken. If the date is in the 2000s, the observation was triggered by the Observatory Control System (OCS). If the date is in the 3000s, the observation was triggered by the Camera Control System (CCS) in an engineering mode. If the date is in the 4000s, the observation is from PhoSim.

This code is usually derived from the observation id, which is expected to look like MC-O-20221103-00001, with MC indicating “main camera” and “-O-” indicating OCS trigger as opposed to “-C-” for CCS trigger.

In your case, I suspect the headers you are trying to emulate come from a test stand image that was indeed triggered by the CCS (since there is no observatory for test stands). If you modify those headers to emulate an OCS-triggered image (replace “-C-” by “-O-” in the observation id) instead, things should work.