Are there plans to update the LSST Science Pipeline Throughput versions from v1.4 to later/current release versions?

pollack · November 16, 2021, 9:25pm

According to the GitHub - lsst/throughputs: LSST Simulations repository for baseline evaluation information the latest release for the LSST throughput curves is v1.7. However, It appears that for both LSST middleware (gen3 butler) and LSST Science Pipeline release (tested with w_2021_41) only v1.4 is accepted. In short, my “imsim-like” FITS header has the following header tags:

PKG00001= ‘throughputs’
VER00001= '1.7 ’

For gen2 butler, I am able to ingest the raw CCD frame, but launching the processCcd.py command results in a KeyError. You can see the traceback below.

For gen3 butler, it fails with the butler ingest-raws command. See the second traceback below.

gen2 traceback:

processCcd INFO: Processing {'run': '125339', 'raftName': 'R22', 'expId': 125339, 'detectorName': 'S02', 'detector': 92}
processCcd.isr INFO: Performing ISR on sensor {'run': '125339', 'raftName': 'R22', 'expId': 125339, 'detectorName': 'S02', 'detector': 92}.
processCcd FATAL: Failed on dataId={'run': '125339', 'raftName': 'R22', 'expId': 125339, 'detectorName': 'S02', 'detector': 92}: KeyError: 'z_sim_1.7'
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-26-ge0dcd2c+2fb927f3cf/python/lsst/pipe/base/cmdLineTask.py", line 432, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-26-ge0dcd2c+2fb927f3cf/python/lsst/pipe/base/cmdLineTask.py", line 496, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-26-ge0dcd2c+2fb927f3cf/python/lsst/pipe/base/timer.py", line 181, in wrapper
    res = func(self, *args, **keyArgs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_tasks/21.0.0-149-gc9d3f519+41f9fd9ce7/python/lsst/pipe/tasks/processCcd.py", line 181, in runDataRef
    exposure = self.isr.runDataRef(sensorRef).exposure
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-26-ge0dcd2c+2fb927f3cf/python/lsst/pipe/base/timer.py", line 181, in wrapper
    res = func(self, *args, **keyArgs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ip_isr/22.0.1-20-g32debb5+56e7f78521/python/lsst/ip/isr/isrTask.py", line 1803, in runDataRef
    ccdExposure = sensorRef.get(self.config.datasetType)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_persistence/22.0.1-4-g243d05b+4f1e4d80e3/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
    return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_persistence/22.0.1-4-g243d05b+4f1e4d80e3/python/lsst/daf/persistence/butler.py", line 1417, in get
    return callback()
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_persistence/22.0.1-4-g243d05b+4f1e4d80e3/python/lsst/daf/persistence/butler.py", line 1415, in callback
    return location.mapper.standardize(location.datasetType, innerCallback(), dataId)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_persistence/22.0.1-4-g243d05b+4f1e4d80e3/python/lsst/daf/persistence/mapper.py", line 179, in standardize
    return func(item, self.validate(dataId))
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_lsst/22.0.1-49-gdef5691+588ceb5dad/python/lsst/obs/lsst/lsstCamMapper.py", line 462, in std_raw
    band = self.filterDefinitions.physical_to_band[obsInfo.physical_filter]
KeyError: 'z_sim_1.7'

gen3 traceback:

 butler ingest-raws test_throughput_v1.7 /pbs/home/j/jpollack/Work/Projects/WORKDIR/output97/output_v1.7.fits
ingest INFO: Successfully extracted metadata from 1 file with 0 failures
ingest WARNING: Exposure LSSTCam-imSim:125339 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
[SQL: INSERT INTO exposure (instrument, id, physical_filter, obs_id, exposure_time, dark_time, observation_type, observation_reason, day_obs, seq_num, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING]
[parameters: ('LSSTCam-imSim', 125339, 'z_sim_1.7', '125339', 30.0, 30.0, 'science', 'imsim', 20230327, 0, '125339', 125339, 'UNKNOWN', '125339', 229.18603, 22.61427, 318.571573314725, 53.612893792430846, 1679900648307511839, 1679900678307511839)]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
ingest INFO: Successfully processed data from 0 exposures with 1 failure from exposure registration and 0 failures from file ingest.
ingest INFO: Ingested 0 distinct Butler datasets
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-20-g59bf897+ddb5b0fb80/python/lsst/obs/base/cli/cmd/commands.py", line 118, in ingest_raws
    script.ingestRaws(*args, **kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-20-g59bf897+ddb5b0fb80/python/lsst/obs/base/script/ingestRaws.py", line 71, in ingestRaws
    ingester.run(locations, run=output_run, processes=processes, file_filter=regex)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-26-ge0dcd2c+2fb927f3cf/python/lsst/pipe/base/timer.py", line 181, in wrapper
    res = func(self, *args, **keyArgs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-20-g59bf897+ddb5b0fb80/python/lsst/obs/base/ingest.py", line 1137, in run
    raise RuntimeError("Some failures encountered during ingestion")
RuntimeError: Some failures encountered during ingestion

ktl · November 16, 2021, 9:33pm

To use 1.7 throughputs, you would have to add new filter definitions like obs_lsst/filters.py at master · lsst/obs_lsst · GitHub.

pollack · November 16, 2021, 9:37pm

Thanks a lot for the fast reply. I shall do that.

parejkoj · November 17, 2021, 6:17pm

Note that the only things you should need there are the physical_filter and band. The lambda fields are deprecated and will go away in the future (they are there to support the afw Filter class that is being removed and replaced by the transmission curves you’re working with). So, you’ll add entries like this:

FilterDefinition(physical_filter="u_sim_1.7",
                     band="u")