Issues with ingestCalib.py using DECAM images

Support
,
#41

I’ve made some progress on this. You can try the following commands, where REPO is the path to your gen3 butler repo. Note that you’ll need a pretty recent pipelines version, as butler register-dataset-type was added at the end of August.

  1. Create an astropy-readable table file containing one row per input file in the PS1 refcat, with two columns: the filename and htm7-pixel index. I think something like this code will get you the right output:
import glob
import astropy.table
table = astropy.table.Table(names=("filename", "htm7"), dtype=('str', 'int'))
for file in glob.glob("ps1_pv3_3pi_20170110/*.fits"):
    table.add_row((file, int(splitext(file)[0])))
table.write("filename_to_htm.ecsv")
  1. Register the new ps1 refcat dataset:
butler register-dataset-type REPO ps1_pv3_3pi_20170110 SimpleCatalog htm7
  1. Ingest the LSST-formatted files you downloaded:
butler ingest-files -t direct REPO ps1_pv3_3pi_20170110 refcats filename_to_htm.ecsv

The ingested data should appear via butler query-datasets --collections refcats REPO.

1 Like
Recreating the LSST Science pipeline tutorial (gen 2) only using Generation 3 command line tasks and the pipetasks
#42

Hello,

I’m having problems reducing DECam data. In particular the bias combine task is giving tons of errors and producing very few master frames. Below I enclose some of the pipeline output (which is a re-run):

ValueError: Failure from formatter ‘lsst.obs.base.formatters.fitsExposure.FitsExposureFormatter’ for dataset d79309f0-85bf-4569-9b78-b177483c4bfe (cpBiasProc from file:///mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_
20211028T155954Z.fits):
File “src/fits.cc”, line 1578, in lsst::afw::fits::Fits::Fits(const string&, const string&, int)
cfitsio error: error reading from FITS file (108) : Opening file ‘/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
Failed to find the END keyword in header (ffgphd).
Failed to read the required primary array header keywords.
ffopen could not interpret primary array header of file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
{0}
lsst::afw::fits::FitsError: 'cfitsio error: error reading from FITS file (108) : Opening file ‘/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
Failed to find the END keyword in header (ffgphd).
Failed to read the required primary array header keywords.
ffopen could not interpret primary array header of file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
’

ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(CalibCombineTask, label=cpBiasCombine) dataId={instrument: ‘DECam’, detector: 13}> failed; processing will continue for remaining tasks.
ctrl.mpexec.mpGraphExecutor INFO: Executed 0 quanta successfully, 13 failed and 44 remain out of total 57 quanta.
timer.lsst.daf.butler.datastores.fileDatastore ERROR: Reading from location file:///mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits with formatter lsst.obs.base.formatters.fitsExposure.FitsExposure
Formatter: Took 17.8089 seconds
ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task ‘cpBiasCombine’ on quantum {instrument: ‘DECam’, detector: 26} failed
Traceback (most recent call last):
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-70-gc781c74d+f4bf734fca/python/lsst/daf/butler/datastores/fileDatastore.py”, line 1215, in _read_artifact_into_memory
result = formatter.read(component=getInfo.component if isComponent else None)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-18-g3ad3863+50f8e256e0/python/lsst/obs/base/formatters/fitsExposure.py”, line 89, in read
return self.readFull()
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-18-g3ad3863+50f8e256e0/python/lsst/obs/base/formatters/fitsExposure.py”, line 560, in readFull
result = self.reader.read(self.checked_parameters)
lsst.pex.exceptions.wrappers.FitsError:
File “src/fits.cc”, line 1419, in void lsst::afw::fits::Fits::readImageImpl(int, T, long int, long int*, long int*) [with T = float]
cfitsio error (/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits): error reading from FITS file (108) : Reading image
cfitsio error stack:
Error reading data buffer from file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits
Error reading elements 1 thru 2813 from column 1 (ffgclb).
error reading compressed byte stream from binary table
{0}
lsst::afw::fits::FitsError: 'cfitsio error (/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits): error reading from FITS file (108) : Reading image
(base) [jahumada@leftraru4 ~]$ ls /mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits*
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits
(base) [jahumada@leftraru4 ~]$ tail plira_23665563.err
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/cli/script/run.py”, line 171, in run
f.runPipeline(qgraphObj, taskFactory, args)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 669, in runPipeline
executor.execute(graph, butler)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 300, in execute
self._executeQuantaMP(graph, butler)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 486, in _executeQuantaMP
raise MPGraphExecutorError(“One or more tasks failed or timed out during execution.”)
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed or timed out during execution.
slurmstepd: error: Detected 48 oom-kill event(s) in step 23665563.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

Sorry about so much mess!

Paulina.

#43

This looks like the file is completely corrupt. Can other tools read it? We usually only encounter problems like this if you’ve run out of disk space.