I have installed the HSC pipeline version 7.9.1 following the download and installation page (https://hsc.mtk.nao.ac.jp/pipedoc/pipedoc_7_e/install_env_e/install.html#install). Installation is fine and then I run the basic steps such as Setting Brighter-Fatter kernel, Creating links to reference catalog for astrometry, Set up the data of the transmission curve for HSC filters, and Y-band stray light. Then I run the task “ingestImages.py” for Setting a registry for raw data. All these tasks run without any error.
After this, I start to make detrending data for HSC. I start to use the task “constructBias.py” and the full command line is given below
constructBias.py $DIR --calib $DIR/CALIB --rerun calib --id visit=130190…130198:2 --clobber-versions --batch-type=smp --cores=32 2>&1 | tee bias_cygob2.log
However, I am getting the following error as given below (full error message pasted here). Any suggestions to resolve this issue? This error I see with version 7.9.1 but not with earlier version 6.7.
250155 ERROR 2020-03-17T11:27:10.695+0530 CameraMapper: Could not turn item=’<lsst.afw.image.image.image.DecoratedImageU object at 0x7f59c37f2e30>’ into an exposure: Column year failed to convert: invalid literal for int() with base 10: ‘<!’
250155 WARN 2020-03-17T11:27:10.696+0530 bias: Unable to process DataId(initialdata={‘visit’: 130190, ‘field’: ‘BIAS’, ‘dateObs’: ‘2017-09-18’, ‘pointing’: 2087, ‘filter’: ‘HSC-R2’, ‘ccd’: 3, ‘taiObs’: ‘2017-09-18’, ‘expTime’: 0.0}, tag=set()): Column year failed to convert: invalid literal for int() with base 10: ‘<!’
ValueError on ln1:250155 in run: Column year failed to convert: invalid literal for int() with base 10: ‘<!’
Traceback (most recent call last):
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/core.py”, line 955, in _convert_vals
converter_func, converter_type = col.converters[0]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 113, in wrapper
return func(*args, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 1070, in run
while not menucommand:
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 240, in wrapper
return func(*args, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 1088, in reduce
result = self._processQueue(context, func, [(index, data)], *args, **kwargs)[0]
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 547, in _processQueue
return self._reduceQueue(context, None, func, queue, *args, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 573, in _reduceQueue
resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/pool.py”, line 573, in
resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.5-hsc+5/python/lsst/pipe/drivers/constructCalibs.py”, line 629, in process
exposure = self.processSingle(sensorRef, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.5-hsc+5/python/lsst/pipe/drivers/constructCalibs.py”, line 648, in processSingle
return self.isr.runDataRef(dataRef).exposure
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/7.0-hsc+1/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.9-hsc/python/lsst/ip/isr/isrTask.py”, line 1441, in runDataRef
ccdExposure = sensorRef.get(self.config.datasetType)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/7.0-hsc/python/lsst/daf/persistence/butlerSubset.py”, line 206, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/7.0-hsc/python/lsst/daf/persistence/butler.py”, line 1396, in get
return callback()
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/7.0-hsc/python/lsst/daf/persistence/butler.py”, line 1394, in callback
return location.mapper.standardize(location.datasetType, innerCallback(), dataId)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/7.0-hsc/python/lsst/daf/persistence/mapper.py”, line 180, in standardize
return func(item, self.validate(dataId))
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc+4/python/lsst/obs/hsc/hscMapper.py”, line 185, in std_raw
exp = super(HscMapper, self).std_raw(item, dataId)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/obs_base/7.0-hsc+1/python/lsst/obs/base/cameraMapper.py”, line 782, in std_raw
trimmed=False, setVisitInfo=True)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/obs_base/7.0-hsc+1/python/lsst/obs/base/cameraMapper.py”, line 1105, in _standardizeExposure
item = exposureFromImage(item, dataId, mapper=self, logger=self.log, setVisitInfo=setVisitInfo)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/obs_base/7.0-hsc+1/python/lsst/obs/base/cameraMapper.py”, line 1372, in exposureFromImage
visitInfo = mapper.makeRawVisitInfo(md=metadata, exposureId=exposureId)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lsst_home/stack/miniconda3-4.5.12-1172c30/Linux64/obs_base/7.0-hsc+1/python/lsst/obs/base/makeRawVisitInfoViaObsInfo.py”, line 122, in call
ut1time = middle.ut1
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/time/core.py”, line 1528, in getattr
tm._set_scale(attr)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/time/core.py”, line 718, in _set_scale
args.append(get_dt(jd1, jd2))
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/time/core.py”, line 1643, in _get_delta_ut1_utc
iers_table = IERS_Auto.open()
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/utils/iers/iers.py”, line 660, in open
cls.iers_table = cls.read(file=filename)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/utils/iers/iers.py”, line 504, in read
iers_a = Table.read(file, format=‘cds’, readme=readme)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/table/connect.py”, line 52, in call
out = registry.read(cls, *args, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/registry.py”, line 523, in read
data = reader(*args, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/connect.py”, line 19, in io_read
return read(filename, **kwargs)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/ui.py”, line 323, in read
dat = reader.read(table)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/cds.py”, line 326, in read
return super().read(table)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/core.py”, line 1207, in read
table = self.outputter(cols, self.meta)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/core.py”, line 986, in call
self._convert_vals(cols)
File “/mnt/home/project/csaumya_gupta/HSCPIPE/hscpipe/7.9.1/lib/python3.7/site-packages/astropy/io/ascii/core.py”, line 971, in _convert_vals
raise ValueError(‘Column {} failed to convert: {}’.format(col.name, last_err))
ValueError: Column year failed to convert: invalid literal for int() with base 10: ‘<!’
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 16