'Dimension mismatch' issue of constructFlat.py in hscPipe v7.0

Support
#1

Hi,

I am testing the new release of hsePipe v7.0, for the HSC data taken in this month. But it came out a error of “dimension mismatch” and “invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian” in near the end of process. Does anyone know what might help here?

This is the command and parameters I executed:

constructFlat.py $WORKDIR --rerun $CALVRERUN --calib=$WORKDIR/CALIB --id visit=172414..172434:2 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --job $CALVRERUN-g --time 300 --clobber-config --cores 40

I also tested the same command with the flat taken at 2018-01-10 (135866…135884:2). I still got a similar errors as previous test . Note that this flat data (2018-01-10) could be processed by v5.4, but not later version.

The following text is the error message:

118310 INFO 2019-04-25T07:28:51.720+0800 flat: Writing {‘ccd’: 110, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:118310
118298 INFO 2019-04-25T07:28:56.444+0800 flat: Writing {‘ccd’: 107, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:118298
afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “0_31” into image of camera:
File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]
Dimension mismatch: 65x32 v. 32x65 {0}
lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’
afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “1_35” into image of camera:
File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]
Dimension mismatch: 65x32 v. 32x65 {0}
lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’
afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “0_35” into image of camera:
File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]
Dimension mismatch: 65x32 v. 32x65 {0}
lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’
afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “1_31” into image of camera:
File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]
Dimension mismatch: 65x32 v. 32x65 {0}
lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.0-hsc/python/lsst/pipe/drivers/constructCalibs.py:1076: RuntimeWarning: divide by zero encountered in log
bgMatrix = np.log(bgMatrix)
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.0-hsc/python/lsst/pipe/drivers/constructCalibs.py:1083: UserWarning: Warning: converting a masked element to nan.
compScales = np.array([(bgMatrix[i1, :] - expScales).mean() for i1 in range(numCcds)])
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.0-hsc/python/lsst/pipe/drivers/constructCalibs.py:145: RuntimeWarning: divide by zero encountered in double_scalars
combined *= finalScale / background
/opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian

#2

That looks like a non-fatal error due to a recently-introduced bug in the camera overview construction code. Could you please confirm that the individual CCD images have been created successfully?

#3

I’m still confused as to where/what the bug actually is, but is it possible that there needs to be a

callback = cgUtils.rawCallback if dtype == “raw” else None

type of clause to the imageSource in visualizeVisit’s makeCameraImage() function (I’m thinking the flat construction is starting from “raw” data, so needs the obeyNQuarter clause that was moved to the rawCallback in DM-19371)? Again, kind of shooting at straws…

#4

What do yo mean “individual CCD images”?
I don’t see the file created in $WORKDIR/CALIB/FLAT
, but I do see the 112 files created in $WORKDIR/rerun/ssp-201904-pi-hp7-flat/FLAT/2018-01-10/HSC-G/FLAT-2018-01-10-HSC*.fits
I remembered that --calib=$WORKDIR/CALIB should be the default path for all FLAT?

I also tried v7.1 today, but it has same error message.

Charles

#5

Is there a solution for this issue now?

#6

I believe this has now been fixed on master of pipe_drivers (at least it seems like a case of DM-19412), and should land in the next LSST weekly (w_2018_19).

I believe it should be included in hscPipe 7.3 as well.

#7

Our system administrator noticed that the latest version of pipe_drivers is 7.0-hsc+1 but the dependence of hscPipe v7.3 is linked to pipe_drivers 7.0-hsc. And I also got the same error message when I running the same process with v7.3. Is there still a issue in the codes?

#8

I suspect you may be running into the problem noted in the email announcing the 7.3 release:

I messed up building the previous releases, in that they were not built in the same stack, which means the version numbering of products with changed dependencies is wrong (the “+1” versions weren’t created). For this reason, it’s necessary to install this release in a completely fresh stack from versions 7.0 through 7.2. Sorry about that!

#9

This got fixed in hscPipe 7.4.

#10

I still got following error message from the process by hscPipe 7.4. It (divide by zero encountered in log) looks like the second part of error message of my original post.

50850 INFO 2019-05-21T10:09:34.503+0800 flat: Writing {‘ccd’: 99, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:50850
50877 INFO 2019-05-21T10:09:47.485+0800 flat: Writing {‘ccd’: 109, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:50877
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.4-hsc/python/lsst/pipe/drivers/constructCalibs.py:1076: RuntimeWarning: divide by zero encountered in log
bgMatrix = np.log(bgMatrix)
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.4-hsc/python/lsst/pipe/drivers/constructCalibs.py:1083: UserWarning: Warning: converting a masked element to nan.
compScales = np.array([(bgMatrix[i1, :] - expScales).mean() for i1 in range(numCcds)])
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/7.4-hsc/python/lsst/pipe/drivers/constructCalibs.py:145: RuntimeWarning: divide by zero encountered in double_scalars
combined *= finalScale / background
/opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true_divide
flatness = (skyLevels[good] - skyMedian) / skyMedian

#11

Those are not errors, but warnings. They might be indicative of something bad going on, but much more likely it’s just numpy being a bit too chatty. It looks worse than it is, because it appears to hold them over until the end and dump a bunch all at once. My guess is that they’re coming from ccd=9, which is dead.

#12

Your guess is about right, I think.
The FLAT files on $WORKDIR/rerun/ssp-201904-pi-hp7-flat/FLAT/201*/HSC-*/FLAT-*.fits with only ~1.1MB (others are ~21MB) are FLAT*009.fits. The pixel values of FLAT*009.fits are all NAN. The values of raw HSC-*009.fits are all “blank” on ds9.

#13

Is there a fix for that in next release?