coaddDriver.py error (HSC data processing)

Vera · July 18, 2017, 10:52pm

So I have been processing the data from HSC with visit=007808-007818 for Abell 2319. The output of singleFrameDrive seems fine with corr/BKGD, CORR and output/ICSRC, SRC, SRCMATCH, SRCMATCHFULL.
However, when I proceed to coaddDriver, there is no output product. Warnings like

WARN  2017-07-17T02:24:20.020 coaddDriver.makeCoaddTempExp ()(makeCoaddTempExp.py:186)- 
  Error processing calexp DataId(initialdata={'taiObs': '2014-09-24', u'pointing': 997, 'visit': 7814, 'dateObs': '2014-09-24', u'filter': 'HSC-I', 'field': 'ABELL2319', 'tract': 0, u'ccd': 70, 'expTime': 240.0}, tag=set([])); skipping it: No locations for get: datasetType:wcs_md dataId:DataId(initialdata={'taiObs': '2014-09-24', u'pointing': 997, 'visit': 7814, 'dateObs': '2014-09-24', u'filter': 'HSC-I', 'field': 'ABELL2319', 'tract': 0, u'ccd': 70, 'expTime': 240.0}, tag=set([]))

are found inside log, not sure if it is the cause though.
Has anyone encountered such problem?
Thank you.

jbosch · July 18, 2017, 11:29pm

The HSC configuration for coaddDriver.py by default tries to use outputs from meas_mosaic, a joint calibration script that can only be used on HSC data. This includes the wcs dataset (improved image registation) and the fcr dataset (improve photometric calibration). You can either run meas_mosaic (you’ll have to install it from git source, as it’s not part of the main LSST pipeline), or you can tell coaddDriver.py to just use the original calibrations from singleFrameDriver.py with the following options. You can find out how to do either here:

https://dmtn-023.lsst.io/#joint-calibration

`

price · July 19, 2017, 12:08am

I didn’t see any instructions on how to run meas_mosaic in DMTN-023, so here are some:

# Build meas_mosaic
cd /work
git clone git://github.com/LSST/meas_mosaic
cd meas_mosaic
scons  # Or 'python $(which scons)' on OSX

# Run meas_mosaic
mosaic.py /path/to/DATA --rerun myRerun -c allowMixedFilters=True --numCoresForRead=12 --diagnostics --diagDir=/path/to/diagnostics --id tract=0 field=MY_FIELD filter=HSC-I^HSC-I2 ccd=0..8^10..103

Here, I’m combining data from the i and i2 filters, and I drop ccd=9 because it usually doesn’t have good data (half the amps are dead).

Vera · July 19, 2017, 1:52am

Thank you so much. I had installed meas_mosaic, yet I have no idea I need to run mosaic.py and how. It is clear now.
For clarification, does field means visit? As I tried patch=0,0^... , it cannot identify it as ID. Also, I reckon meas_mosaic should be run after makeDiscreteSkyMap.py? Then the order in https://dmtn-023.lsst.io/#joint-calibration is a bit confusing.

Besides, as I run
mosaic.py DATA --rerun output8 --numCoresForRead=4 --diagnostics --diagDir=DATA/diagnostics --id tract=0 visit=007802..007806:2 filter=HSC-I ccd=0..8^10..69^71..103
( ccd=70 has an error of lsst::pex::exceptions::InvalidParameterError: 'Mismatch between yaw (-409.047253 deg) and nQuarter (3)) ,
the following error pops out:

Mosaic INFO: Output Flux Diagnostic Figures...
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:827: RuntimeWarning: invalid value encountered in divide
  ra /= numbers
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:828: RuntimeWarning: invalid value encountered in divide
  dec /= numbers
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:829: RuntimeWarning: invalid value encountered in divide
  mag /= err
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:830: RuntimeWarning: invalid value encountered in sqrt
  err = numpy.sqrt((var - mag*mag*err)/err)

As I checked, basically all the fcr and wcs fits are produced, though.

price · July 19, 2017, 2:10am

Those RuntimeWarnings from utils.py are normal. You should now be able to run coaddDriver.py.

I don’t understand the problem with ccd=70; could you elaborate? What command did you run that produced that error? Can you post the entire output from the command?

field doesn’t mean visit. It’s a different way of specifying data. For example, you can specify field=M31 dateObs=2017-07-18 filter=HSC-I or you can give a list of visits, visit=1234..1248:2. Think of it like a database query that will provide the CCDs of interest.

Vera · July 19, 2017, 2:45pm

The error output is like this:

Number good: 72672, 1025
Traceback (most recent call last):
  File "/home/verawu/my_packages/meas_mosaic/bin/mosaic.py", line 5, in <module>
    MosaicTask.parseAndRun()
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 514, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 223, in run
    resultList = list(mapFunc(self, targetList))
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 79, in __call__
    result = task.run(*args)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 1029, in run
    numCoresForReadSource, readTimeout, verbose)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 918, in mosaic
    snapshots, self.outputDir)
lsst.pex.exceptions.wrappers.InvalidParameterError: 
  File "src/shimCameraGeom.cc", line 25, in lsst::afw::geom::Angle lsst::meas::mosaic::getYaw(std::shared_ptr<const lsst::afw::cameraGeom::Detector>)
    Mismatch between yaw (-409.047253 deg) and nQuarter (3) for detector 70: abs(yaw - 90*nQuarter) = 679.047253 is > 90 deg {0}
lsst::pex::exceptions::InvalidParameterError: 'Mismatch between yaw (-409.047253 deg) and nQuarter (3) for detector 70: abs(yaw - 90*nQuarter) = 679.047253 is > 90 deg'

When I tried to include more visits, the error is gone.

price · July 19, 2017, 2:55pm

That looks like a bug in meas_mosaic. Could you please post the command that you ran to get that output, so we can fix it?

Vera · July 19, 2017, 3:14pm

The command is:

mosaic.py DATA --rerun output8 --numCoresForRead=4 --diagnostics --diagDir=DATA/diagnostics --id tract=0 visit=007802..007806:2 filter=HSC-I ccd=0..8^10..103

When i picked visit=007802..007818:2 for the same command, the error is gone and the data product is in place. Though calexp-[filter]-[tract]-[patch].fits is missing, it should be the problem mentioned in DM-10413. Hope the information is helpful.

Thank you.

Vera · July 20, 2017, 8:01pm

Sorry for brining questions again.
As I ran mosaic.py for another filter, HSC-G for 007852 - 007868 of ABELL2319, the warning appeared for many of the ID, e.g.
afw.table WARN: Persisted match record with ID 160992895358036012 not found in catalog 1.
I suspected that this might be something of the processCcd task? Though the output seems normal.

Also, the following error comes out:

Traceback (most recent call last):
  File "/home/verawu/my_packages/meas_mosaic/bin/mosaic.py", line 5, in <module>
    MosaicTask.parseAndRun()
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 514, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 223, in run
    resultList = list(mapFunc(self, targetList))
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 79, in __call__
    result = task.run(*args)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 1029, in run
    numCoresForReadSource, readTimeout, verbose)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 918, in mosaic
    snapshots, self.outputDir)
RuntimeError: solving linear equation failed: dgesv returned 1093

price · July 20, 2017, 8:47pm

That warning means it couldn’t find the reference source in the catalog, and the RuntimeError means it couldn’t solve the matrix. I suppose you’re using the PS1 reference catalog and the latest weekly release?

I think I’m going to have to try to reproduce this myself. I was already working on the previous problem when we hit cluster maintenance time. Can you please let me know the full range of data you’re working on, so I can download it? If you can make the data available in the US that would save some time. And could you please post the list of commands that you’ve run as part of this reduction so I can duplicate?

Vera · July 20, 2017, 9:56pm

Yes, I have retrieved the shards with ra=290.2833, dec=43.3078 and radius=1.6deg from https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110/. For the DM stack and packages, it should be latest.

I am working on the data of ABELL2319 from HSC, which I downloaded from SMOKA. The FRAME ID of SHOTs is HSCA00780[0,1]XX-00781[8,9]XX which is for filter=HSC-I and HSCA00785[0,1]XX-00786[8,9]XX which is for filter=HSC-G.

I have set up meas_extensions_photometryKron and meas_extensions_convolved
The commands I ran for this part is:

ingestImage.py ...    
singleFrameDriver.py DATA --rerun output --id visit=007852..007868:2 --cores=8
makeDiscreteSkyMap.py DATA --rerun output:output9 --id visit=007852..007868:2 --config skyMap.projection="TAN"
mosaic.py DATA --rerun output9 --numCoresForRead=8 --diagnostics --diagDir=DATA/diagnostics3 --id tract=0 visit=007852..007868:2 filter=HSC-G ccd=0..8^10..103

For the numCoresForRead, I have tried 4 and 12. All of them have the same error.

The following works fine with calexp outputs:

ingestImage.py  ...
singleFrameDriver.py DATA --rerun output --id visit=007802..007818:2 --cores=8
makeDiscreteSkyMap.py DATA --rerun output:output9 --id visit=007802..007818:2 --config skyMap.projection="TAN"
mosaic.py DATA --rerun output9 --numCoresForRead=8 --diagnostics --diagDir=DATA/diagnostics2 --id tract=0 visit=007802..007818:2 filter=HSC-I ccd=0..8^10..103
coadd.py ....

I ran the process separately for different filters just as I wanted to try out if it works. Hope the information is clear and helpful.

price · July 21, 2017, 4:57pm

I’ve downloaded the data, and will try it out.

Here’s a thought: I believe you’re using a relatively recent weekly release of the LSST pipeline (not sure which?). It’s possible you’re running into little problems that pop up and get fixed as we go along in the course of development. If you want a more stable release that is known to work pretty well on HSC data, you could try the HSC pipeline, which is based on the LSST pipeline as of a couple of months ago. Instructions for building it are here. The latest version is hscPipe 5.2.

price · July 24, 2017, 3:51pm

I’ve tracked down the problem causing the Persisted match record with ID 160992895358036012 not found in catalog 1 warnings: DM-11356. I’ll fix that and see if it helps meas_mosaic do better.

price · July 25, 2017, 8:11pm

I have a solution for you, @Vera.

Firstly, I think DM-11356 turned out to be mostly a red herring for this case (though I’m glad I fixed it). The main problem is bad astrometry coming out of singleFrameDriver.py. Unfortunately, the astrometry has been a bit unstable recently. My recommendation, below, combines two attempts to improve this situation:

@laurenam recently tweaked an important parameter; this is now on master and is in the weekly release w_2017_29.
@cmorrison has developed a fixed astrometry matcher with better performance. This is available in current weekly releases, but is not active by default.

I used the following configuration with singleFrameDriver.py:

if hasattr(config, "processCcd"):
    config = config.processCcd

# Improves astrometric matching
from lsst.meas.astrom import MatchPessimisticBTask
for cfg in (config.calibrate.astrometry,
            config.charImage.ref_match,
            config.calibrate.photoCal):
   cfg.matcher.retarget(MatchPessimisticBTask)
   cfg.matcher.maxOffsetPix = 250  # not required for w_2017_29 and following

Then meas_mosaic completes successfully with the g-band data.

jeffcarlin · January 26, 2018, 11:21pm

price:

I didn’t see any instructions on how to run meas_mosaic in DMTN-023, so here are some:
# Build meas_mosaic
cd /work
git clone git://github.com/LSST/meas_mosaic
cd meas_mosaic
scons  # Or 'python $(which scons)' on OSX

# Run meas_mosaic
mosaic.py /path/to/DATA --rerun myRerun -c allowMixedFilters=True --numCoresForRead=12 --diagnostics --diagDir=/path/to/diagnostics --id tract=0 field=MY_FIELD filter=HSC-I^HSC-I2 ccd=0..8^10..103
Here, I’m combining data from the i and i2 filters, and I drop ccd=9 because it usually doesn’t have good data (half the amps are dead).

Hi Paul -

I’m trying to process HSC (MADCASH) data on lsst-dev, and am having troubles getting meas_mosaic to work. I’m following the steps you outlined above, but get the following:

[jcarlin@lsst-dev01 meas_mosaic]$ scons
scons: Reading SConscript files ...
EUPS integration: enabled
Checking who built the CC compiler...gcc=4.8.5
CC is gcc version 4.8.5
Checking for C++14 support
Checking whether the C++ compiler works... no
C++14 extensions could not be enabled for compiler 'gcc'

Which I assume is the reason why when I setup meas_mosaic, it is not found in my path?
[jcarlin@lsst-dev01 meas_mosaic]$ cd …/
[jcarlin@lsst-dev01 madcash]$ setup -jr meas_mosaic
[jcarlin@lsst-dev01 madcash]$ mosaic.py
bash: mosaic.py: command not found

Any tips?

-Jeff

price · January 26, 2018, 11:34pm

You need a more modern C++ compiler. I think you want at least gcc 5.0.

jeffcarlin · January 26, 2018, 11:37pm

This is on lsst-dev. Is there not a more modern compiler on lsst-dev? (And can I install things on lsst-dev?)

price · January 27, 2018, 12:31am

Dunno where you’re getting your ancient compiler:

pprice@lsst-dev01:~ $ g++ --version
g++ (GCC) 6.3.1 20170216 (Red Hat 6.3.1-3)
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Maybe something’s cached? Try running scons -c and then scons again.

jeffcarlin · January 27, 2018, 12:34am

price:

Dunno where you’re getting your ancient compiler:

pprice@lsst-dev01:~ $ g++ --version
g++ (GCC) 6.3.1 20170216 (Red Hat 6.3.1-3)
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Maybe something’s cached? Try running scons -c and then scons again.

Still got the same (old) version:

[jcarlin@lsst-dev01 meas_mosaic]$ scons -c
scons: Reading SConscript files ...
EUPS integration: enabled
Setting up environment to build package 'meas_mosaic'.
find . \( -name .svn -prune -o -name \* \) \( -name \*~ -o -name core -o -name \*.so -o -name \*.os -o -name \*.o -o -name \*.pyc -o -name \*.pkgc \) -exec rm -f {} \; ; rm -rf .sconf_temp .sconsign.dblite .sconsign.tmp config.log
Running pytest with 1 process
pytest: running on 2 Python test files.
scons: done reading SConscript files.
scons: Cleaning targets ...
scons: done cleaning targets.
[jcarlin@lsst-dev01 meas_mosaic]$ scons
scons: Reading SConscript files ...
EUPS integration: enabled
Checking who built the CC compiler...gcc=4.8.5
CC is gcc version 4.8.5
Checking for C++14 support
Checking whether the C++ compiler works... no
C++14 extensions could not be enabled for compiler 'gcc'
[jcarlin@lsst-dev01 meas_mosaic]$ g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

price · January 27, 2018, 12:51am

Oh, you need to do source scl_source enable devtoolset-6. See here.