Obs_decam + processCcd.py not finding data

[jmac (Jacqueline McCleary)]

Hello there,

I’m fairly new to using DMstack and obs_decam, and am working off the (outdated?) wiki and the documentation here: https://github.com/lsst/obs_decam. I’m using a set of test DECam images, taken from GitHub.

I ingest the testdata_decam images with the command:

ingestImagesDecam.py /Users/jmcclear/Research/decam_test/ --mode=link /Users/jmcclear/Research/testdata_decam/cpData/original_files/instcal/*fits.fz

As expected, a registry is created in the top level of the repository, as is a directory called “0229388/” containing the relevant data files. There are a few ingest.parse errors thrown, but this thread (Error of using obs_decam) seems to suggest that input.parse errors aren’t the biggest deal.

However, the command:

processCcd.py /Users/jmcclear/Research/decam_test/ --id visit=229388 ccdnum=13 --output /Users/jmcclear/Research/decam_test/outDir --config calibrate.doAstrometry=False calibrate.doPhotoCal=False

throws the following error:

root INFO: Loading config overrride file '/Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/obs_decam/14.0-4-g3127c16+22/config/processCcd.py’
CameraMapper INFO: Loading exposure registry from /Users/jmcclear/Research/decam_test/registry.sqlite3
root WARN: No data found for dataId=OrderedDict([(‘visit’, 229388), (‘ccdnum’, 13)])
root INFO: Running: /Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pipe_tasks/14.0-27-g4464a260+2/bin/processCcd.py /Users/jmcclear/Research/decam_test/ --id visit=229388 ccdnum=13 --output /Users/jmcclear/Research/decam_test/outDir --config calibrate.doAstrometry=False calibrate.doPhotoCal=False
root WARN: Not running the task because there is no data to process; you may preview data using “–show data”

If I try to run processCcd.py within the folder created by IngestImagesDecam.py, I get a whole bunch more exotic-looking errors. Just in case it’s useful to know my directory structure, it looks like this:

ls /Users/jmcclear/Research/decam_test/*/*fz
/Users/jmcclear/Research/decam_test/0229388/dqmask0229388.fits.fz /Users/jmcclear/Research/decam_test/0229388/wtmap0229388.fits.fz
/Users/jmcclear/Research/decam_test/0229388/instcal0229388.fits.fz

And the command

sqlite3 registry.sqlite3 ‘SELECT * FROM raw WHERE visit=229388 AND ccdnum=13’

yields the following, i.e. the entry definitely should exist:

13|229388|z|2013-09-01|2013-09-01T06:02:55.753848|200.0|13|13|13|/Users/jmcclear/Research/testdata_decam/cpData/original_files/instcal/c4d_130901_060255_ooi_z_a1.fits.fz|/Users/jmcclear/Research/testdata_decam/cpData/original_files/dqmask/c4d_130901_060255_ood_z_a1.fits.fz|/Users/jmcclear/Research/testdata_decam/cpData/original_files/wtmap/c4d_130901_060255_oow_z_a1.fits.fz|2012B-0001|DES supernova hex SN-S1 tiling 22

ETA In case it’s relevant, I downloaded the /master/ version of loadLSST.bash this past weekend, and am running the w_2017_48 version of DMstack. An earlier set of obs_decam problems I was having were resolved by updating to this version, thanks to @price .

[price (Paul Price)]

The file the pipeline is looking for should be named /Users/jmcclear/Research/decam_test/<date>/<filter>/decam<visit>.fits.fz.

Perhaps you’re trying to reduce data that has been processed with the Community Pipeline?

[jmac (Jacqueline McCleary)]

Hello!

I am in fact trying to do just that – reduce data that has already been through the Community Pipeline. The documentation online even has a line for ingesting from a folder called instcal/, which suggests that the data has been through basic detrending.

Is there something else other than processCcd.py I should be running to get things like shape measurements, photometry, etc.?

Thank you!

[price (Paul Price)]

I don’t think that’s a mode we support any more (from the command-line, at least; there used to be a processCcdDecam.py). @hsinfang may know.

EDIT: It turns out I was completely wrong. See below.

[ctslater (Colin Slater)]

Everything you’ve done so far looks good, the only thing missing is that a configuration file has to be supplied to processCcd.py that tells it to expect Community Pipeline images. This is item #6 in the obs_decam README file, which I understand is somewhat hidden place for this information to live.

The config file looks like:

 from lsst.obs.decam.decamNullIsr import DecamNullIsrTask
 config.isr.retarget(DecamNullIsrTask)

This file is supplied on the command line with the -C my_config_file.py, for example.

[jmac (Jacqueline McCleary)]

Woohoo! Thank you @ctslater, that was the fix! I had seen that line in the readme, and had wondered how I was supposed to know to configure the config file. Guess it didn’t connect that “instcal” meant not-raw, CP-processed data; I got a little confused and thought that item #6 referred to re-reducing raw DECam data from scratch.

Anyway, the following command (with the --clobber-config option on!) yielded an icSrc file in my output directory.

processCcd.py /Users/jmcclear/Research/decam_test/ --id visit=229388 ccdnum=13 --output /Users/jmcclear/Research/decam_test/outDir -C override_config.py --config calibrate.doAstrometry=False calibrate.doPhotoCal=False --clobber-config

The process ended in a segfault, which I assume is a good thing – whatever was stored in memory being dumped.

I’ll spend some more time parsing the output, but meanwhile thank you again for your patience and help!

[ctslater (Colin Slater)]

Segfault!? That’s really bad news and not expected. I’d be happy to have a look at that too.

[jmac (Jacqueline McCleary)]

Hello!

I didn’t realize segfaults were a no-go, so thanks for the follow-up. I guess the best thing I can do would be to paste the entire output, long though it might be. Sorry in advance for the tl;dr…

processCcd INFO: Processing {‘visit’: 229388, ‘ccdnum’: 13, ‘hdu’: 13}
processCcd.isr INFO: Loading DECam community pipeline file {‘visit’: 229388, ‘ccdnum’: 13, ‘hdu’: 13}
makeWcs WARN: Stripping PVi_j keys from projection RA—TPV/DEC–TPV
processCcd.charImage INFO: Processing {‘visit’: 229388, ‘ccdnum’: 13, ‘hdu’: 13}
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.repair INFO: Identified 97 cosmic rays.
processCcd.charImage.detection INFO: Detected 127 positive sources to 50 sigma.
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage.measurement INFO: Measuring 127 sources (127 parents, 0 children)
processCcd.charImage.measurePsf INFO: Measuring PSF
processCcd.charImage.measurePsf.reserve INFO: Reserved 0/55 sources
processCcd.charImage.measurePsf INFO: PSF star selector found 55 candidates
processCcd.charImage.measurePsf INFO: PSF determination using 33/55 stars.
processCcd.charImage INFO: iter 1; PSF sigma=1.87, dimensions=(29, 29); median background=3540.54
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.repair INFO: Identified 115 cosmic rays.
processCcd.charImage.detection INFO: Detected 119 positive sources to 50 sigma.
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage.measurement INFO: Measuring 119 sources (119 parents, 0 children)
processCcd.charImage.measurePsf INFO: Measuring PSF
processCcd.charImage.measurePsf.reserve INFO: Reserved 0/55 sources
processCcd.charImage.measurePsf INFO: PSF star selector found 55 candidates
processCcd.charImage.measurePsf INFO: PSF determination using 34/55 stars.
processCcd.charImage INFO: iter 2; PSF sigma=1.87, dimensions=(29, 29); median background=3540.55
processCcd.charImage.repair INFO: Identified 106 cosmic rays.
processCcd.charImage.measurement INFO: Measuring 119 sources (119 parents, 0 children)
processCcd.charImage.measureApCorr INFO: Measuring aperture corrections for 2 flux fields
processCcd.charImage.measureApCorr INFO: Aperture correction for base_PsfFlux: RMS 0.018234 from 32
processCcd.charImage.measureApCorr INFO: Aperture correction for base_GaussianFlux: RMS 0.016684 from 32
processCcd.charImage.applyApCorr INFO: Applying aperture corrections to 2 flux fields
processCcd.calibrate INFO: Processing {‘visit’: 229388, ‘ccdnum’: 13, ‘hdu’: 13}
processCcd.calibrate.detection INFO: Detected 586 positive sources to 5 sigma.
processCcd.calibrate.detection INFO: Resubtracting the background after object detection
processCcd.calibrate.deblend INFO: Deblending 586 sources
processCcd.calibrate.deblend INFO: Deblended: of 586 sources, 90 were deblended, creating 452 children, total 1038 sources
processCcd.calibrate.measurement INFO: Measuring 1038 sources (586 parents, 452 children)
Caught signal 11, backtrace follows:
0 libutils.dylib 0x000000011098e5b1 lsst::utils::(anonymous namespace)::signalHandler(int) + 81
1 libsystem_platform.dylib 0x00007fff7606af5a (null) + 26
1 libsystem_platform.dylib 0x00007fff7606af5a _sigtramp + 26
2 ??? 0x000000014be9c148 0x0 + 5568577864
3 libc++.1.dylib 0x00007fff73e598fd std::__1::__shared_weak_count::__release_shared() + 43
4 applyLookupTable.so 0x000000010d2a150b pybind11_object_dealloc + 43
5 libpython3.6m.dylib 0x000000010a8c2604 frame_dealloc + 132
6 libpython3.6m.dylib 0x000000010a966bb8 fast_function + 408
7 libpython3.6m.dylib 0x000000010a965901 call_function + 401
8 libpython3.6m.dylib 0x000000010a9620d2 (null) + 27362
8 libpython3.6m.dylib 0x000000010a9620d2 _PyEval_EvalFrameDefault + 27362
9 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
9 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
10 libpython3.6m.dylib 0x000000010a966f42 (null) + 738
10 libpython3.6m.dylib 0x000000010a966f42 _PyFunction_FastCallDict + 738
11 libpython3.6m.dylib 0x000000010a89ae87 (null) + 247
11 libpython3.6m.dylib 0x000000010a89ae87 _PyObject_FastCallDict + 247
12 libpython3.6m.dylib 0x000000010a89afa5 (null) + 149
12 libpython3.6m.dylib 0x000000010a89afa5 _PyObject_Call_Prepend + 149
13 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
14 libpython3.6m.dylib 0x000000010a9623b5 (null) + 28101
14 libpython3.6m.dylib 0x000000010a9623b5 _PyEval_EvalFrameDefault + 28101
15 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
15 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
16 libpython3.6m.dylib 0x000000010a966f42 (null) + 738
16 libpython3.6m.dylib 0x000000010a966f42 _PyFunction_FastCallDict + 738
17 libpython3.6m.dylib 0x000000010a89ae87 (null) + 247
17 libpython3.6m.dylib 0x000000010a89ae87 _PyObject_FastCallDict + 247
18 libpython3.6m.dylib 0x000000010a89afa5 (null) + 149
18 libpython3.6m.dylib 0x000000010a89afa5 _PyObject_Call_Prepend + 149
19 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
20 libpython3.6m.dylib 0x000000010a9623b5 (null) + 28101
20 libpython3.6m.dylib 0x000000010a9623b5 _PyEval_EvalFrameDefault + 28101
21 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
21 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
22 libpython3.6m.dylib 0x000000010a966c41 fast_function + 545
23 libpython3.6m.dylib 0x000000010a965901 call_function + 401
24 libpython3.6m.dylib 0x000000010a962171 (null) + 27521
24 libpython3.6m.dylib 0x000000010a962171 _PyEval_EvalFrameDefault + 27521
25 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
25 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
26 libpython3.6m.dylib 0x000000010a966c41 fast_function + 545
27 libpython3.6m.dylib 0x000000010a965901 call_function + 401
28 libpython3.6m.dylib 0x000000010a9620d2 (null) + 27362
28 libpython3.6m.dylib 0x000000010a9620d2 _PyEval_EvalFrameDefault + 27362
29 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
29 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
30 libpython3.6m.dylib 0x000000010a95b5a4 PyEval_EvalCodeEx + 100
31 libpython3.6m.dylib 0x000000010a8c42ad function_call + 381
32 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
33 libpython3.6m.dylib 0x000000010a9623b5 (null) + 28101
33 libpython3.6m.dylib 0x000000010a9623b5 _PyEval_EvalFrameDefault + 28101
34 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
34 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
35 libpython3.6m.dylib 0x000000010a966c41 fast_function + 545
36 libpython3.6m.dylib 0x000000010a965901 call_function + 401
37 libpython3.6m.dylib 0x000000010a962171 (null) + 27521
37 libpython3.6m.dylib 0x000000010a962171 _PyEval_EvalFrameDefault + 27521
38 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
38 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
39 libpython3.6m.dylib 0x000000010a966c41 fast_function + 545
40 libpython3.6m.dylib 0x000000010a965901 call_function + 401
41 libpython3.6m.dylib 0x000000010a962171 (null) + 27521
41 libpython3.6m.dylib 0x000000010a962171 _PyEval_EvalFrameDefault + 27521
42 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
42 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
43 libpython3.6m.dylib 0x000000010a95b5a4 PyEval_EvalCodeEx + 100
44 libpython3.6m.dylib 0x000000010a8c42ad function_call + 381
45 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
46 libpython3.6m.dylib 0x000000010a9623b5 (null) + 28101
46 libpython3.6m.dylib 0x000000010a9623b5 _PyEval_EvalFrameDefault + 28101
47 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
47 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
48 libpython3.6m.dylib 0x000000010a966c41 fast_function + 545
49 libpython3.6m.dylib 0x000000010a965901 call_function + 401
50 libpython3.6m.dylib 0x000000010a962171 (null) + 27521
50 libpython3.6m.dylib 0x000000010a962171 _PyEval_EvalFrameDefault + 27521
51 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
51 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
52 libpython3.6m.dylib 0x000000010a95b5a4 PyEval_EvalCodeEx + 100
53 libpython3.6m.dylib 0x000000010a8c42ad function_call + 381
54 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
55 libpython3.6m.dylib 0x000000010a9623b5 (null) + 28101
55 libpython3.6m.dylib 0x000000010a9623b5 _PyEval_EvalFrameDefault + 28101
56 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
56 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
57 libpython3.6m.dylib 0x000000010a966f42 (null) + 738
57 libpython3.6m.dylib 0x000000010a966f42 _PyFunction_FastCallDict + 738
58 libpython3.6m.dylib 0x000000010a89ae87 (null) + 247
58 libpython3.6m.dylib 0x000000010a89ae87 _PyObject_FastCallDict + 247
59 libpython3.6m.dylib 0x000000010a89afa5 (null) + 149
59 libpython3.6m.dylib 0x000000010a89afa5 _PyObject_Call_Prepend + 149
60 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
61 libpython3.6m.dylib 0x000000010a9623b5 (null) + 28101
61 libpython3.6m.dylib 0x000000010a9623b5 _PyEval_EvalFrameDefault + 28101
62 libpython3.6m.dylib 0x000000010a966dbc (null) + 348
62 libpython3.6m.dylib 0x000000010a966dbc _PyFunction_FastCallDict + 348
63 libpython3.6m.dylib 0x000000010a89ae87 (null) + 247
63 libpython3.6m.dylib 0x000000010a89ae87 _PyObject_FastCallDict + 247
64 libpython3.6m.dylib 0x000000010a89afa5 (null) + 149
64 libpython3.6m.dylib 0x000000010a89afa5 _PyObject_Call_Prepend + 149
65 libpython3.6m.dylib 0x000000010a89acc0 PyObject_Call + 96
66 libpython3.6m.dylib 0x000000010a8fae15 slot_tp_call + 117
67 libpython3.6m.dylib 0x000000010a89ae55 (null) + 197
67 libpython3.6m.dylib 0x000000010a89ae55 _PyObject_FastCallDict + 197
68 libpython3.6m.dylib 0x000000010a95723a map_next + 154
69 libpython3.6m.dylib 0x000000010a8c62da listextend + 666
70 libpython3.6m.dylib 0x000000010a8c7723 list_init + 163
71 libpython3.6m.dylib 0x000000010a8f7eb9 type_call + 313
72 libpython3.6m.dylib 0x000000010a89ae55 (null) + 197
72 libpython3.6m.dylib 0x000000010a89ae55 _PyObject_FastCallDict + 197
73 libpython3.6m.dylib 0x000000010a965828 call_function + 184
74 libpython3.6m.dylib 0x000000010a9620d2 (null) + 27362
74 libpython3.6m.dylib 0x000000010a9620d2 _PyEval_EvalFrameDefault + 27362
75 libpython3.6m.dylib 0x000000010a966b9d fast_function + 381
76 libpython3.6m.dylib 0x000000010a965901 call_function + 401
77 libpython3.6m.dylib 0x000000010a9620d2 (null) + 27362
77 libpython3.6m.dylib 0x000000010a9620d2 _PyEval_EvalFrameDefault + 27362
78 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
78 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
79 libpython3.6m.dylib 0x000000010a966c41 fast_function + 545
80 libpython3.6m.dylib 0x000000010a965901 call_function + 401
81 libpython3.6m.dylib 0x000000010a9620d2 (null) + 27362
81 libpython3.6m.dylib 0x000000010a9620d2 _PyEval_EvalFrameDefault + 27362
82 libpython3.6m.dylib 0x000000010a96638f (null) + 2447
82 libpython3.6m.dylib 0x000000010a96638f _PyEval_EvalCodeWithName + 2447
83 libpython3.6m.dylib 0x000000010a95b534 PyEval_EvalCode + 100
84 libpython3.6m.dylib 0x000000010a991511 PyRun_FileExFlags + 209
85 libpython3.6m.dylib 0x000000010a990cc3 PyRun_SimpleFileExFlags + 851
86 libpython3.6m.dylib 0x000000010a9a9c49 Py_Main + 3529
87 python 0x000000010a884df8 main + 232
88 libdyld.dylib 0x00007fff75dea145 start + 1
Segmentation fault: 11

[rbliu (Robert_Liu)]

Hi Jacqueline! As far as I can tell, I haven’t seen similar segfault on some older versions (DM stack version 13.0/12.0 with python2 on Mac or CentOS). If you see this traceback repeatedly, I guess there might be some processCcd-related methods incompatible with the python library being used. It might be helpful if you could also paste your installation environment here…

[jbosch (Jim Bosch)]

Yup, we do take segfaults in our code very seriously, but about 90% of the ones that are reported are actually due to inconsistent versions of packages being set up together. It’d be very helpful if you could try to repeat all of this starting from a new shell (while recording any EUPS setup commands so you can report them here). If the segfault happens again and we don’t see anything fishy in how in how you set up your environment, that’d be a fairly high-priority bug for us.

[jmac (Jacqueline McCleary)]

Hi, too, Robert!
Thanks for jumping in, @rbliu, @jbosch and @ctslater, I really appreciate your expertise. For the sake of other newcomers to community.lsst.org who might run into this issue, I am going to create a new thread called obs_decam+processCcd.py segfault and will link to it here shortly. I will follow the advice of setting up a brand-new shell and copying all setup commands.