CFHT Megacam data

fraserw · December 11, 2019, 11:53pm

Hi Community Support

I am interested in pushing some CFHT Megacam data through the LSST pipeline. More specifically, I wish to astrometrically calibrate a handful of images against PS1-DR2, warp them into tracts and associated patches, and produce a mean template from a subset of the images.

I do have v19 of the pipeline setup and successfully tested.

A bit more info - I have images in raw format, as well as already detrended and calibrated against GAIA with Stephen Gwyn’s megapipe system (good per-chip photometric calibration, and astrometric calibration using the PV formalism). Can start with raw data if necessary.

Unfortunately the tutorial fails at the processCcd step so haven’t learned much from that. So anyways, not entirely sure where to begin.

If anyone has some suggested reading, or can help me out a bit, that would be very much appreciated!

Thanks

Wes

KSK · December 12, 2019, 5:24pm

Can you post the log from the run that fails? We run CFHT regularly in the continuous integration system, so it’s certainly possible.

fraserw · December 12, 2019, 5:42pm

Hi Simon. Thanks for the response.

It’s not that something fails with the CFHT imagery, it’s that I don’t even know where to begin. I can’t use the tutorial as a starting place, because it is broken, and don’t know where else to turn to. @price suggested I raise a support post here.

price · December 12, 2019, 6:28pm

It would be helpful if you could post the output demonstrating the particular problem you’re having, so we can attempt to reproduce the problem.

fraserw · December 16, 2019, 8:38pm

Well after some trial and error, and a few notes from Dominique, I have at least been able to ingest some megacam data. MegacamMapper doesn’t seem to recognize the gri filter, but the real problem I am running into is with a missing defects file for the camera during processCcd.py step. I can see that I am supposed to ingest that file, but my google-fu hasn’t returned the location of that file. Anyone have it?

I have copied the command and error output below if useful.

Thanks for the help!

Wes

(lsst-scipipe-4d7b902) ubuntu@lsst-test:/mnt$ processCcd.py DATA_CFHT --rerun processCcdOutputs --id
root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_cfht/19.0.0+2/config/processCcd.py’
CameraMapper INFO: Loading exposure registry from /mnt/DATA_CFHT/registry.sqlite3
root INFO: Running: /home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/processCcd.py DATA_CFHT --rerun processCcdOutputs --id
WARNING: You are using OpenBLAS with multiple threads (4), but have not
specified the number of threads using one of the OpenBLAS environment variables:
OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
This may indicate that you are unintentionally using multiple threads, which may
cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
what you are doing and want threads enabled implicitly, set the environment
variable LSST_ALLOW_IMPLICIT_THREADS.
processCcd INFO: Processing {‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘filter’: ‘r2’, ‘visit’: 2426913, ‘state’: ‘p’, ‘extension’: 1, ‘ccd’: 0, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}
processCcd.isr INFO: Performing ISR on sensor {‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘filter’: ‘r2’, ‘visit’: 2426913, ‘state’: ‘p’, ‘extension’: 1, ‘ccd’: 0, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}.
daf.persistence.butler WARN: butlerLocationExists for non-supported storage None at None(/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_cfht/19.0.0+2/policy/…/megacam/defects/test.mask.0.40.01.nn/ccd00.fits)
daf.persistence.butler WARN: butlerLocationExists for non-supported storage None at None(/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_cfht/19.0.0+2/policy/…/megacam/defects/test.mask.0.40.01.nn/ccd00.fits)
processCcd FATAL: Failed on dataId={‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘filter’: ‘r2’, ‘visit’: 2426913, ‘state’: ‘p’, ‘extension’: 1, ‘ccd’: 0, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: NoResults: No locations for get: datasetType:defects dataId:DataId(initialdata={‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘filter’: ‘r2’, ‘visit’: 2426913, ‘state’: ‘p’, ‘extension’: 1, ‘ccd’: 0, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}, tag=set())
Traceback (most recent call last):
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in call
result = self.runTask(task, dataRef, kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/processCcd.py”, line 184, in runDataRef
exposure = self.isr.runDataRef(sensorRef).exposure
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_isr/19.0.0/python/lsst/ip/isr/isrTask.py”, line 1466, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_isr/19.0.0/python/lsst/ip/isr/isrTask.py”, line 926, in readIsrData
if self.config.doDefect else None)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/butlerSubset.py”, line 206, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/butler.py”, line 1380, in get
raise NoResults(“No locations for get:”, datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:defects dataId:DataId(initialdata={‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘filter’: ‘r2’, ‘visit’: 2426913, ‘state’: ‘p’, ‘extension’: 1, ‘ccd’: 0, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}, tag=set())