Correct use of ingestImages.py for calibrations

Support
#1

I’ve been struggling to figure out the correct use of ingestImages.py to ingest HSC calibration data. I have raw images in $CI_HSC_DIR/raw and the ingest works perfectly with the command listed in the tutorial.

I have flats and biases in $CI_HSC_DIR/cals, but not sure what flags to provide for ingest to create the CALIBS directory. Obviously run as

ingestImages.py DATA $CI_HSC_DIR/cals/*.fits --mode=link

is wrong because processCcd.py throws the warning: HscMapper WARN: Unable to find calib root directory before failing catastrophically.

Thanks for the help!

#2

I think you want ingestCalibs.py, if the flats and biases are “master” coadded images produced by the HSC/LSST code and not “raw” telescope images, but an HSC expert should probably weigh in here.

#3

Thanks @ktl

What I have are the raw calib frames and not masters. As it is, I need the code to produce those masters, and I can’t ingest the frames in the first place.

#4

If these are raw calibration frames, then they should indeed be ingested with ingestImages. I’d have to see what the actual error message is; the WARN you mention is exactly that: merely a warning and not an actual problem.

Oh: the error is coming from processCcd? Then yes, you need to run master calibration generation code based on your raw calibrations (or get masters from somewhere else) and ingest them with ingestCalibs. Doing that calibration processing is far beyond me :slight_smile:

#5

I see. Thanks. Yes I only have the raw calibration frames. I know Gemini automatically creates calibration masters on a ~monthly basis, but I am unaware of any service that would provide something different for the HSC.

FWIW I tried ingestCalibs on the raw frames, with no luck. The following error is thrown, suggesting that it is expecting fits files of a different format (data arrays in different locations in the fits file).

(lsst-scipipe-4d7b902) ubuntu@lsst-test:/media/NHproc$ ingestCalibs.py DATA $CI_HSC_DIR/CALIB/smoka/P*_data/*fits --mode=link --validity 60
root INFO: Loading config overrride file '/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/19.0.0+2/config/ingestCalibs.py'
HscMapper WARN: Unable to find calib root directory
CameraMapper INFO: Loading exposure registry from /media/NHproc/DATA/registry.sqlite3
Traceback (most recent call last):
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/ingestCalibs.py", line 3, in <module>
    IngestCalibsTask.parseAndRun()
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingest.py", line 425, in parseAndRun
    task.run(args)
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingestCalibs.py", line 220, in run
    with self.register.openRegistry(calibRoot, create=args.create, dryrun=args.dryrun) as registry:
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingestCalibs.py", line 90, in openRegistry
    return RegisterTask.openRegistry(self, directory, create, dryrun, name)
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingest.py", line 294, in openRegistry
    registryName = os.path.join(directory, name)
  File "/home/ubuntu/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
#6

Yes, ingestCalibs on the raw frames definitely won’t work properly. It fails in your case because the CALIB directory doesn’t exist. If you created it manually, it might appear to work but not actually do the right thing, or it might fail. In any case, apologies for the inscrutable error messages.

1 Like
#7

When were your science frames taken? I might have some pre-built calibs you could use without having to worry about building them yourself.

#8

The data I am currently playing with are June 2014, but we have multiple upcoming runs that I am trying to get ready for aka. familiarize myself with all parts of the pipeline.

@price Paul, do you have a machine automatically producing flats, biases, and such?

#9

I can definitely cover June 2014 with existing calibs if you like, but it sounds like you want to learn. Which tutorial are you following?

I don’t have anything routinely generating calibs. I usually just put in a few days worth of effort once a year or two.

#10

You might want to look at the log I kept from creating calibs earlier this year: http://tigress-web.princeton.edu/~pprice/CALIB-20200115/calib-20200115.txt

It’s probably a larger scale production than you care about, but hopefully there will be some helpful guidance in there.