Run ap_pipe on existing calexps/wcs

[racine (Racine Benjamin)]

Hi all,

I would like to run ap_pipe.py using data that were already processed with singleFrameDriver.py, jointcal.py and coaddDriver.py.
I ran jointcal.py night by night since these are nights 1 year appart from each others, and so far proper motion is not handled.
I then use the wcs produced by jointcal to make night-by-night coadds.

So far I have managed to point to the template of night_1 with ap_pipe using the -- template option, to run DIA on the visits of night_2.
It works fine, but ap_pipe re-processes the ‘ccdProcessing’ for night_2, which is time consuming and unnecessary since I already have these done.
I tried using --reuse-outputs-from ccdProcessor, but I think this is only able to reuse the ouputs of an earlier run of ap_pipe, and doesn’t look into the input path. I guess I could link the calexps into the output path, but that is not ideal.

Is there a proper way of doing this?

And also, since I already have improved wcs for these night_2 visits, that jointcal generated. Is there a way to use them?

So technically I would like something like
ap_pipe.py Input_path --calib calib_path --template template_night1_path **--calexp calexp_night2_path --wcs wcs_night2_path** --output output_path -c diaPipe.apdb.db_url="sqlite:////link/to/association.db" -c diaPipe.apdb.isolation_level="READ_UNCOMMITTED" -c diaPipe.apdb.connection_timeout=240 --id visit=visitlist_night2

Thanks !

Ben

[racine (Racine Benjamin)]

Hi again,

I managed to go around reprocessing calexps by linking the outputs of singleFrameDriver in the output of ap_pipe before running.
It’s not ideal, but it works…
The problem with this is that I still don’t have an option to use the wcs from jointcal.

As an alternative I was wondering if it is possible to use a ref catalog coming from jointcal instead of panstarrs. That would have better astrometry than panstarrs when reprocessing the calexp, and the wcs created in that process would be kind of equivalent to the one returned by jointcal.

As an additional follow up, I was wondering if there is a way to turn off the alert production. So far, it created files, like alerts/*.avro that are really heavy. I don’t need them in my current analysis, so I would love to have an option to turn it off.
I see in the config file that there is config.diaPipe.alertPackager.alertWriteLocation but that is only a way to decide where to output, I don’t see a way to turn it off…

Best,

Ben

[mrawls (Meredith Rawls)]

I believe the only way for ap_pipe to use existing calexps is if they already exist in the output repository. If someone else knows better, please do chime in!

I don’t think there is a way to use jointcal WCS data products either, since ap_pipe is designed to run in “real time” and there will never be some external pre-existing better WCS.

Finally, as best I can tell, the “automatically write a boatload of humongous alerts” behavior is a quirk of release v20. Persisting alerts now defaults to False in the latest weekly build of diaPipe, but unfortunately that option didn’t exist yet in v20. You’ll either need to use a more recent ap_pipe or just delete them after the fact.

[parejkoj (John Parejko)]

As Meredith says, ap_pipe is not designed to take external WCS information, because that data will not be available to our alert pipeline in realtime processing of LSST camera data. ap_pipe is a full pipeline for taking raw images as input and producing the full subtracted output. ImageDifferenceTask is probably the part of this pipeline that you are looking for, and it takes the calexp to be differenced, so would not re-do the image processing.

A quick scan suggests that ImageDifferenceTask itself does not have an option for an external WCS. This is something that we may want to use in future DRP processing, but it has is not a priority at the current time.

@racine

As an alternative I was wondering if it is possible to use a ref catalog coming from jointcal instead of panstarrs. That would have better astrometry than panstarrs when reprocessing the calexp, and the wcs created in that process would be kind of equivalent to the one returned by jointcal .

It is certainly possible to take an LSST-style output catalog and covert it to an LSST-style refcat, but I’m not aware of us having any current code to do that. You’d want to do it on a catalog produced from a coadded image to get the most depth. lsst.afw.table.updateSourceCoords() will do an in-place update of the ra/dec coordinates of an in-memory SourceCatalog using a SkyWcs (like that produced by jointcal), if you want to experiment with this on single images.