I took a look at using the tasks that ProcessFile uses to reduce some data in a workshop at LSST@Europe2 but I found that there were a number of changes that contravened the rules for using Tasks. Clearly putting ProcessFile under CI would have caught this, but we really shouldn’t have made these changes in the first place. Maybe the confusion is that very few cmdLineTasks are only used from the command line, so the no-butler rules apply to all tasks — they may add methods to allow their use from the command line, but they must still work directly from python.
With some pretty-trivial changes everything worked nicely (those are the workbooks I mentioned in https://discourse-dev.lsst.codes/t/notebooks-for-lsst-europe2-in-belgrade).
In more detail:
-
CalibrateTask.run()
is no longer an interface that can be used to build pipelines – it takes a dataId-related ExposureIdInfo -
CalibrateTask.calibrate()
cannot be used either as it takes an ExposureIdInfo -
CharacterizeImageTask.__init__()
has a compulsory butler object
I can work around this (import ExposureIdInfo and build one with meaningless values; pass None to CharacterizeImageTask).
My reading of https://jira.lsstcorp.org/browse/RFC-173 did not permit “functional” tasks to depend on anything to do with butlers or dataIds, although I (now) note that Russell said that he was going to add this butler argument in an RFC-173 comment so I should have known this. The short-term fix is to default these objects to be None (e.g. https://jira.lsstcorp.org/browse/DM-6631) and maybe that’s OK.
Additionally, CalibrateTask assumes that a Psf is present, and does not provide a way to pass sigma through to the detection task. This implies that you /must/ run CharacterizeImageTask first, or add a fake Psf to the Exposure. I don’t like this, and at the very least it should be documented as a pre-condition for the task.
Finally, the packaging is a bit odd:
from lsst.pipe.tasks.characterizeImage import CharacterizeImageTask
from lsst.meas.algorithms.detection import SourceDetectionTask
from lsst.meas.deblender import SourceDeblendTask
from lsst.meas.base import SingleFrameMeasurementTask
Why is CharacterizeImageTask in pipe_tasks but the others in meas? Should it be in meas_base? I don’t know the correct answer here, but surely we can come up with a consistent solution.
R