Map_raw_filename error when trying to ingest files for my own obs_package

Support
#1

Hello,
I’m trying to develop an obs_package for our CTMO observatory (repo here) trying to emulate, necam example, VISTA and others. It seems the installation works ok and now I am trying to see if I can make butler ingest some files to process.

There are a few things I don’t understand and I haven’t found any info anywhere. How does butler now what .fits is a dark, a bias, a flat, a light? Surely from header HDU cards, yet, I haven’t found what translation I have to provide for those. Also, does directory structure matter when ingesting? or does ingest.py scans the whole directory tree looking for calibration images?

Anyway, I get this long error, apparently related to my CamMapper subclass.

ingestImages.py DATA $CTMO_TEST_DIR/*.fit --mode=link
Downloading https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat
|==================================================| 1.3k/1.3k (100.00%)         0s
root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_ctmo/v1/config/ingest.py'
CameraMapper INFO: Loading exposure registry from /home/lsst/pipe/DATA/registry.sqlite3
ingest WARN: /home/lsst/pipe/CTMO_1_filter/*.fit doesn't match any file
ingest.register INFO: Table "raw" exists.  Skipping creation
(lsst-scipipe-1a1d771) [lsst@65fd9f18c3b8 pipe]$ ingestImages.py DATA $CTMO_TEST_DIR/lights/*.fit --mode=link
root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_ctmo/v1/config/ingest.py'
CameraMapper INFO: Loading exposure registry from /home/lsst/pipe/DATA/registry.sqlite3
ingest.register INFO: Table "raw" exists.  Skipping creation
ingest WARN: Failed to ingest file /home/lsst/pipe/CTMO_1_filter/lights/V_BX_Tri_lights-001_i.fit: 'CtmoCamMapper' object has no attribute 'map_raw_filename'
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/ingest.py", line 599, in run
    self.runFile(infile, registry, args, pos)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/ingest.py", line 576, in runFile
    outfile = self.parse.getDestination(args.butler, fileInfo, infile)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/ingest.py", line 189, in getDestination
    raw = butler.get("raw_filename", info)[0]
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/butler.py", line 1397, in get
    location = self._locate(datasetType, dataId, write=False)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/butler.py", line 1316, in _locate
    location = repoData.repo.map(datasetType, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/repository.py", line 239, in map
    loc = self._mapper.map(*args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/mapper.py", line 162, in map
    func = getattr(self, 'map_' + datasetType)
AttributeError: 'CtmoCamMapper' object has no attribute 'map_raw_filename'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/bin/ingestImages.py", line 3, in <module>
    IngestTask.parseAndRun()
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/ingest.py", line 405, in parseAndRun
    task.run(args)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/ingest.py", line 603, in run
    raise IngestError(f"Failed to ingest file {infile}", infile, pos) from exc
lsst.pipe.tasks.ingest.IngestError: Failed to ingest file /home/lsst/pipe/CTMO_1_filter/lights/V_BX_Tri_lights-001_i.fit

I appreciate if anyone could give me a hand pointing out what I’m missing from my understanding.

Thanks.

#2

I think you will likely be better off developing a pure-“Gen3” obs_ctmo package rather than trying to get the Gen2 mapper/policy working properly, even if that means waiting a couple of weeks until Gen3 Butler is officially certified for internal Pipelines use.

This particular error stems from the raw dataset type not being defined within an exposures section in the YAML policy file.

1 Like
#3

Thank you for your quick response. I will try to fix it. I have read here and there about gen2 and gen3 but couldn’t find any official documents about it. Will there be any official channel for the announcement?

I’m working now on a docker image lsstsqre/centos:7-stack-lsst_distrib-v20_0_0. How can I know which release will contain gen3 butler?

Thanks again.
Martin.

#4

obs_necam now supports gen3, and can be run in v20. gen3 employs a completely different means of executing commands - see the demos in obs_necam/data/bin/ .

To give full disclosure: obs_necam is not part of the LSST stack proper and was not written by the DM team. I say that not to take credit from them (indeed, their advice helped a lot when writing it), but rather to protect them from blame. :slight_smile:

3 Likes
#5

Thank you James, your guides have been very useful so far, I’m so glad I found them.

Of course, I wished there were some official place to learn a bit more about pipeline releases.