I think the problem is mainly that ingestProcessed.py is still operating in the same way as the previous source ingestion scripts. That is, it outputs rows conforming to an externally defined schema using available metadata rather than using the available metadata to determine the output schema. As you say, the metadata and schemas are camera specific, so you end up with a bunch of IFs and potentially have to modify code every time you encounter a new camera.
My thoughts were that we would either:
- have pipeline runs output afw tables containing the desired metadata, and ingest exposure metadata by feeding these tables to the existing ingestCatalogTask
- or, write an ingestExposureTask which builds a schema based on the metadata present in FITS files. The task config probably needs to be able to explicitly list the interesting metadata keys or exclude undesirable keys.
In the latter case, I’d rather the exposure metadata ingestion script not be tasked with computing sky coordinates for the 4 corners of the exposure, or FWHMs, even though the code for that seems to be identical across all cameras supported thus far. I’d prefer to see that sort of thing done by the pipeline code.
Does that seem acceptable?