Extension of the standardized curated calibrations system

Data Management
#1

In June, I presented the new defects handling in the following post: Changes to how defects are handled: implementation of RFC-595.

With DM-21155 landing today, there are a few refinements to the way curated calibrations are handled in general. I have generalized the calibration ingest script so that it can handle anything that:

  1. has the correct metadata as described in the previous post
  2. can be read by astropy.table.Table.read()

The result is that I renamed the ingest script and the task it calls. Now the task is IngestCuratedCalibsTask as opposed to IngestDefectsTask and the command line script is ingestCuratedCalibs.py versus ingestDefects.py.

The other (much less user visible) change is that now the function that reads the files form the obs_*_data repositories has changed from read_all_defects to just read_all and now returns a dictionary of the calibration objects and the calibration type as a tuple where before it returned just a dictionary of Defects objects.

I think these changes should make it easier to add new calibration types in the future.

2 Likes
Changes to how defects are handled: implementation of RFC-595
Gen3 Instruments, registries, and ingestion
#2

Simon, this sounds awesome!

Calibration types that I think should be handled using this new system:

  • Gains
  • Read noise
  • Linearity models
  • Defects
  • QE curves
  • Crosstalk matrices
  • Brighter-fatter kernels

Others will likely spring to mind, but I think that’s a start. If people agree/disagree with any of those, or want to extend that list, I think this would be good place to do so.

#3

@merlin that’s a good list. Currently gains and read noise are part of the camera geometry description, and that can’t fit directly into this system since the ingest code needs a camera to begin with. We could think about factoring out table like data from the camera geometry yaml, if we want to version that separately, but we will need to solve the versioned camera problem sooner or later as well.

#4

What’s the impact of factoring out gain and read noise into new standalone classes outside of cameraGeom?

#5

The main impact is that we would have to refactor YamlCamera to reflect the different source of information for those two quantities. The benefit would be that we could version the two independently, but I don’t know if that’s even desirable.

#6

I really meant “impact” as in whether we would treat gain/read noise as new distinct calibration classes like QE curves and remove them from cameraGeom completely (not separate them but still access them through cameraGeom). Don’t suppose we can rename cameraGeom to be something that doesn’t imply it’s only defining camera geometry?

#7

I see. Separating them completely would require a significant change in how we access these quantities now. Though they are not used by a large amount of pipelines code. It may not be too hard to do.