Combining data from two instruments with two obs packages

[raphaelshirley]

Hi,

I am developing an obs package to process VISTA VIRCAM imaging. I would like to produce joint HSC-VISTA catalogues and eventually LSST-VISTA by warping the images to the same sky map and merging the detection catalogues as is done for HSC bands in the pipelines examples. I am currently working on the VISTA only part as a test.

My question is would I need to, and is it possible to, use two obs packages simultaneously. Can I for instance import HSC images to the same Butler repository as my VISTA images using the obs_subaru package alongside my new obs_vista package? The command line tasks seem to assume the Butler repository is for one instrument but I wonder if I delve into the code directly I can use two.

Many thanks for any help or interest.

Best,

Raphael Shirley

(Postdoc at University of Southampton and junior associate of LSST UK)

[jbosch (Jim Bosch)]

Our current middleware (“Gen2”) does not support putting data from different obs packages in the same butler repository. There’s nothing fundamentally wrong with having algorithmic code that uses two butlers to access two different repositories, but none of our existing pipelines do, and it sounds like that’s what you’re really interested in.

The CLAUDS project (a group attached to the HSC collaboration) works around this for CFHT and HSC data (and maybe some IR data as well; I don’t recall exactly how they handle that) by making CFHT coadds look like HSC coadds, and merging repositories manually after coaddition.

We also have a new version of the middleware (Gen3) that we hope will replace Gen2 soon, and it will support multiple instruments in the same repository. It’s not yet feature complete and (more importantly) the data repository format is not yet stable, but for early-stage development work it may be a better choice. Introductory documentation for Gen3 is scattered and incomplete as well, but if you’re interested, I’d start with:

• the ci_hsc_gen2 and c_hsc_gen3 git packages include scripts to convert a Gen2 repository to Gen3 and bootstrap a new Gen3 repository, respectively, for HSC data;
• the Gen3 butler itself lives in the daf_butler package, and it already contains a lot of (especially reference) documentation, even if there are big gaps (https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/index.html);
• the obs package interface for Gen3 is defined by the lsst.obs.base.Instrument ABC. You can find example implementations (usually in an _instrument.py file) in obs_subaru, obs_decam, and obs_lsst.

You may note that I linked to weekly-build documentation for daf_butler, rather than the last release - that’s because Gen3 is still moving quite quickly, and I’d recommend anyone using it develop against our weekly releases to avoid falling too far behind.

[sophiereed]

Hello, I am interested in doing this as well. Let me know if I can help at all. I worked with Richard McMahon on VHS at Cambridge and now work for LSST DM so am fairly familiar with both datasets.

[raphaelshirley]

Hi,

Thank you very much for your reply. I will look into the CFHT method as we were aiming to warp the VISTA images to the HSC pixel base anyway. The other links are also helpful.

Best,

Raphael.

[raphaelshirley]

Hello,

Ok great, perhaps we could have a telecon at some point. I have just started a postdoc with Manda Banerji and we are also working with Richard McMahon on this.

Best,

Raphael.

[sophiereed]

Say hi to Manda for me!

[raphaelshirley]

FYI I am developing the obs_vista package here:

raphaelshirley/obs_vista