Generation 3 run (LSST science pipeline tutorial) using (w_2021_34)

---

Contents

---

• Useful links
• Setting up butler repository
  • Creating a butler repository
  • Registering a instrument to the butler repository
  • Importing the refcats into the butler repository
  • Ingesting the raws in the butler repository
  • Defining the visits
  • Writing the curated calibrations
• Processing the LSST pipeline tutorial data
  • ProcessCcd
  • Coaddtions
  • Multiband

Useful links

---

• https://github.com/lsst/pipe_tasks/blob/master/pipelines/DRP.yaml [1]
• https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/scripts/butler.html [2]
• https://github.com/lsst [3]
• https://community.lsst.org/t/generation-3-butler-tutorial/4965/8 [4]

---

Setting up butler repository

---

Creating a butler repository

• Note: the butler create command line task creates a directory and populates it with a butler.yaml file (base config) and a registry (sqlite3 file).

! butler create GEN3_run

Registering a instrument to the butler repository

! butler register-instrument GEN3_run/ lsst.obs.subaru.HyperSuprimeCam

Importing the refcats into the butler repository

• I used a hack to get the refcats into this butler repository

  • I used the butler convert command line task to convert the gen 2 repository (LSST Science pipeline tutorial) to a gen 3 repository
  • Then I used the butler import command link task to import into the new butler repository (GEN3_run)

• Question - Is there a way of ingestiing the refcats in a gen 3 format ??

! butler import GEN3_run/ ~/lsst_stack/DATA_gen3/ --export-file exports.yaml

Ingesting the raws in the butler repository

! butler ingest-raws GEN3_run/ DATA_gen3/HSC/raw/

Defining the visits

! butler define-visits GEN3_run/ HSC 

Writing the curated calibrations

!butler write-curated-calibrations GEN3_run/ HSC

Processing the LSST pipeline tutorial data

• I'm using the latest version on the pipe_task file"DRP.yaml" thats found on github [1]

---

ProcessCcd

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/characterizeImage.py
  • https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/isrTask.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/" and
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/")
  
  

pipetask run  -b ~/LSST-RAL-ECHO-EXP/lsst/GEN3_run/ --input HSC/raw/all,refcats,HSC/calib --register-dataset-types 
    -p "${PIPE_TASKS_DIR}/pipelines/DRP.yaml"#processCcd --instrument lsst.obs.subaru.HyperSuprimeCam 
    --output-run processCcdOutputs  -c isr:doBias=False -c isr:doBrighterFatter=False -c isr:doDark=False -c 
    isr:doFlat=False -c isr:doDefect=False 

• The the processccd outputs directory structure

Coaddtions

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/healSparseMapping.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/"
  
  

Before you can run the coaddtions pipe task you have to run the make-discrete-skymap command line task

• https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/script/makeDiscreteSkyMap.py

! butler make-discrete-skymap test1/GEN3_run HSC 

coaddtions pipetask

! pipetask run  -b GEN3_run/ --input processCcdOutputs --input skymaps  --register-dataset-types 
    -p "${PIPE_TASKS_DIR}/pipelines/DRP.yaml"#coaddition
    --instrument lsst.obs.subaru.HyperSuprimeCam --output-run coadd -c makeWarp:doApplySkyCorr=False -c
    makeWarp:doApplyExternalSkyWcs=False -c makeWarp:doApplyExternalPhotoCalib=False 
    -c assembleCoadd:doMaskBrightObjects=False

• The skymap command creates this directory within the butler repository
  
  

• The coaddtions directory structure looks like this

• Very similar to the gen 2 repository which was converted into gen3. (assuming that the name convention of files has cahnge from gen2 to gen3 format)

Multiband

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/multiBand.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeDetections.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/deblendCoaddSourcesPipeline.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeMeasurements.py
  • https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/" and
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/meas_base/21.0.0+087873070b/python/lsst/meas/base/")
  
  

pipetask run  -b ~/LSST-RAL-ECHO-EXP/lsst/GEN3_run/ --input processCcdOutputs --input  HSC/raw/all,refcats,HSC/calib 
  --input skymaps --input coadd --register-dataset-types --input  HSC/raw/all,refcats,HSC/calib  
  -p "${PIPE_TASKS_DIR}/pipelines/DRP.yaml"#multiband 
  --instrument lsst.obs.subaru.HyperSuprimeCam --output-run multiband

Removing the BRIGHT_OBJECT in the obs_subaru configs

• cd ~/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/obs_subaru/22.0.1-20-g904645ea+7c6b33a4e9/config/
  
• Within this repository there is the "measureCoaddSources.py" and "forcedPhotCoadd.py"
  
• Comment out these lines in both of the scripts above:
  
  config.measurement.plugins['base_PixelFlags'].masksFpCenter.append('BRIGHT_OBJECT') ()
  config.measurement.plugins['base_PixelFlags'].masksFpAnywhere.append('BRIGHT_OBJECT')
  

• With the measureCoaddSource.py comment out lines 23 and 24
  
  

• With the forcedPhotCoadd.py comment out lines 13 and 14
  
  

Potential issue 1

• An error with the psfmatch on the forcedPhotCoadd pipetask

Potential issue 2

• exposureidinfo - This is not used in the forcedPhotCoadd.py and as a result i get error like this " "Exposure ID "8" is too large"
  
  

• I got this similar error when i was editing v21 in the mergeDetections task
  
  

Temporary fix 2

• I used the exposureidinfo for idfactory instead of lsst.afw.table.IdFactory.makeSource
  
  

• line 245 and 246 is what i did to fix the error as well as importing exposureidfinfo
  
  

• I'm going to edit line 243 to "expId= exposureIdInfo.expId" for consistency
  
  

• The coaddtions directory structure looks like this