Verification Datasets Meeting 2016-07-20 Minutes

Data Management
#1

Verification Datasets telecon
Attendees: Simon, David, Colin, Zeljko

Colin:

  • new Horizons data (Pluto) with HSC, deep in the bulge
  • horrific image, packed with stars
  • slow processCcd problems
  • much of the code makes assumptions about the number of sources, and there are performance hits
  • matcher having issues, lots of loops
  • 53 min, 30 min were in the matcher
  • Colin used a k-d tree of the sources to help with the matching, matching now ~1 min.
    afw.match.GaussianProcess
  • also set hard upper limit on the number of PSF stars (~100), it was trying to use ~2000
  • now down to ~11 min
  • one of these images could be used to test the runtime of a package, make sure no changes
    dramatically affect the runtime
  • deblending turned on but hasn’t characterized it yet
  • more saturated stars
  • it was trying to use the brightest stars in the frame for matching but
  • using PanSTARRS as reference catalog
  • maybe add a rough zeropoint to each obs package that can help figure out the
    saturation limit for each visit
    Slater_bulge_image.png1958Ă—962 551 KB

Simon:

  • people at DESC meeting right now
  • Rahul Biswas and Bryce Kallenbach (at UW) have been trying to compare input SNE and AGN lightcurves
    and comparing to what comes out of the stack
  • finding systematically lower values of flux (a little bit) than predicted by
    the truth values, less than a ~1% or so
  • could be aperture correction or something like that
  • u-band is complete garbage, off by ~4x, not sure what’s wrong
    https://github.com/DarkEnergyScienceCollaboration/Monitor/issues/29
  • issues with blended objects, no deblending in forced photometry, multifit is supposed to fix this
    but it doesn’t allow amplitudes to vary per epoch (wihin a given band)
    the way to do the lightcurves is to do forced photometry on difference images (also no deblending),
    but if both are varying then you are hosed
  • run 2 between now and Sep., a few blockers, need to be fixed first
    • want to be able to choose the data based on some quality cuts
      -not clear how this will work, Simon added some things to LDM-151 that we need tools for “data discovery”
      in software primatives (9.22)
    • need a PSF homogenized template, PSF footprint size is allowed to vary which can cause problems
    • some rolloff on the edges which are causing lots of false positives on the edges, because not flatfielding
      sims specific issue
  • want to add difference imaging

David:

  • working on rerunning the COSMOS data with PSFEx
1 Like
#2

This is most likely aperture corrections. Simulations tend to insert sources with the flux levels set for an infinite radius. However, the stack corrects everything to a 12 pixel aperture (slot_calibFlux, which is base_CircularApertureFlux_12_0). The difference (the flux from 12 pixels to infinity) doesn’t really matter for most applications.

Have you tried using the multiband processing scheme (either multibandDriver.py from pipe_drivers or detectCoaddSources.py + mergeCoaddDetections.py + measureCoaddSources.py + mergeCoaddMeasurements.py + forcedPhotCoadd.py from pipe_tasks)?

#3

Which data? The input data to be processed, or the output data? In the latter case, I think the expectation is that we would ingest into a database and use SQL joins with the exposure metadata table to do cuts. However: while loading Twinkles into the Prototype Data Access Center is desirable, it may not align well in terms of schedule with DESC’s needs.

#4

Yes. That’s how we’ve been doing things. When SNe land on galaxies, we don’t recover the input lightcurves for the SNe very well. See examples here: https://github.com/DarkEnergyScienceCollaboration/Monitor/issues/29.

#5

It’s the output data.

I agree that the way to do this in production is to load the exposure info into a database. The problem is that we don’t currently make it easy to do this. If we can load these data into the PDAC, I’m all for it. What would be the timeline for that?

#6

@rbiswas gave us a talk in the sims telecon this morning about this as well. He said the other issue (perhaps more so than the slightly lower flux?) was that the uncertainty reported by the DM pipeline was smaller than expected. It may actually be related to the same effect … if it’s bright, the uncertainty should be low, but then the offset could be coming from the aperture correction.

In a side note - understanding what DM and SQuARE will need for reference catalogs and doing the comparison between those reference catalogs and the DM measurements (especially if there are any alterations that we should put into the sims reference catalogs) is something sims is very interested in understanding.

#7

By how much, and at what stage? If it’s on the coadds, that’s not at all surprising because variance has been lost as covariance which we don’t track. If it’s elsewhere, it might indicate that the variance plane isn’t being set correctly.

#8

You’d have to talk to @rbiswas or someone else who is in on the twinkles analysis for more details I’m afraid. I don’t think it’s coadds, because it’s measurements that were going into a lightcurve.