Pixel noise estimate for footprint threshold

Data Management
#1

I have a question concerning the footprint threshold used for source detection in the Pipelines. For context, a SourceDetectionTask constructs footprints from an exposure via the detectFootprints method. Using all default settings, by my understanding, that method broadly works as follows: It smooths the background-subtracted exposure by convolving it with the PSF, estimates an average per-pixel noise level, multiplies that by 5 to establish a “5-sigma” threshold, and creates an initial set of footprints by selecting pixels in the smoothed image with an instrumental flux above that threshold value. These initial footprints are later expanded (and merged as necessary) to form final footprints. A similar strategy is used by DynamicDetectionTask, with an extra correction factor that incorporates estimated PSF characteristics. This corresponds to the general strategy outlined in doi:10.1093/pasj/psx080.

The Pipelines seem to diverge from that strategy, however, in the specific method used to estimate the average noise level. In both versions of detectFootprints, the smoothed image is passed to applyThreshold, which in turn passes it to makeThreshold, which ultimately calls lsst.afw.math.makeStatistics to compute a clipped standard deviation of the values stored in each pixel. But since the smoothed image is used, the standard deviation it finds is several times smaller than what it would be if the original image were used. My understanding of section 4.7.1 of doi:10.1093/pasj/psx080 is that the proper value of sigma to use here is the per-pixel noise in the original, un-smoothed image.

I haven’t found any documentation to suggest that using the noise level of the smoothed image is preferable over that of the original image. What is the rationale for this choice?

#2

We’re not looking for single pixels above a pixel-to-pixel threshold, but entire PSFs above a PSF threshold. Measuring the noise on the PSF-convolved image reflects this. That noise level is smaller than the pixel-to-pixel noise on the original image, but we’re searching for PSFs, not individual pixels: when you know what the signal should look like, you can go below the pixel-to-pixel noise. If you’re not convinced, try some simulations with a GaussianPsf.

This discussion is only relevant for thresholdType="stdev" (the default) and everything assumes that there’s no covariance in the original image, which is not the case for coadds (or anything that has been resampled; this is why we use the DynamicDetectionTask in those cases) and the variance is relatively constant over the original image. If the variance varies over the image, use thresholdType="pixel_stdev" but in that case you’ve got to be sure to set the variance plane accurately.

Section 5.2 in the infamous Photo unpaper covers detection, but with a broad brush. I think @RHL may have a better description in some of the talks he’s given over the years?

1 Like
#3

Thank you for the recap. I realize now that the root of my misunderstanding was a silly algebra error. My understanding now is that the noise on the PSF-convolved image is mathematically equal (assuming nice, locally constant noise) to the noise on the original image times the square root of the effective area of the PSF, which is exactly the denominator in eq. 10 of doi:10.1093/pasj/psx080. With this, the Pipeline method makes sense.

I originally suspected this might be the case, but a) I made a mistake when I first tried to prove it algebraically, and b) when I attempted to check this equality in a DC2 coadd, I saw that the clipped standard deviation of the original image * sqrt(effective area) was only two-thirds as large as the clipped standard deviation of the PSF-convolved image. But of course it’s nontrivial to do a truly apples-to-apples comparison of the two in a busy image.