Questions on the data contents of the alerts - DIAForcedSources in particular

Data Management
#1

To inform our future Lasair Database, we’ve been looking more closely now at the contents of the alert packets, and the Prompt Product Database. I have some questions arising from this - I’m working off LSE-163 Data Products and Definitions, Juric et al. 2019-07-29.. We also need to figure out what we can and can’t release as public data.

The alert packets will contain the median noise of difference images covering that piece of sky for 12 months (providing the history upper limits on assumption no detections). OK, so that information is all in the public domain.

Within 24hrs, the precovery PSF forced photometry for images taken within the last 30 days will be added to the DIAForcedSources table (in the Prompt Product Database). OK. That information is not public.

The DPD doc also says that DIAForcedSource measurements will not be immediately issued (as above), but then there is a statement saying “the resulting DIAForcedSource measurements will be included in future alerts from the DIAObject”. Which would imply that when a second alert is issued based on the detection of the next data point on the lightcurve (for that DIAObject) then the full DIAForcedSources WILL be released, and therefore be public. Correct ?

If this is true, will the 2nd (and all subsequent) alerts associated with this DIAObject include both the 12 month image noise measuremetns AND the 30 day forced photometry ?

More questions to come, but let’s take it one at a time !

2 Likes
#2

The best source of information on data rights issues is not the LSE-163, but rather the draft version of LDO-13, the LSST Data Policy which @rblum5 posted a couple of months ago. In particular, note DPOL-504:

The image differencing source (DIASource), forced photometry (DIAForcedSource), and object (DIAObject and SSObject) records produced in Prompt Processing are public. Access to these catalogs held at the LSST DACs will be restricted to LSST Users, ​but the information in the Prompt Products Database (PPDB) tables listed here in DPOL-504 can be shared by broker teams or others with access rights to the LSST DACs

For how long will forced photometry be run on DIAObjects, after discovery?
#3

Thanks, John. Now clear that DIAForcedSource table data is allowed (by policy) to be distributed publicly by brokers.

But do you or others know the answer to the other two questions (data management rather than policy) - will DIAForcedSource measurements be included in the second alert (and all subsequent alert packets) of an object, assuming that second alert is >24hrs from the first ?
LSE-163 seems to say this, but I’m looking for confirmation.

Also see question on both the 12 months history and 30 day forced photometry information.

#4

That is my understanding. Note in particular §3.2.1 point 11: “Alerts will not be issued with precovery photometry information but the resulting DIAForcedSource measurements will be included in future alerts from this DIAObject”.

I believe this is answered in §3.2.1 point 7: “For each visit in the last twelve months at the position of the DIAObject, if there are no DIASource or DIAForcedSource records for that visit, the time of that visit, the filter, and the difference image noise are included in the alert packet”; that is, the image noise measurement will only be included if there is no associated DIA(Forced)Source record for that image.

I would defer to @ebellm on both of the above, though.

#5

@ebellm @mjuric can you confirm :

when a second alert is issued for a DIAObject, based on the detection of the next data point on the lightcurve (for that DIAObject) then the full DIAForcedSources (for 30 days) be released, immediatelym within the alert packet.

and

will this 2nd (and all subsequent) alerts associated with this DIAObject include both the 12 month image noise measurements AND the 30 day forced photometry ?

We would like both (for cross-check) i.e. not either or.

#6

The current plan (as @swinbank correctly described) is that historical image noise measurements will only be included when a DIASource and/or DIAForcedSource measurement is not available. We will include any DIAForcedSource measurements we have when issuing subsequent alerts, as the forced photometry measurements are of course more informative.

Including both is of course possible, but it makes the alert packets yet larger. Is there a reason you need this information in real time? Would it be sufficient to query the PPDB for cross-checks (e.g., of a subset of objects, or offline verification)?

Note that the image noise is an average over the CCD–it’s not measured at the position of the DIAObject. It’s intended to provide rough upper limits prior to the precovery Forced Photometry runs.

1 Like
#7

thanks @ebellm - and now I realise this is perfect. The first alert has the 12 month historical noise measurements, the second has the 30 day DIAForcedSource. All of which we will cache in the broker, and have all info available to users. Useful to have both 12 month and 30 day info (even if the 12 month is the cruder CCD average noise). And both in public domain, so no issues of relaying.

2 Likes