Magnitude column's name in the fake catalog used with processwitfakesDriver

Support Rubin Science Platform
#1

Hello all,
I am creating a csv catalog file to be used with processwitfakesDriver. I am using Z magnitudes from the coadd catalog. The name for the magnitude column in the csv file is zmagVar. I then use this catalog with processwithfakesdriver to insert fakes in visits in different filters: SC-G, Y, I2, … With this, I got the following error when running the processwithfakesdriver on a visit in G filter:

*KeyError: 'gmagVar'*
*Traceback (most recent call last):*
*  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-1a1d771/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4410, in get_value*
*    return libindex.get_value_at(s, key)*
*  File "pandas/_libs/index.pyx", line 44, in pandas._libs.index.get_value_at*
*  File "pandas/_libs/index.pyx", line 45, in pandas._libs.index.get_value_at*
*  File "pandas/_libs/util.pxd", line 98, in pandas._libs.util.get_value_at*
*  File "pandas/_libs/util.pxd", line 83, in pandas._libs.util.validate_indexer*
*TypeError: 'str' object cannot be interpreted as an integer*
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_base/20.0.0/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_base/20.0.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_drivers/20.0.0/python/lsst/pipe/drivers/processCcdWithFakesDriver.py", line 70, in runDataRef
    result = self.processCcdWithFakes.runDataRef(sensorRef)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/processCcdWithFakes.py", line 272, in runDataRef
    sfdSourceCat=sfdSourceCat)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/processCcdWithFakes.py", line 351, in run
    self.insertFakes.run(fakeCat, exposure, wcs, photoCalib)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/insertFakes.py", line 318, in run
    image = self.addFakeSources(image, starImages, "star")
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/insertFakes.py", line 560, in addFakeSources
    for (fakeImage, xy) in fakeImages:
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_tasks/20.0.0/python/lsst/pipe/tasks/insertFakes.py", line 506, in mkFakeStars
    flux = photoCalib.magnitudeToInstFlux(row[self.config.magVar % band], xy)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-1a1d771/lib/python3.7/site-packages/pandas/core/series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-1a1d771/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4418, in get_value
    raise e1
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-1a1d771/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4404, in get_value
    return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
  File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 90, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'gmagVar'
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v20.0.0/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-1a1d771/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4410, in get_value
    return libindex.get_value_at(s, key)
  File "pandas/_libs/index.pyx", line 44, in pandas._libs.index.get_value_at
  File "pandas/_libs/index.pyx", line 45, in pandas._libs.index.get_value_at
  File "pandas/_libs/util.pxd", line 98, in pandas._libs.util.get_value_at
  File "pandas/_libs/util.pxd", line 83, in pandas._libs.util.validate_indexer
TypeError: 'str' object cannot be interpreted as an integer

how can I solve this issue? Does this mean that if I want to add fakes on visits in G filter, I should create their input catalog with gvarMag, and so yvarMag and ivarMag for HSC-Y and I bands? Thank you in advance for your answer

#2

You need to give it a magnitude in each band you want to input it in. So if you want to add fakes to a g band image then you need a gmagVar column in your input catalogue. Are you giving it a CSV file or converting to fits?

#3

Hello, thank you for your answer. No I use it with CSV format (I already use it and I had no problems). Is it recommended to convert it to fits?

#4

It is expecting a fits file so I am impressed it works with a CSV.

#5

A KeyError means the key you gave pandas isn’t valid. Before doing anything with the data frame, use print(df.columns) to see what keys are available.

print(df.columns)

I was getting a similar kind of error in one of my codes. Turns out, that particular index was missing from my data frame as I had dropped the empty dataframe rows. If this is the case, you can do df.reset_index(inplace=True) and the error should be resolved.