Error with lsst-v12.0: 'AstrometryConfig' object has no attribute 'solver'

Support
#1

Hi,

I am trying to use processCcd.py with lsst-v12.0 stack stored on cvmfs (/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0).

I am using the obs_cfht package (git clone https://github.com/lsst/obs_cfht) and I have executed the following command lines successfully:

source ${STACK_PATH}/loadLSST.bash
setup pipe_tasks
cd /processing/my_packages/obs_cfht/ && setup -k -r .
scons
eups declare --nolocks -r . obs_cfht cfhtprod
setup obs_cfht cfhtprod
cd /processing/input/CFHT-Deep/
eups declare -r /processing/input/CFHT-Deep -m none astrometry_net_data CFHT_Deep
setup -k astrometry_net_data CFHT_Deep
cd /processing
ingestImages.py input input/rawDownload/*.fz --mode link

However I get the following error message thereafter:

# > processCcd.py input --output ${OUTPUT_DIR} --id visit=803022 ccd=13 --configfile ${VOLUME_DIR}/input/processConfig.py
: Loading config overrride file '/processing/my_packages/obs_cfht/config/processCcd.py'
Traceback (most recent call last):
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pipe_tasks/12.0/bin/processCcd.py", line 25, in <module>
    ProcessCcdTask.parseAndRun()
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pipe_base/12.0/python/lsst/pipe/base/cmdLineTask.py", line 450, in parseAndRun
    parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pipe_base/12.0/python/lsst/pipe/base/argumentParser.py", line 436, in parse_args
    self._applyInitialOverrides(namespace)
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pipe_base/12.0/python/lsst/pipe/base/argumentParser.py", line 599, in _applyInitialOverrides
    namespace.config.load(filePath)
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pex_config/12.0/python/lsst/pex/config/config.py", line 532, in load
    self.loadFromStream(stream=code, root=root)
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pex_config/12.0/python/lsst/pex/config/config.py", line 552, in loadFromStream
    exec stream in {}, local
  File "/processing/my_packages/obs_cfht/config/processCcd.py", line 43, in <module>
    'i2': 'i',
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/Linux64/pex_config/12.0/python/lsst/pex/config/configurableField.py", line 103, in __getattr__
    return getattr(self._value, name)
AttributeError: 'AstrometryConfig' object has no attribute 'solver'

I do not understand why processCcd.py is not working as expected. Any help would be appreciated :wink:

Thanks,

Yvan

#2

This seems to be because of changes made to obs_cfht after the release of v12 of the stack. Can you try reverting to a slightly older version of obs_cfht? I think that abdb9b9 should work. That is, after cloning from GitHub, but before building and setting up obs_cfht, run

git checkout abdb9b9

in your obs_cfht directory.

#3

What is in this file? I suggest you try running the same command except remove --configfile ${VOLUME_DIR}/input/processConfig.py and add --show config. if that passes then your config override file has an outdated setting, one that may be clear from the displayed config.

#4

I now have another error message:

# processCcd.py input --output ${OUTPUT_DIR} --id visit=803022 ccd=13 --configfile ${VOLUME_DIR}/input/processConfig.py

: Loading config overrride file '/processing/my_packages/obs_cfht/config/processCcd.py'
: Config override file does not exist: '/processing/my_packages/obs_cfht/config/megacam/processCcd.py'
usage: processCcd.py input [options]
processCcd.py: error: cannot load config file '/processing/input/processConfig.py': No module named extensions.psfex.psfexPsfDeterminer

My processConfig.py contains the following lines:

# Apply the brighter fatter correction
config.isr.doBrighterFatter=False

config.charImage.repair.cosmicray.nCrPixelMax=1000000

#from lsst.meas.astrom.anetAstrometry import ANetAstrometryTask
#config.calibrate.astrometry.retarget(ANetAstrometryTask)
#config.calibrate.astrometry.solver.sipOrder=3

config.charImage.astrometry.wcsFitter.order=3
#config.charImage.astrometry.matcher.maxMatchDistArcSec=3.0

#config.calibrate.astrometry.refObjLoader.filterMap = { 'i2': 'i',
#                                                           }
config.charImage.astrometry.refObjLoader.filterMap = { 'i2': 'i'}

import lsst.meas.extensions.psfex.psfexPsfDeterminer
config.charImage.measurePsf.psfDeterminer.name='psfex'

#config.calibrate.photoCal.fluxField='base_CircularApertureFlux_6_0_flux'
#config.calibrate.photoCal.magLimit=22.0
#config.calibrate.photoCal.photoCatName="e2v"
#config.calibrate.photoCal.applyColorTerms=True
#config.calibrate.photoCal.badFlags=['base_PixelFlags_flag_edge', 'base_PixelFlags_flag_interpolated', 'base_PixelFlags_flag_saturated', 'base_PixelFlags_flag_crCenter']

import lsst.pipe.tasks.colorterms
config.calibrate.photoCal.colorterms.data['e2v'].data['i2']=lsst.pipe.tasks.colorterms.Colorterm()
config.calibrate.photoCal.colorterms.data['e2v'].data['i2'].c2=0.0
config.calibrate.photoCal.colorterms.data['e2v'].data['i2'].c1=0.085
config.calibrate.photoCal.colorterms.data['e2v'].data['i2'].c0=0.0
config.calibrate.photoCal.colorterms.data['e2v'].data['i2'].primary='i'
config.calibrate.photoCal.colorterms.data['e2v'].data['i2'].secondary='r'

# The following should be included for u filter in order to lower the source detection threshold
# config.charImage.detectAndMeasure.detection.includeThresholdMultiplier=1.0

# use Chebyshev background estimation
config.charImage.background.useApprox=True
config.charImage.detectAndMeasure.detection.background.binSize=128
config.charImage.detectAndMeasure.detection.background.useApprox=True
config.charImage.background.binSize = 128
config.charImage.background.undersampleStyle = 'REDUCE_INTERP_ORDER'
config.charImage.detectAndMeasure.detection.background.binSize = 128
config.charImage.detectAndMeasure.detection.background.undersampleStyle='REDUCE_INTERP_ORDER'
config.charImage.detectAndMeasure.detection.background.binSize = 128
config.charImage.detectAndMeasure.detection.background.undersampleStyle = 'REDUCE_INTERP_ORDER'

#config.charImage.detectAndMeasure.deblend.minFootprintAxisRatio=0.0
#config.charImage.detectAndMeasure.deblend.maxFootprintSize=0
#5

Given the (quoted) contents of line 43, I’m guessing it’s the default obs_cfht override.

#6

Ok so it seems to work now. What I did:

# processCcd.py input --output /processing/output_one_ccd/ --show config > /processing/input/processConfig.py

and:

# processCcd.py input --output /processing/output_one_ccd/ --id visit=803022 ccd=13 --configfile /processing/input/processConfig.py
: Loading config overrride file '/processing/my_packages/obs_cfht/config/processCcd.py'
: Config override file does not exist: '/processing/my_packages/obs_cfht/config/megacam/processCcd.py'
: input=/processing/input
: calib=None
: output=/processing/output_one_ccd
CameraMapper: Loading registry registry from /processing/output_one_ccd/_parent/registry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in root: /processing/output_one_ccd/calibRegistry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper: Loading Posix registry from /processing/output_one_ccd
CameraMapper: Loading registry registry from /processing/output_one_ccd/_parent/registry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in root: /processing/output_one_ccd/calibRegistry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper: Loading Posix registry from /processing/output_one_ccd
processCcd: Processing {'taiObs': '2005-07-14T07:26:21.14', 'extension': 14, 'object': 'D3', 'visit': 803022, 'filter': 'r', 'state': 'p', 'runId': '05AL01', 'date': '2005-07-14', 'ccd': 13, 'expTime': 300.238}
[....]
processCcd.calibrate.photoCal: Magnitude zero point: 32.139278 +/- 0.000487 from 40 stars
processCcd.calibrate: Photometric zero-point: 32.139278

So if I understand correctly, I have to revert to an older version of obs_cfht and I have to generate the processConfig.py configuration file before to execute processCcd.py with a given visit and ccd id.

Am I correct?

#7

Weird. The error is

but your config file doesn’t ask for that to be loaded: it says

which looks fine. Indeed, I just tried your config on my system, and it works fine.

Are you sure there wasn’t a copy and paste error somewhere? :slight_smile:

#8

Revert to an older version of obs_cfht: yes.

Generating processConfig.py depends on what you want to achieve. If you’re just using the default configuration, there’s no need to dump it to a file with --show config then load that when you start processing: processCcd.py will run with that default configuration without your intervention.

On the other hand, if you want to modify the configuration, then you could dump it to a file like that and use it as a template. If you do that, then yes, it’s possible the configuration options will change for different versions of the stack and/or pipe_tasks, so you might find it safest to dump a fresh copy of the configuration when the versions change.

Note, though, that your config file only needs to specify the parameters that you need to override. A minimal override file is less likely to be affected by changes than one in which you list absolutely everything, even the defaults.

#9

Ok understood. I will create an empty processCcd.py file and add configuration options later if needed (I have no idea of the meaning of those parameters). This way, I will keep the --configfile option as a reminder :stuck_out_tongue:

One last question: is there a way to know which version of obs_cfht is compatible with a given version of the stack?

#10

Our usual recommendation is to run without overrides unless you are sure you need them. Otherwise it means you are using a unique configuration nobody else understands, which makes it hard to help you if you run into problems.

As to version compatibility…good question and I don’t have the answer. obs_cfht is not yet part of the standard distribution (though I think there are plans to change that). I hope there is a better way, but one thing that ought to work is to check the obs_cfht history in git and pick master that matches the date of the release.

#11

I’m not aware of a better way to find a compatible version that the one Russell suggests. This is a known problem (DM-5661): I hope we’ll be able to fix it soon.

#12

Ok thanks a lot @swinbank and @rowen for your help and for your advices :slight_smile: