Brilliant - seems to be working nicely now on our local cluster with OMP_NUM_THREADS=1. Thanks!
The data set we were given didn’t seem to include any FLAT, FRINGE, or BFKERNEL files (or at least ingestImages.py didn’t make any links with those categories). So I’ve downloaded the ones you linked to on tigress above. Would the first step with those be ingestImages.py or would I skip straight to ingestCalibs.py for some reason?
I have no idea what data you were given, but I don’t think there are standard calibs so you would need to build your own from raw exposures if the data you were given contains biases darks and flats, or download some built by someone else.
If you download our calibs, you shouldn’t have to do any ingestImages.py or ingestCalibs.py because they have already been ingested (put in calibRegistry.sqlite3 — make sure you also get that file!). You can jump directly to processing your data with processCcd.py or singleFrameDriver.py.
Okay, brilliant - thanks - my problem was that the data I was given had biases and darks but no flats, as far as I could tell. So I decided to try ignoring the calibs from there and just using the ones you built.
I downloaded the bits of your calibrations that were nearby my test visit in time and used the same filter, removed everything I didn’t download from the calibRegistry.sqlite3 with some SQL delete statements.
Things seem to go okay but I now hit a seg fault:
... Lots of reasonable looking output, then:
processCcd.calibrate.detectAndMeasure.deblend: Deblended: of 485 sources, 45 were deblended, creating 108 children, total 593 sources
processCcd.calibrate.detectAndMeasure.measurement: Measuring 593 sources (485 parents, 108 children)
processCcd.calibrate.detectAndMeasure.measurement: Measuring 593 sources (485 parents, 108 children)
processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 2 flux fields
processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation
processCcd.calibrate.detectAndMeasure.measurement: Measuring 593 sources (485 parents, 108 children)
processCcd.calibrate: Copying flags from icSourceCat to sourceCat for 121 sources
processCcd.calibrate.astrometry.refObjLoader: Loading reference objects using center (1023.5, 2087.5) pix = Fk5Coord(248.4861175, 12.8905772, 2000.00) sky and radius 0.109876613994 deg
JAZ anDir = /home/zuntz/lsst/ci_hsc/sdss-dr9-fink-v5b
JAZ: andConfigPath: /home/zuntz/lsst/ci_hsc/sdss-dr9-fink-v5b/andConfig.py
Program received signal SIGSEGV, Segmentation fault.
Those last two lines are from me debugging an earlier problem - no changes apart from the print out. I ran under gdb and got this traceback:
#0 0x00007fffffff4dc0 in ?? ()
#1 0x00000031eb23493b in msort_with_tmp () from /lib64/libc.so.6
#2 0x00000031eb2348b8 in msort_with_tmp () from /lib64/libc.so.6
#3 0x00000031eb234f1c in qsort_r () from /lib64/libc.so.6
#4 0x00007fffd101f69f in permuted_sort (realarray=realarray@entry=0x7fffffff4ec0, array_stride=array_stride@entry=8, compare=<optimized out>, perm=perm@entry=0x7fffffff4e50, N=N@entry=4)
at permutedsort.c:93
#5 0x00007fffd101eccf in healpix_distance_to_xyz (hp=hp@entry=263, Nside=Nside@entry=8, xyz=xyz@entry=0x7fffffff4f50, closestxyz=closestxyz@entry=0x7fffffff4f70) at healpix.c:1355
#6 0x00007fffd101f0e8 in healpix_distance_to_radec (hp=263, Nside=8, ra=248.48611753584646, dec=<optimized out>, closestradec=0x0) at healpix.c:1419
#7 0x00007fffd0b13ce4 in healpixDistance (hp=hp@entry=263, Nside=Nside@entry=8, coord=...) at python/lsst/meas/astrom/astrometry_net_wrap.cc:5886
#8 0x00007fffd0b14912 in _wrap_healpixDistance (args=<optimized out>) at python/lsst/meas/astrom/astrometry_net_wrap.cc:28049
#9 0x00007ffff7cfe412 in ext_do_call (nk=-889630080, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffff51a8, func=0x7fffd0f32ab8) at Python/ceval.c:4661
... Followed the usual many layers of PyEval_EvalFrameEx
Any advice? Many thanks again for all your help!
Most of the segfaults we see come from version conflicts - somewhere in the dependency tree, someone has two versions of a package (usually one living outside the LSST stack), and the wrong one is being picked up by the linker. I’d look in the dependencies of your astrometry.net package for anything you also have installed outside the stack (e.g. Python, wcslib, cfitsio), and see what your linker thinks about them (i.e. ldd, nm, or the OS X equivalents). If you’ve got multiple LSST versions of these package installed, you might want to try running your EUPS setup lines again (or in a new terminal), and if you think that is the problem, you could post those lines here.
I can’t see any external libraries with lsof -p on the running process. Here’s a pastebin of its output:
http://pastebin.com/Ts5Shcpn
and:
ldd /home/zuntz/lsst/anaconda/opt/lsst/meas_astrom/python/lsst/meas/astrom/_astrometry_net.so | grep -v anaconda
only reveals the obvious system libraries too.
My ASTROMETRY_NET_DATA_DIR is one possible suspect - I found if I didn’t set it manually then the system picked up the wrong one (in the conda package). Possibly some of my loading is in the wrong order? Here’s is the script I source to set up:
export PATH=/home/zuntz/lsst/anaconda/bin:$PATH
source activate lsst
source /home/zuntz/lsst/anaconda/bin/eups-setups.sh
setup lsst
setup obs_sdss
setup lsst_apps
setup obs_subaru
setup pipe_drivers
setup -j -r /home/zuntz/lsst/ci_hsc/
setup -j -r /home/zuntz/lsst/psfex/
setup astrometry_net_data
export ASTROMETRY_NET_DATA_DIR=/home/zuntz/lsst/ci_hsc/sdss-dr9-fink-v5b
export OMP_NUM_THREADS=1
Joe
The reference catalog included in ci_hsc is drastically pared down to reduce the package size. It doesn’t include all of SDSS-DR9, which may be causing this segfault (if so, we should fix this so it fails nicely). I’ve put up the entire sdss-dr9-fink-v5b catalog for download: http://tigress-web.princeton.edu/~pprice/sdss-dr9-fink-v5b/
More discussion about reference catalogs (including the reference to cite for the sdss-dr9-fink-v5b catalog) is available here.
Okay, thanks for the help - I’ve made a lot of progress.
I’ve managed to processCcd.py on all the data I have, though I had to set:
config.calibrate.doAstrometry = False
config.calibrate.doPhotoCal = False
as otherwise I couldn’t get rid of the astrometry.net healpix seg faults. I also had to roll my own parallelization as the inbuilt MPI and SMP approaches with singleFrameeDriver.py both had their own issues.
So now I have a bunch of files in a rerun subdirectory, in some_number/HSC-I/output and some_number/HSC-I/corr. There seems to be PSF information in the corr/CORR* files, but I can’t quite figure out how to deserialize it (my goal is to get to the point where I can call some function and get a PSF image at a chosen location in the plane).
I thought it could be to do with lsst.meas.extensions.psfex.PsfexPsf.readFits but I couldn’t get that to return anything other than None.
Any help would be amazing!
Cheers,
Joe
I’m afraid that the moment you’ll have to read the full images in and then extract PSFs from them. This is something we’re actively working on fixing, but it’s a bit structural, so that may not be immediate.
That will look something like:
import lsst.daf.persistence
butler = lsst.daf.persistence.Butler("/path/to/rerun")
calexp = butler.get("calexp", visit=NNN, ccd=XXX, immediate=True)
psf = calexp.getPsf()
# get an lsst.afw.image.ImageD of the PSF at point (x, y), centered on the middle pixel of the image.
image = psf.computeKernelImage(lsst.afw.geom.Point2D(x, y))
array = image.getArray() # get a numpy array
You can use computeImage instead of computeKernelImage if you want the PSF image to be centered at (x, y), so you could e.g. use it to subtract a star at that position. That will still return a postage stamp; Image objects have an “xy0” integer offset that will be set here to indicate how to align the postage stamp with the larger image. (For computeKernelImage, that offset is set to put the origin of the postage stamp on the middle pixel).
Brilliant - thanks Jim - that seems to work nicely.
It would be good to get to the bottom of these issues so they won’t bite others. Did you use the full SDSS-DR9 I provided? What was the problem with singleFrameDriver.py?
I used the full SDSS-DR9 one that you pointed to, yes. I suppose the target regions might be outside the DR9 region, not sure.
I just get the message “Aborted (signal 6)” from the PBS version of single_frame_driver.py. There are various error-looking messages in there but I can’t tell which ones should be expected and which ones are red flags. I will email you the full output from the job (the observer asked that we keep the target names private for now).
Cheers,
Joe
This may indicate a problem in the code. The singleFrameDriver.py (and the ctrl_pool framework) is more sensitive to memory problems than processCcd.py. I’ll have a dig around.
I’ve filed a ticket to reduce the chatter.
Oh, I found the problem in the log you sent:
python: src/hsm/PSFCorr.cpp:731: void galsim::hsm::find_ellipmom_1(galsim::ConstImageView<double>, double, double, double, double, double, double&, double&, double&, double&, double&, double&, double&, boost::shared_ptr<galsim::hsm::HSMParams>): Assertion `iy1 <= iy2' failed.
This was fixed a month ago, both in our stack and upstream in GalSim. Please check that you’re using a recent version of the LSST stack.
Dunno why you wouldn’t have seen this error with your own parallelisation, except that perhaps you didn’t have the same environment (e.g., didn’t setup meas_extensions_shapeHSM).
What is the easiest/cleanest way to update GalSim to deal with this problem?
I tried to follow the instructions in here but, even if the correct version appeared when I did eups list -s, when I tried to do import galsim it didn’t work. I also tried doing eups distrib install with the latest weekly release but, when I tried to use it in combination with other packages (that worked fine with v12_0) I had a strange memory allocation problem when using galsim inside an ipython notebook.
eups distrib install is the way to go. Did you mix and match versions after installing the latest weekly release? If you tried to use, e.g., lsst_distrib from v12_0 and galsim from the latest weekly, e.g., using setup -j, then that could cause memory problems. You need to use a consistent set of versions.
If that’s not the problem, could you please post more details?
Yes, I was mixing lsst_distrib and galsim from v12_0 and the new one. Thanks!
Hi all. We’re coming back to this project after a bit of a hiatus now.
Things seemed to have changed a bit (a good sign I think!) I’m trying to access an old repository I made with version 11 with the data butler and the newest version of the ci_hsc module .
Traceback (most recent call last):
File "test1.py", line 9, in <module>
butler = lsst.daf.persistence.Butler("/global/cscratch1/sd/zuntz/lsst/wilman-run/run2")
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/daf_persistence/12.1/python/lsst/daf/persistence/butler.py", line 285, in __init__
self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/daf_persistence/12.1/python/lsst/daf/persistence/butler.py", line 375, in _addRepo
"Could not infer mapper and one not specified in repositoryArgs:%s" % args)
RuntimeError: Could not infer mapper and one not specified in repositoryArgs:RepositoryArgs(root='/global/cscratch1/sd/zuntz/lsst/wilman-run/run2', cfgRoot=None, mapper=None, mapperArgs={}, tags=set([]), mode='rw')
I get similar errors when trying to create a new repository by ingesting raw data again:
Traceback (most recent call last):
File "/opt/lsst/software/stack/Linux64/pipe_tasks/12.1-20-g324f6d3+6/bin/ingestImages.py", line 3, in <module>
IngestTask.parseAndRun()
File "/opt/lsst/software/stack/Linux64/pipe_tasks/12.1-20-g324f6d3+6/python/lsst/pipe/tasks/ingest.py", line 380, in parseAndRun
args = parser.parse_args(config)
File "/opt/lsst/software/stack/Linux64/pipe_base/12.1-5-g06c326c+6/python/lsst/pipe/base/argumentParser.py", line 459, in parse_args
namespace.camera = mapperClass.getCameraName()
AttributeError: 'NoneType' object has no attribute 'getCameraName'
Are there some changes I have to make now? Thanks so much for ongoing help - I know you guys are amazingly busy.
I’ve managed to fix these problems, which were mainly due to having wrong versions of repositories and a wrong _parent link because I copied things from a remote machine.
Cheers,
Joe
Hi all - congratulations on the data release!
One more problem I’ve hit trying to use the NERSC installation v12.1 - I can ingest okay but when I try processCcd.py I get this - wondered if anyone had seen it before?
Traceback (most recent call last):
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/cmdLineTask.py", line 346, in __call__
result = task.run(dataRef, **kwargs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_tasks/12.1/python/lsst/pipe/tasks/processCcd.py", line 181, in run
icSourceCat = charRes.sourceCat,
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_tasks/12.1/python/lsst/pipe/tasks/calibrate.py", line 383, in run
icSourceCat=icSourceCat,
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_tasks/12.1/python/lsst/pipe/tasks/calibrate.py", line 462, in calibrate
sourceCat=sourceCat,
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_astrom/12.1/python/lsst/meas/astrom/astrometry.py", line 197, in run
res = self.solve(exposure=exposure, sourceCat=sourceCat)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_astrom/12.1/python/lsst/meas/astrom/astrometry.py", line 285, in solve
calib=expMd.calib,
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_algorithms/12.1/python/lsst/meas/algorithms/loadReferenceObjects.py", line 214, in loadPixelBox
loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_astrom/12.1/python/lsst/meas/astrom/loadAstrometryNetObjects.py", line 98, in loadSkyCircle
self._readIndexFiles()
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pipe_base/12.1/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_astrom/12.1/python/lsst/meas/astrom/loadAstrometryNetObjects.py", line 162, in _readIndexFiles
self.multiInds = AstrometryNetCatalog(self.andConfig)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_astrom/12.1/python/lsst/meas/astrom/multiindex.py", line 186, in __init__
self._initFromCache(cacheName)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/meas_astrom/12.1/python/lsst/meas/astrom/multiindex.py", line 238, in _initFromCache
with pyfits.open(filename) as hduList:
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/hdu/hdulist.py", line 124, in fitsopen
return HDUList.fromfile(name, mode, memmap, save_backup, **kwargs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/hdu/hdulist.py", line 266, in fromfile
save_backup=save_backup, **kwargs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/hdu/hdulist.py", line 823, in _readfrom
hdu = _BaseHDU.readfrom(ffo, **kwargs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/hdu/base.py", line 370, in readfrom
**kwargs)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/hdu/base.py", line 430, in _readfrom_internal
header = Header.fromfile(data, endcard=not ignore_missing_end)
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/header.py", line 423, in fromfile
padding)[1]
File "/global/common/cori/contrib/lsst/lsstDM/v12_1/Linux64/pyfits/3.4.0+6/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/header.py", line 492, in _from_blocks
raise IOError('Header missing END card.')
IOError: Header missing END card.I suggest having a look at your astrometry_net_data package. Specifically, you might check that the andCache.fits file isn’t corrupted.