Problem Chi2 minimization failure using jointcal.py

Yuanyu · November 17, 2021, 8:00am

I’m dealing with the data of m31 using the HSC pipeline 8.5.3,when I process jointcal,using the command:

 jointcal.py /data/HSC/ --calib /data/HSC/CALIB --rerun m31 --id visit=184620..184632:2 ccd=0..8^10..103

erros hapened:

jointcal WARN: ccdImage 184624_49 has only 78 measuredStars (desired 100)
jointcal WARN: ccdImage 184624_50 has only 84 measuredStars (desired 100)
jointcal WARN: ccdImage 184626_49 has only 84 measuredStars (desired 100)
jointcal WARN: ccdImage 184628_49 has only 85 measuredStars (desired 100)
jointcal WARN: ccdImage 184628_50 has only 84 measuredStars (desired 100)
jointcal WARN: ccdImage 184630_50 has only 77 measuredStars (desired 100)
jointcal WARN: ccdImage 184632_50 has only 88 measuredStars (desired 100)
jointcal WARN: still some outliers but chi2 increases - retry
jointcal.AstrometryFit INFO: assignIndices: Now fitting Distortions Positions
CHOLMOD warning: not positive definite
jointcal.AstrometryFit ERROR: minimize: factorization failed
jointcal INFO: Model chi2/ndof : 3.15378e+19/547524=5.76008e+13
jointcal WARN: ccdImage 184624_49 has only 78 measuredStars (desired 100)
jointcal WARN: ccdImage 184624_50 has only 84 measuredStars (desired 100)
jointcal WARN: ccdImage 184626_49 has only 84 measuredStars (desired 100)
jointcal WARN: ccdImage 184628_49 has only 85 measuredStars (desired 100)
jointcal WARN: ccdImage 184628_50 has only 84 measuredStars (desired 100)
jointcal WARN: ccdImage 184630_50 has only 77 measuredStars (desired 100)
jointcal WARN: ccdImage 184632_50 has only 88 measuredStars (desired 100)
jointcal FATAL: Failed processing tract 0, RuntimeError: Chi2 minimization failure, cannot complete fit.

And I modify the config

jointcal.py /data/HSC/ --calib /data/HSC/CALIB --rerun m31 --id visit=184620..184632:2 ccd=0..8^10..103 --config minMeasuredStarsPerCcd=70 --clobber-config

jointcal.py still cannot process
log :

jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 391.139 y: 1058.38 in image 184624_8
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 1179.35 y: 2873.91 in image 184624_36
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 148.654 y: 3063.83 in image 184624_36
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 391.139 y: 1058.38 in image 184624_8
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 1179.35 y: 2873.91 in image 184624_36
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 148.654 y: 3063.83 in image 184624_36
jointcal.AstrometryFit INFO: findOutliers: found 7 meas outliers and 0 ref outliers
jointcal.AstrometryFit WARN: chi2 went up, skipping outlier rejection loop
jointcal.AstrometryFit INFO: Number of outliers (Measured + Reference = Total): 19 + 0 = 19
jointcal INFO: Model chi2/ndof : 2.94896e+19/547534=5.38589e+13
jointcal WARN: still some outliers but chi2 increases - retry
jointcal.AstrometryFit INFO: assignIndices: Now fitting Distortions Positions
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 534.234 y: 2534.19 in image 184630_26
jointcal.AstrometryFit WARN:  Inconsistent measurement errors :drop measurement at x: 534.234 y: 2534.19 in image 184630_26
jointcal.AstrometryFit INFO: findOutliers: found 2 meas outliers and 0 ref outliers
jointcal.AstrometryFit INFO: findOutliers: found 1 meas outliers and 0 ref outliers
jointcal.AstrometryFit INFO: findOutliers: found 2 meas outliers and 0 ref outliers
jointcal.AstrometryFit WARN: chi2 went up, skipping outlier rejection loop
jointcal.AstrometryFit INFO: Number of outliers (Measured + Reference = Total): 5 + 0 = 5
jointcal INFO: Model chi2/ndof : 3.15378e+19/547524=5.76008e+13
jointcal WARN: still some outliers but chi2 increases - retry
jointcal.AstrometryFit INFO: assignIndices: Now fitting Distortions Positions
CHOLMOD warning: not positive definite
jointcal.AstrometryFit ERROR: minimize: factorization failed
jointcal INFO: Model chi2/ndof : 3.15378e+19/547524=5.76008e+13
jointcal FATAL: Failed processing tract 0, RuntimeError: Chi2 minimization failure, cannot complete fit.

And I search on github, jointcal.py
The code is like that:

elif result == MinimizeResult.NonFinite:
                filename = self._getDebugPath("{}_failure-nonfinite_chi2-{}.csv".format(name, dataName))
                # TODO DM-12446: turn this into a "butler save" somehow.
                fitter.saveChi2Contributions(filename+"{type}")
                msg = "Nonfinite value in chi2 minimization, cannot complete fit. Dumped star tables to: {}"
                raise FloatingPointError(msg.format(filename))
            elif result == MinimizeResult.Failed:
                raise RuntimeError("Chi2 minimization failure, cannot complete fit.")

So,what’s the true reasons that casued that?How should I handle that,Has anyone occued the same problems like that?
Thank you!

parejkoj · November 17, 2021, 6:10pm

Something is failing in the actual fit process, but there’s not enough information here to know. Please run with --loglevel jointcal=DEBUG and attach the full log.

What filter is this data taken with?

The minMeasuredStarsPerCcd config option just determines what level jointcal warns at: we have not yet decided whether to error out if there are few stars.

Yuanyu · November 18, 2021, 7:43am

Thank you! The filter is HSC-R2. jointcal.out (942.8 KB)

parejkoj · November 23, 2021, 10:32pm

There is definitely something problematic with your data: even the initialization-step chi2 values are very high. I suspect you have some bad detectors/visits being included. Run with writeChi2FilesInitialFinal=True and/or writeChi2FilesOuterLoop=True plus debugOutputPath set to the location you want those debug files written to.

The output will be very large .csv files containing all of the stars used in the fit and their parameters at that stage of the fit, including a chi2 column. You can read them with astropy.table.Table.read(file, format='ascii.csv', delimiter='\t', data_start=2) and sort on the chi2 column to see which detector+visit combinations have stars with the highest chi2.

@furuswhs had a similar problem in May 2020, and I since improved jointcal’s checks on increasing chi2, so that it fails earlier in these cases. In that case, detector=3 needed to be excluded from the fit for all visits.