Python Code Coverage now enabled in sconsUtils

Data Management DM Notifications
#1

With DM-11725 I have enabled Python code coverage when running tests. When you build a package that uses sconsUtils code coverage will be enabled in pytest. The coverage output is written as a summary report to the terminal, and also as HTML into tests/.tests/*-htmlcov/ directories (there will be a directory per pytest invocation – usually the main pytest invocation will generate a file named something like tests/.tests/pytest-PACKAGE.xml-htmlcov/). The location of the coverage files is reported in the coverage summary.

This is the summary output for utils:

---------- coverage: platform darwin, python 3.6.5-final-0 -----------
Name                                   Stmts   Miss Branch BrPart  Cover
------------------------------------------------------------------------
python/lsst/__init__.py                    3      0      0      0   100%
python/lsst/utils/__init__.py             10      0      0      0   100%
python/lsst/utils/get_caller_name.py      21      0     10      2    94%
python/lsst/utils/tests.py               309    101    134     16    66%
python/lsst/utils/version.py               5      0      0      0   100%
python/lsst/utils/wrappers.py            136     16     84     15    85%
tests/test_backtrace.py                   24      2      4      2    86%
tests/test_cache.py                       71     13     18      3    80%
tests/test_executables.py                 18      5      2      1    70%
tests/test_getPackageDir.py               22      2      2      1    88%
tests/test_getTempFilePath.py             90      2      2      1    97%
tests/test_get_caller_name.py             37      2      2      1    92%
tests/test_ordering.py                    17      3      2      1    79%
tests/test_pybind11.py                   116     23     20      1    81%
tests/test_raDecToStr.py                  54      2     18      1    96%
tests/test_utils.py                      179      6      4      2    96%
tests/test_wrappers.py                   320      6      8      3    97%
------------------------------------------------------------------------
TOTAL                                   1432    183    310     50    84%
Coverage HTML written to dir tests/.tests/pytest-utils.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-utils.xml-cov-utils.xml

The low number for lsst.utils.tests is mostly due to code that is only available when daf_base is installed, and the presence of some plotting code that is only used for interactive debugging. The HTML report looks something like this:

38%20AM
38%20AM1272×274 30.3 KB

As part of testing this code, I improved the code coverage of utils tests and removed the lsst.utils.tests.findFileFromRoot() function: it was not used anywhere, and now we discover related testing files using __file__.

1 Like
#2

Will the -htmlcov output be displayed by Jenkins? IIRC it has an option for test coverage output, but I have no idea if pytest is specifically supported.

#3

I hope so, but I wanted to get this feature released now rather than wait for Jenkins. The code was written last summer and, cough, in the SPIE paper I mentioned we do python code coverage so I felt I should enable it.

The XML output file is in Cobertura format and there does seem to be a Jenkins Cobertura plugin but I have no idea if it works well.

@josh can comment on whether it’s best to use the XML output in Jenkins directly like we do for test reports, or simply make the HTML and XML output files available as artifacts for download.

Ideally, we would also like to generate merged coverage reports from all tests during a weekly build so that we can see how coverage is changing as a function of time across all of science pipelines.

#4

That might best be done as a step after the weekly in Jenkins pipeline where we set up lsst_distrib and run pytest on all the installed packages at once.

#5

Combined coverage is easier than I thought. Once you’ve built the software using lsstsw you can do:

% cd build
% coverage combine */.coverage
% coverage html -i

and you get a report. I just did this for lsst_apps and got 81% code coverage.

#6

Also note, you’ll start seeing .coverage turning up in your build trees. It needs to be added to .gitignore for each package.