Task documentation and the Numpydoc & Flake8 transitions

Data Management
#1

This topic summarizes what to do about Task documentation during the Numpydoc transition, and how it relates to the Flake8/Travis CI effort that’s also underway.

Summary:

  • Don’t transition Task class docstrings to Numpydoc just yet. Leave them in Doxygen format. If you’re working on docstrings in a package that has Tasks, see the automodapi work-around described below.
  • The Doxygen-formatted docstrings of Tasks conflict with Flake8’s E266. You’ll need to add a temporary exception to packages with Tasks.

Flake8 and packages with Doxygen-formatted Task docstrings

@rowen pointed out that the existing Doxygen docstrings of our Task classes conflict with the E266 Flake8 error code (see Slack conversation). The best way to work around this in the interim, while still having good Flake8 testing, is to add an extra “ignore” to the package’s setup.cfg configuration file. Here is an example setup.cfg based on the example from the Developer Guide:

[flake8]
max-line-length = 110
# TODO: remove E266 when Task documentation is converted to rst in DM-14207
ignore = E133, E226, E228, N802, N803, N806, E266
exclude = __init__.py, tests/testLib.py

[tool:pytest]
addopts = --flake8
flake8-ignore = E133 E226 E228 N802 N803 N806
    # TODO: remove E266 when Task documentation is converted to rst in DM-14207.
    moduleName.py E266

Note: moduleName.py is the filename (without directories) of any Python module that has a Task class in it. Repeat that line as necessary.

Transitioning Task documentation to Sphinx

Related to this is the overall transition of Task documentation to Sphinx. This area is still under design and development, and my recommendation for developers is to avoid migrating Task docstrings to Numpydoc at the this stage.

If you’re working on adding Numpydoc docstrings to a package that has Tasks, you will need to temporarily omit those APIs from the Sphinx output. You’ll do this by modifying the automodapi directive in the doc/lsst.package.name/index.rst file to be something like this:

.. automodapi:: lsst.package.name
   :skip: MyTask, OtherTask

.. TODO: Re-add Task classes after Numpydoc conversion in DM-14207

What I can forecast about Sphinx-based Task documentation is this:

  • Task classes will be documented just like any Python class with Numpydoc. There won’t be anything particularly special about them.
  • For every Task, there will be a reStructuredText file in the package’s doc/ directory that provides user-facing Task documentation. These pages will be highly structured. You can read about the initial design of these Task topic pages in DMTN-030.
  • Remaining work on my part is:
    1. Finish the design of Task topics.
    2. Implement some reStructuredText tooling to semi-automate some aspects of Task documentation.
    3. Convert several Tasks to the new documentation format as proofs of concept.
    4. Roll out authoring documentation for the DM team.
2 Likes