Three things for productivity

Data Management
#1

The recent discussion on RFC-81 brought up an interesting question (interesting to me at least). How do we know what potential changes to our workflow / tooling / etc are really relevant for increasing our productivity?

As a random start to gain more information on this topic I ask interested members of the community to do the following:

List three potential changes that you expect would most increase your own productivity (in descending priority).

My list would be:

  1. C++14 support
  2. Replace or improve (the interface to) Jenkins
  3. Automatic code formatting (clang-format and autopep8)

Other suggestions could be:

  • Python 3 only
  • Python 2 only
  • Nightly binary releases (to develop against instead of building)
  • IPython notebook based stack as web service (use stack without installing)
  • More Pythonic API
  • Deep freeze on non-essential API changes below layer …
  • Add your own here

Please note that I deliberately left out non-technical changes. I don’t think it would be useful to mix these in, but please feel free to disagree.

And please also feel free to ignore this post if you think it is stupid.

1 Like
#2

I’m not sure of ordering, but the following would be a big help to me:

  • Drop Python 2 support
  • Improve the interface to Jenkins
    • Push notifications for my jobs
    • Make it much easier to specify what is run and to know what will run (no more “Skip Demo” checkbox, sensible defaults for python 3 runs)
  • Standardize the existing code:
    • Run autopep8 on the Python and clang-format on the C++
    • Clean up older packages, especially afw, by moving comments to header files and consistently using namespace {. (I plan to take on afw as “technical debt” work after pybind11 is done, given permission).
  • Persist the results of each Jenkins run for a few days, so if a job fails I can connect to the virtual machine and poke at the build (e.g. run unit tests and the debugger).
  • Specify C++14 as the minimum required
  • Less churn in tools (though I’d be willing to make an exception to get rid of JIRA’s horrible markup language) and standards. Every time we change documentation format and line length (or just spend time arguing about it) we lose time.
  • Jenkins runs include clang on unix and clang on MacOS as standard by default
1 Like
#3
  1. Python 3 only
  2. Less redundancy in the review process between Jira and GitHub
  3. Jenkins improvements (either doing away with the subjobs in stack-os-matrix, or at least making them consistently map to the correct job, consistently have terminal output reachable from the job page, etc.)
1 Like
#4

That’s a relatively new annoyance. The whole “approves this PR” followed by going to JIRA and saying “I approve this” is indeed a bit silly. We have to decide which end we get rid of and I guess for project traceability we have to keep the JIRA “reviewed” step.

2 Likes
#5
  • Python 3. Besides the problems of debugging two interpreters, there are python 3 features I’d like to use, including asyncio and new unittest methods.
  • All of @rowen’s comments about jenkins.
  • Agreed with @rowen’s comments about docstring location.
  • We already cleaned up a lot of the pep8 problems as part of the py3 compatibility work, so that’s much less of a concern to me now. Especially since SublimeLinter-flake8 does all the work for me in new code.
  • I think scons is quite powerful, but we don’t have a good explanation of how to use it. Even something as simple as “what commands can I give it (like lib, python, test) to run a subset of something”.
  • This is related to “pythonic API”, but usable repr/str for all of our C++ objects would be a huge help to debugging.

It’s not exactly tooling, but our lack of documentation of pipeline output (e.g. definitions of flags or the algorithms that go into specific catalog outputs) causes me a fair bit of friction.

1 Like
#6

[eech, stray return posted this a bit early]

Hey thanks to @pschella for starting this very useful thread. I really believe even the small things are important - if the annoyance is breaking your flow, that’s a big problem.

Obviously SQuaRE can help with some but not others. For the stuff we can help with (eg. Jenkins - yeah I know, it’s been on my radar for a while) I’ll block some time in the Mar-May part of the cycle and perhaps we can talk about wishlists at JTM.

For the redundancy between JIRA and Github, I imagine it would be possible to auto-move the JIRA ticket state based on the PR mechanics, or vice versa. Let me have a look at it and if it looks simple, I’ll do a quick RFC.

Nightly binaries - real-soon-now. Should be done before the JTM if @timj’s plan works.

The issue of running auto-whatever (flake8, etc) is a bit of a religious topic. Obviously we can, if people agree, but in general I feel it is best left to people’s editor hooks. I won’t veto it if everybody wants it.

I enthusiastically endorse python3-only and pythonic API issues but sadly I can do nothing in this area - if I could, @timj would have made me already :slight_smile:

#7

Oh one more (boy the list is getting quite long) -

Stack on a notebook server I believe is on the roadmap, but I don’t know timelines at this point.

#8

:100: Yes, I think that documentation will give us a tremendous productivity boost.

I also want to point out that developer productivity is also something I care a lot about with documentation engineering. I want to make documentation as easy as possible to write. LSST the Docs was a start, and soon we’ll be shipping version dashboards and a documentation hub. Chatbot-based project creation (e.g., EUPS package and technotes) is also coming. A tool to automatically convert docstrings to numpydoc format is planned to land before JTM, https://jira.lsstcorp.org/browse/DM-9015.

The documentation design work we’re doing (like https://dmtn-030.lsst.io) will also make documentation much easier to write: we provide the templates and guidance, you fill in the details.

Given how clear the Python Style guide now is, I think it’s safe to integrate flake8 (hopefully via a pytest plugin) into CI. It’d prevent uncertainty about code style during a review.

1 Like
#9
  1. Bite-sized tutorials that produce something useful and explain it with every piece of the stack
  2. Any flavor of “use the stack without building it” that is most feasible and does not require acknowledging the existence of eups (e.g., a notebook server like already mentioned, or even just regular conda updates)
  3. Nuke Jenkins from orbit Replace or improve Jenkins because I’m beyond tired of an old white guy cryptically telling me my code is broken while referencing masters and slaves
#10

@mrawls re point no.3: I’ll see what I can do (the iconography, not the nuking). I mean if I could actually de-offensify the Jenkins symbolism I can probably get a talk at some hot Silicon Valley tech conference on the back of it :slight_smile:

PS. And switching to Travis won’t help with that either! (though at least Travis does have female and non-white persona icons)

1 Like
#11

+1 for usable repr/str of c++ objects.

#12

My three:

  • repr/str for objects, though if we do this I’d very much like to see it done by adding iostream operators in C++ and making that available as repr/str in Python.
  • I would love to have more utilities for managing eups stacks, especially undeclaring/removing multiple products using the same sort of smarts that are in setup or eups list.
  • Ability to log into VMs from failed CI runs to debug.
#13

Would you treat repr and str as distinct things or in C++ just treat them as a single stringification?

Methods for Stringification
#14

+1 for dropping python 2 support.
Nightly binaries would be nice.

2 Likes
#15

how many of us are now developing in python 3 by default?

1 Like
#16

Maybe a better question is, who needs Python 2.7 support from the LSST Stack at this point? I don’t think DM needs it anymore (are there any outstanding Python 2.7-only packages?). Do DM Stack consumers need Python 2.7 — Sims, Camera and DESC?

#17

They all need 2.7 at the moment.

For lsst_distrib we are blocked on the scisql port at the moment. DAX haven’t gone at all yet.

#18

In an attempt to answer two questions at once, I do not use py3 in my day to day, but only because it is still the standard for testing in the stack. I would love to go to py3.

As for external consumers:

  • I don’t know about Camera.
  • Sims would be happy to move over, but need ~six months for OpSim 4 to come out (already py3 compatible) and to translate existing sims code.
  • DESC is an interesting question. I think most of the people doing the bulk of the coding would be happy to move, but my sense is that there is a fairly large group of people that would present significant inertia to moving.
#19

My stack development is all py3 now. It lets me experiment with py3 features (e.g. concurrent.futures),

For Camera and Sims, would DM be able to provide the necessary assistance (say, people who took significant part in the py3 compatibility port for DM) to get them converted?

I feel like there was some polling done in DESC about this last year, but I don’t recall any details.

#20

Hey folks, just an update on the Jenkins user experience concerns. There are some skins around but they are putting lipstick on a pig really. Our best bet seems to be Blue Ocean, which is a new first-party UI https://jenkins.io/projects/blueocean/ - for those familiar with Travis CI, it will feel a lot more similar to that. This short video will give you an idea: https://www.youtube.com/watch?v=3dITffteCD4

It is currently in beta, expected out of beta around March. Unfortunately it only works with the newer Jenkins pipeline model (not the matrix jobs) so it will involve significant re-engineering on our stack CI jobs (we’ve already moved to pipelines for the newer jobs). As a result, you’ll have to suffer for a little bit longer. I haven’t planned it out yet, but my guess is a realistic target is the August AHM.

Y’all be pleased to know that judging from what I have seen, there’s no supercilious white male smirking at you :slight_smile:

1 Like