When you use the Notebook Aspect of the RSP you are given the option of picking an image before you start in the JupyterHub screen that looks something like this:
Unless you have a very specific reason to do otherwise, you should leave the default (Recommended) selection checked. But what is The Recommended (as you will often hear project staff refer to it)?
Behind the Scenes
When you use the Notebook Aspect, everything you do happens inside a container [1]. That container is based on a Docker image [2] and the RSP menu pictured above is offering you a choice of which image you want the RSP to launch (aka “spawn”) on your behalf.
These images are the end results of the Rubin Data Management software development chain. Every night our build system builds an image of the latest version of all the Science Pipelines and supporting software and mints an image (which you can actually use to run the Science Pipelines on your computer.
This “daily build” from the science pipelines is then used by the build system to make the RSP daily images by adding a whole bunch of RSP not-so-secret-sauce[3], which is why you can’t necessarily run our RSP notebooks in a stock Jupyter environment.
The same process goes for other builds, including weekly builds (which have longer automated testing as part of their production) and the “official” builds (produced from Science Pipelines major releases).
How we promote the Recommended
The less frequent the release cycle, the more testing and verification we can do on it. Dailies have minimal post-build testing. Weeklies are more stable but are still considered “engineering” releases without official documentation. Official releases (like the current version 22.0) have a much more formal process including release notes, a metrics characterisation report and formal approval by the Data Management Change Control Board.
At the current stage of the project lifecycle (pre-first light), these official releases are far apart compared to the rate of development of new features and the rate of discovery and fixes of bugs. The RSP teams are in regular contact with the Science Pipelines, Middleware and Community Engagement teams to monitor the sweet spot between new capabilities and stability. When that balance tips towards the new capabilities, we select the latest weekly and do some additional (human) testing on it, and if all is well we “promote” it to Recommended.
What you see in the screenshot above is that the weekly build of the 33rd week of the year (weekly 2021_33 - i.e. mid August) is the current Recommended.
The options listed on that menu have one more thing that is special: since we think that these are the most likely images that project staff and early access users will be looking for, they have already been distributed (“pre-pulled”) on all the nodes of the RSP cluster. Since these images are very large (~ 16 GB), they take a while to be downloaded from the container repository to the cluster node that you will have landed on, which increases the start-up time you experience when starting (“spawning”) your RSP Notebook session. Behind the scenes a service makes sure that these images are already available on all the nodes to minimize start-up time.
While other images are available from the drop-down menu, these are not pre-pulled and take a long time to start, and the service may even time-out in some circumstances before that operation completes (which is a behaviour we are hoping to fix)[4].
What does this mean for you
The Recommended is almost always the right option for you, which is why it is the default option in the menu. It is carefully selected to have the right balance between offering you new features and bug fixes while reducing the risk of you getting a release that is broken in some way, and it has reduced startup times from most other images.
You should generally not use other images unless there is a specific reason, such as being asked to test a bug fix you have reported in the latest daily or doing a reproducibility comparison with an official release.
As we move through operations and system stability, we expect that eventually the best option will be an official point release rather than a weekly build - but then that will be Recommended, so the advice to use Recommended will still apply.
Expand for Footnotes
-
Technically, a Kubernetes pod but for the purposes of this explainer you can ignore that difference. You could think of a container as an isolated environment where the applications the RSP runs on your behalf (in this case a custom JupyterLab server) reside. ↩︎
-
A Docker image is a binary that Docker (and Kubernetes) can use to create a streamlined virtual machine for you that your application can run in. ↩︎
-
The RSP Notebook Aspect is much more than an unmodified (“vanilla”) Jupyter environment. We customize both the installation and our environment to make it easier for you to access RSP data and services, ensure you have up-to-date tutorials available and interact with our authentication and authorisation system in a transparent manner. ↩︎
-
Some of you have seen a form of this when suddenly a lot of load hits the IDF cluster at data.lsst.cloud. That cluster is set to autoscale (automatically add nodes during heavy demand) but nodes that are being added to the cluster may not have had a chance to pre-pull the recommended image before a user is allocated to them. We’re thinking about how to engineer around this problem. ↩︎