SUIT: importing external code

Data Management
#1

@davidciardi @xiuqin @roby @gpdf

Hi! Quick question about the SUIT vision/plan:

I am helping write an FAQ for new LSST DESC members, and want to write the following sentences:

All DESC members will have LSST data access (or else they could not have become members). The easiest (and project-recommended) way to interact with the data will be through the project-provided “Science User Interface and Tools” (SUIT), which is currently planned to provide a jupyter notebook-style interface to code running at the Data Access Centers, including various visualization widgets. DESC code will be able to be installed at the DAC for use in this system.

About this last sentence: is it true? At the moment I can type

!pip install corner.py
import corner

into any of my local jupyter notebooks and then go ahead and make corner plots. Will this be possible in a SUIT notebook that is running at NCSA, thus allowing me to make corner plots of LSST DR1 galaxy properties? If not, by what mechanism will we be able to install code packages so that they are available for our use in the SUIT? Apologies if this was in your presentations in Tucson last week and I missed it!

Also, any other feedback on my FAQ answer would be gratefully received :slight_smile:

#2

Hey Phil,

We’re trying to solidify this literally over the next week; mind if you pause on this question until next ~Wednesday? I think it’s important we’re precise with what we think we’ll deliver (not under- or over-promise).

#3

Sure, no problem! While some of our packages will be as lightweight as corner.py, we are also likely to have bigger packages that will need much more careful maintenance, by a DESC member logging into an NCSA machine. CosmoSIS is a present-day example of the kind of thing I’m talking about. Thanks @mjuric!

#4

@mjuric hasn’t replied yet, but what I think can be said with some confidence is that we recognize that it will be important for users to be able to establish persistent customized environments to which they can add software of interest to them, and that will be supported in the user-computing capabilities we provide. Whether that’s a traditional “home directory” on a shared file system or some container-based way of creating and instantiating user environments is the level of thinking that is still in progress. I would also like to be able to support shared customized environments set up by science collaborations; we will still have to look at the many ways one can imagine to do that.

But my basic answer is: yes, you will be able to make use of externally sourced Python packages of your choice in addition to the software provided by default by LSST. Also, once you have a notebook open in the LSST user environment, it’s just a notebook and you will be able to use mainstream notebook-friendly tools of your choice to make plots in the notebook to augment the native visualization capabilities that the SUIT provides.

With regard to matrices of plots often used to display twofold correlations among a set of N variables (sometimes called “corner plots”), we will probably (I can’t quite promise this yet, but it is something I really want) support this natively in the SUIT. But even if we do, your original question still stands, obviously - we are certainly not intending (nor would it be wise) to chase all the cool visualizations that people in the data-science world invent, so we need to make sure that they can simply be used in our environment.

I would say “… which is currently planned to provide both web-portal- and Jupyter-notebook-style interfaces to the data and to enable running code at the Data Access Centers, including a variety of visualization widgets.”

But, @drphilmarshall, what was the Q for which this was an A?

#5

+1 on @gpdf’s post above (thanks Gregory!).

I have one more visit to do next week to discuss/understand one more aspect of this concept, and we’ll then put together an overview document (plus a set of slides) laying this out in much more detail. The terminology may change a bit (e.g., what’s SUIT, what’s Level 3, what’s something else) – we want to simplify the way we speak and present the totality of services offered to the user.

But the short answer is yes, you will be able to install/run your own code on DAC-provided compute resources!

Thanks for being patient!

#6

Excellent - thanks very much Gregory, Mario.