Installing other packages(?) alongside the stack

Support
#1

Hi

I have the lsst stack installed (v22).

How do I go about installing other packages i.e. partition and qserv.

I’ve tried this approach

https://pipelines.lsst.io/install/package-development.html

i.e. git cloning > setup > scons

for partition this led to an issue with nlohmann.json not found (I manually got round this)

For qserv I hit problems with antlr4 not found.

So I’m thinking I should be doing it another way, via eups, and the approach on

https://developer.lsst.io/stack/eups-tutorial.html

but I hit problems this way too as I seem to lose the already installed stack packages (scons, sphgeom).

Is there an example somewhere of installing an existing lsst github package that’s not included with lsst_distrib or apps.

Thanks
Mike

#2

Hi Mike,

I can speak to some of the Qserv-related portions of your question.

It has been assumed that Qserv end users will in almost all cases be consuming project-built containers and deploying these via Kubernetes, and that from-source builds outside the Qserv dev team will be a rarity. If your primary interest in Qserv is setting up experimenting with a running instance, a good starting point is the Kubernetes Qserv Operator, available here: OperatorHub.io | The registry for Kubernetes Operators.

If your interest really is in building Qserv ground-up from source, please note the following:

  • Qserv builds themselves are currently entirely containerized (builds take place within a build docker container, with toolsets and prerequisites pre-installed in that build-container.) The starting point for this is the script at https://github.com/lsst/qserv/blob/master/admin/tools/docker/1_build-deps.sh. Prerequisite package management (eups-mediated or otherwise) for Qserv builds is thus confined to the build container, and typically does not interact with source or stack environments on the host outside the build container.

  • The Qserv team has a PR currently in review which substantially revises Qserv container factorization and build process. Anything learned of the current build system is unfortunately likely to be rendered obsolete when that PR hits (anticipated in next couple of week) – it would be good to wait for that merge, if possible, before digging too far into the current Qserv build system…

  • It is still theoretically possible to do an “unrolled” build in a non-containerized environment, but this has not been the path used by either the development team or by our CI servers for the past several years, so this may be somewhat rough going. This would really only be feasible in a Centos 7 or recent Ubuntu environment. If this is your interest, please DM and I’d be happy to work with you and provide further details.

For the somewhat simpler problem of setting up the partition package, I would typically do this with a -k argument to eups setup to avoid knocking over any other eups packages I had already setup? Others here are without doubt more qualified to help with this, or to comment on lsstsw build practices with which I am much less familiar.

cheers,
–FritzM.

1 Like
#3

Hi Fritz

Thanks, our full qserv deployment will be indeed be via container so hopefully more straightforward. It was just that in looking at partition there was mention of example config files in qserv and that’s what led me to also trying to have my own local qserv installation. Good to have that qserv background info too.

If someone does have step by step instructions on how to get/install partition (or any other package) alongside the stack that would be great as I think I’m missing something obvious as I keep hitting build issues.

My latest attempt was

eups distrib install -t latest partition

but that failed trying to build [ 1/6 ] doxygen 1.8.5

Mike

#4

Basically, as Fritz indicated, the answer is “don’t do that”. The next answer is “you can’t do it ‘alongside’ the stack” as it requires a different conda environment (specifically scipipe_conda_env/conda-linux-64.lock at qserv · lsst/scipipe_conda_env · GitHub) and hence a separate eups installation altogether. Finally, there’s no latest tag in eups; you would use w_latest or d_latest, except that only packages in lsst_distrib are tagged that way, so instead you need to use -t qserv-dev.