Help needed for EUPS installing packages not included in an official release of the LSST stack

Support
,
#1

I need to install a package which is not included in lsst_apps on top of a read-only, functional installation of the LSST stack v12.1. The package I need to install (e.g. obs_subaru or obs_cfht) actually exists in LSST EUPS repository (i.e. http://sw.lsstcorp.org/eupspkg) with the appropriate tag.

The concrete use case I have is that some users of the CernVM-FS binary distribution (which is read-only) need to add some specific LSST packages to their individual software stack.

I don’t know if this is a supported use case by EUPS. After re-reading the EUPS documentation, I thought that I could set the EUPS_PATH variable to include both a R/W area (e.g. $HOME) and the R/O area where the stack is already installed.

So this is the sequence of commands I use:

# Tell EUPS to also use $HOME
$ mkdir -p $HOME/.eups && export EUPS_PATH=$HOME/.eups

# Initialize my working environment to use LSST v12.1
$ source /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/loadLSST.bash  

# EUPS_PATH looks good
$ echo $EUPS_PATH
/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1:/home/fabio/.eups

# This is the LSST software distribution server. So far, so good.
$ eups distrib path
http://sw.lsstcorp.org/eupspkg

# EUPS documentation says that one use the 'eups path' command to 
#    'Print the product stack directories given via EUPS_PATH'.
# In this case, it does not print /home/fabio/.eups,  which is actually in $EUPS_PATH
$ eups path
/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1

# Try to install 'obs_subaru'
# I was expecting the command below to work, but instead, it prints an error message
# which does not really help me understanding what is wrong 8-[
$ eups distrib install -I $HOME/.eups -t v12_1 obs_subaru
eups distrib: coercing to Unicode: need string or buffer, list found

# As expected, the command below does not work because
# /cvmfs/... is a read-only file system
$ eups distrib install -t v12_1 obs_subaru
   [  1/66 ]  apr 1.5.2 (already installed)                              done. 
eups distrib: You don't have permission to assign a global tag b2307 in /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/ups_db

Is this use case supported by EUPS? If so, what would be the recommended way to achieve my goal?

Any help with this would be very appreciated.

#2

Is this eups version 2.0.2? It’s possible that we still have some python3 portability bugs in there. It’s also possible that it’s fixed in current eups master.

#3

It is v2.0.2:

$ eups --version
EUPS Version: 2.0.2

Does your answer mean that this is the correct procedure to use for my use case?

#4

Please note that the result of eups path looks unexpected (at least for me).

#5

You may need to set EUPS_PATH before sourcing the eups setups.[zc]sh files.

#6

You can set EUPS_PATH after initialising things. I’ll look at why the “eups path” command didn’t list ~/.eups

#7

That’s also my understanding. This is why I set EUPS_PATH to $HOME/.eups before sourcing loadLSST.bash which itself sets up EUPS.

#8

Great. Thanks !

[One cannot be concise: Discourse wants me to post at least 20 characters, so here I go]

#9

@FabioHernandez, we encourage you to use the ‘Like’ (heart) button instead. See https://community.lsst.org/faq#keep-tidy

1 Like
#10

It does not seem to be fixed in master:

$ eups --version
EUPS Version: 2.0.2-58-gfe1cd0b

$ eups distrib install -I $HOME/.eups -t v12_1 obs_subaru
eups distrib: coercing to Unicode: need string or buffer, list found
#11

It’s deliberate (things like tags get declared in ~/.eups). I just pushed a patch so that “eups path -v” will include ~/.eups

That being said, I’d choose someone else to install these extra packages, and add that to $EUPS_PATH

#12

Do you have any thoughts as to how we can get a traceback for @FabioHernandez’s eups distrib install unicode error?

#13

Sorry, I thought this was resolved. Use “eups --debug=raise …” (this was where LSST’s --doraise came from)

#14

I don’t think anyone told @FabioHernandez to run that (maybe I’m missing something).

#15

Thanks for your feedback. I should have thought that $HOME/.eups had a special meaning for EUPS. Indeed, if I use another directory for installing my EUPS packages, the command eups path works as expected. For the record:

$ mkdir -p $HOME/myeupspackages/ups_db
$ export EUPS_PATH=$HOME/myeupspackages
$ source /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/loadLSST.bash
$ eups path
/home/fabio/myeupspackages
/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1
#16

Here is what I get using the option --debug=raise :

$ eups --debug=raise distrib install -I $HOME/myeupspackages -t v12_1 obs_subaru
Traceback (most recent call last):
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/bin/eups_impl.py", line 57, in <module>
    status = cmd.run() 
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/cmd.py", line 245, in run
    return self.execute()
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/cmd.py", line 172, in execute
    return ecmd.run()
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/cmd.py", line 1903, in run
    return self.execute()
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/cmd.py", line 1923, in execute
    return ecmd.run()
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/cmd.py", line 245, in run
    return self.execute()
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/cmd.py", line 2213, in execute
    if not utils.isDbWritable(self.opts.installStack) and \
  File "/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/eups/python/eups/utils.py", line 291, in isDbWritable
    return os.access(dbpath, (os.F_OK|os.R_OK|os.W_OK))
TypeError: coercing to Unicode: need string or buffer, list found
#17

That’s an ancient bug; git blame exonerates me! I just pushed a fix to git@github.com:RobertLuptonTheGood/eups.git

The workaround would be to prepend $HOME/myeupspackages to EUPS_PATH.

#18

Modifying EUPS_PATH works around the error but does not seem to help making the command ultimately succeed:

$ eups --version
EUPS Version: 2.0.2

$ echo $EUPS_PATH
/home/fabio/myeupspackages:/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1

$ eups path
/home/fabio/myeupspackages
/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1

$ eups distrib install -t v12_1 obs_subaru
   [  1/66 ]  apr 1.5.2 (already installed)                              done. 
eups distrib: You don't have permission to assign a global tag b2307 in /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/ups_db

It seems to me that EUPS still wants to write to the /cvmfs/... component of EUPS_PATH which is read-only. Does EUPS require that I have read-write permissions to all the components of my EUPS_PATH?

#19

OK, I don’t know that -I will work, but the problem with reordering the path is due to someone adding a setup eups to the setups.[c]sh script (which can reorder EUPS_PATH). I have pushed a fix, but it’s to the mksetup script which is used to generate the setups.* scripts.

So you can pull eups from github and run configure, then “setup -r .” to use a version of eups which solves this problem.

That version of eups has the -I bug fixed, but I’m not sure that it ever worked; that’s probably a question for @mjuric (but I might get a chance to look at it).

An alternative to pulling a new eups would be to edit your current eups.table file (you can find it with eups list eups -m) and changing the pathAppend(EUPS_PATH, ...) to pathPrepend(EUPS_PATH, ...) – untested, but I think it’ll work as a short-term alternative.

#20

I cloned EUPS master from github and followed your instructions. The path reordering problem seems to be solved:

$ eups --version
EUPS Version: 2.0.2-63-g12c7bd9

$ echo $EUPS_PATH
/home/fabio/myeupspackages:/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1

$ eups path
/home/fabio/myeupspackages
/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1

However, installing an additional package in a R/W area does not work, neither using the -I option nor without:

$ eups distrib install -t v12_1 obs_subaru
   [  1/66 ]  apr 1.5.2 (already installed)                              done. 
eups distrib: You don't have permission to assign a global tag b2307 in /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/ups_db

$ eups distrib install -I $HOME/myeupspackages -t v12_1 obs_subaru
   [  1/66 ]  apr 1.5.2 (already installed)                              done. 
eups distrib: You don't have permission to assign a global tag b2307 in /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/ups_db

So at this point, I’m stuck.