Processing raft test data with eotest and the stack


These are some step-by-step instructions on how to use the stack to ingest raft test-stand data and process it with eotest.

  1. Install a weekly build, with a version greater than or equal to w_2017_50

    • installation instructions can be found here:
    • alternatively, use a shared stack on, e.g. lsst-dev (all the weekly builds are on there already, it’s really easy and potentially saves a lot of installation pain!)
  2. Setup your stack. This example should work on lsst-dev:

    . /ssd/lsstsw/stack3/loadLSST.bash
    setup lsst_distrib -t w_2017_50

  3. Setup eotest (as it’s not part of lsst_distrib). Note that this is a fork of eotest, to be used until the changes made here are merged upstream (and perhaps even then, e.g. if you want to make pdfs from the outputs)
    NB: you must be within that directory, and in future you will need to do something like setup -j eotest -r ~/path/to/eotest each time you setup the stack.

    git clone
    cd eotest
    git checkout tickets/DM-11348
    setup -j -r .

  4. Make some TS8 data available. Let’s pick one raft, I’m going to use RTM-005 (aka RTM2). This would go something along the lines of:

    mkdir /path/to/ts8_data
    rsync -r <username>* /path/to/ts8_data --progress

If you’re working on lsst-dev you can find this data already copied over, in /datasets/comCam/raw/BNL/prod/LCA-11021_RTM/LCA-11021_RTM-005.

  1. Now we enter stack-land. Make a repository for this data, put a mapper in it, and ingest the data. We want to only ingest the .fits files which are part of the actual acquisitions, so we use the following pattern to select them. Hopefully this is robust. Note that we are using the ComCamMapper from obs_comCam as this is a single raft. If you want to process data from more than one raft, make a different repo for each raft. (Note that you can have multiple runs in each repo, but you can’t have multiple rafts in a repo).

    mkdir /path/to/ts8_repo
    echo "lsst.obs.comCam.ComCamMapper" > /path/to/ts8_repo/_mapper /path/to/ts8_repo /path/to/ts8_data/*/*_acq/v0/*/S*/*.fits

  2. Assuming all the above worked, in that repo you should now have the _mapper, a directory called “raw” containing symlinks to all the ingested files, and a sqlite3 registry file called “registry.sqlite3”. We can now open a python session/jupyter notebook and do things. Let’s start by making a butler for that repo.


# import the relevant stuff
import lsst.daf.persistence as dafPersist
from lsst.cp.pipe import RunEotestTask

# make ourselves a butler
repo_path = '/path/to/ts8_repo/'
butler = dafPersist.Butler(repo_path)

Now, if you used the path on the SLAC servers in this example, you will have copied and ingested data from many different runs, which obviously need to be processed separately. So, we ask the butler which runs are available like so:

runs = butler.queryMetadata('raw', ['run'])

which should return something like

['3849', '4080', '4390', '4418', '5597', '5704', '5819', '4068', '4389', '4417', '5105', '5703', '5811']

So, pick a run number to use. If you’re very brave you can try to deduce which the good runs are using things like eTravellers and the Camera Data Portal. I’m just going to pick 4389 because word on the grape-vine is that that’s a good one.

# instantiate a config for our task, and set the mandatory output config parameter
runEoConfig = RunEotestTask.ConfigClass()

# examples of how you turn the eotest sub-tasks on and off. All on by default
# runEoConfig.doFe55 = False
# runEoConfig.doReadNoise = False
# runEoConfig.doBrightPixels = False
# runEoConfig.doDarkPixels = False
# runEoConfig.doTraps = False
# runEoConfig.doCTE = False
# runEoConfig.doPTC = False
# runEoConfig.doFlatPair = False

# instantiate our task use the config
eotestTask = RunEotestTask(config=runEoConfig)

# run the task, specifying our chosen run if there is more than one in the repo
eotestTask.runEotestDirect(butler, run=4389)

# once the task has run, make a pdf of the results

If all went well, eotest should have run, and produced all its results and written them in the usual eotest output format. If things went really well and you have a TeX distro installed with pdflatex on your path, then pdfs for each sensor in the raft should have been produced in /path/to/ts8_repo/eotest_output/plots.

Note the use of a ticket branch for eotest. It is a ticket branch that forks the LSSTTD-1101 branch of which has been updated to run with python 3 and a modern stack. These changes might be pulled in upstream at some point, and if so, I will try to edit the OP to reflect the use of the main branch.

Other than making the modernisation updates required for things to run, that branch should include no functional changes.

Thanks Merlin! Note that if eotest lists the proper dependencies you could skip the setup lsst_distrib step and directly setup -r . (i.e. without -j) I think.

Hi! I have already installed lsst pipeline and I’d like to use eotest. I wonder whether this still works? Thanks!

Hi! We don’t really support doing things like this any more. You can still run eotest code, but it’s now done quite differently, using the Gen3 butler and some pipelines that have been written by the camera team.

If you provide a little more detail on exactly what you’re doing and trying to achieve then I might be able to give you slightly better advice/direction.

Hi! Thank you for your reply. These days, I am interested in CTI simulation and correction. I found related code on GitHub. It requires eotest. So I was wondering if there’s possibility for me to run that code, so that I could learn more from it.