Failed to ingest git-cloned HSC raw image files

Support
#1

Hi all,

I was following the LSST pipeline tutorial
https://pipelines.lsst.io/getting-started/data-setup.html

and had a problem when I tried to ingest the git-cloned HSC raw image files to Butler DATA directory with messages like

ingest WARN: Failed to ingest file /home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits:
File “src/fits.cc”, line 1427, in lsst::afw::fits::Fits::Fits(const string&, const string&, int)
cfitsio error: error reading from FITS file (108) : Opening file ‘/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits
ffopen could not interpret primary array header of file:
/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits
{0}
lsst::afw::fits::FitsError: 'cfitsio error: error reading from FITS file (108) : Opening file ‘/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits
ffopen could not interpret primary array header of file:
/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits
’

Any comments would be appreciated, thanks!

Duho from South Korea

#2

My guess is that you’ve not set up git-lfs. What are the contents of /home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits?

#3

Hi Paul, Thanks for your response.
The contents seem to be just an ascii-text 132 bytes file that reads:

version https://git-lfs.github.com/spec/v1
oid sha256:00f64a7ae336ce419a9eaac7d16c58f70236736c0781616f7120d3cf7f04591d
size 6897600"

I followed the instruction that you linked on the course of the tutorial. Did I miss something? Thanks. - Duho

#4

That’s what you would expect if you haven’t properly set up git-lfs. Try running:

git lfs checkout
#5

Continuing the discussion from Failed to ingest git-cloned HSC raw image files:

I’m using Docker container lsstsqre/centos:7-stack-lsst_distrib-v19_0_0 and git-lfs seems to be not properly set up in it. When I run the commend, it responds:

git: ‘lfs’ is not a git command. See ‘git --help’.
Did you mean this?
log

I’m currently figuring out how I can set up ‘git-lfs’ in the Docker environment. Do you have any idea about that? Thank you! - Duho

#6

One option is to install git-lfs in a running container. As the container root:

yum install -y git-lfs

You would need to do this every time you start a container wherein you want to use git lfs.

If you want git-lfs available every time you start a container without needing an extra step, another option is to build a new Docker image over the lsstsqre/centos:7-stack-lsst_distrib-v19_0_0 image, with a Dockerfile something like this (untested!):

FROM lsstsqre/centos:7-stack-lsst_distrib-v19_0_0
USER root
RUN yum -y update && yum install -y git-lfs && yum clean all -y
USER lsst

and build with docker build -t duho/centos:7-stack-lsst_distrib-v19_0_0. Then run that image instead of lsstsqre/centos:7-stack-lsst_distrib-v19_0_0.

Another option is to install git-lfs on your machine’s native system, use it to get the files you need, and then map the relevant directory into your container when you run it:

docker run -ti -v /path/to/your/files:/home/lsst/files lsstsqre/centos:7-stack-lsst_distrib-v19_0_0.

For example, I have a bash alias that maps ~/LSST from my native system into the container’s ~/LSST:

lsstDocker () {
    docker run -ti -v ~/docker:/home/lsst -v ~/LSST:/home/lsst/LSST $DOCKER_OPTIONS --cap-add=SYS_PTRACE --security-opt seccomp=unconfined lsstsqre/centos:7-stack-lsst_distrib-${1-$(lsst_latest_weekly)}
}
#7

I resolved this issue by following the first option! Thank you very much! I appreciate it, Paul. - Duho