Qserv core* files causing docker to fail to start

Data Management
#1

Dear Qserv developers,

I have recently taken over a test install of Qserv from Teng
(One of his previous posts - Installing and testing latest version of Qserv in Docker)
and have run into an issue involving very large (2.5G) core* files being generated locally on the master container. Which then stops the container from starting back up as it is out of disk space.

I was able to get around this by copying out the core files to somewhere local
# docker cp master.localdomain:/qserv/core* /somewhere_local

Then I was able to bring up the container and delete the core file.
docker start master.localdomain && docker exec -it master.localdomain rm -rf /qserv/core.2277

I was just wondering if there was any way of moving the location of these core files?
There are a few mounted volumes that are present on the container, so an easy solution would be to dump them there if at all possible.

Many thanks,
Craig

#2

Hi Craig,

Since Docker containers all share the kernel of the host node, core file location is mediated by the kernel core_pattern setting for the host node and is not per-container. You can investigate how this might be set on your host node (it differs somewhat between Linux distributions, but for many may be set via /proc/sys/kernel/core_pattern.)

We could also investigate options to prevent the Qserv containers from generating core files at all, but this begs the question: how often is the system dumping core at your site, and can we assist in identifying the underlying problem?

cheers,
–FritzM.

#3

One note: wherever the core file is dumped, you can mount a host filesystem volume (or network volume) at that spot to avoid filling the container’s filesystem. But as Fritz says, not having a core dump at all is the best solution :slight_smile:

#4

Happy new year.

Thank you both for replying.
Thankfully it appears this has just been a once-off, but if I run into it again I can let you know if I need some assistance in narrowing down what the cause might be.

A bit more Googling led me to this: https://unix.stackexchange.com/questions/192716/how-to-set-the-core-dump-file-location-and-name
Which mentions that it is possible to set a core file size limit which might also be useful.

Cheers,
Craig