ScyllaDB Docs ScyllaDB Operator Troubleshoot Collect core dumps

For AI agents: a documentation index is available at https://operator.docs.scylladb.com/master/llms.txt. A Markdown version of this page is at https://operator.docs.scylladb.com/master/troubleshoot/configure-coredumps.md.

Caution

You're viewing documentation for an unstable version of ScyllaDB Operator. Switch to the latest stable version.

Collect core dumps¶

This guide explains how to configure core dump collection on Kubernetes nodes running ScyllaDB Operator-managed ScyllaDB clusters and how to retrieve the resulting dump files.

Background¶

Core dump handling is controlled by kernel.core_pattern (see Linux man page). In Kubernetes, writing dumps to an absolute path inside the container means they are lost on pod restart. We recommend piping dumps through the systemd-coredump tool, which stores them on the host filesystem independently of pod lifetime.

Platform requirements¶

Collecting a core dump requires the following prerequisites on the Kubernetes worker node where the process expected to crash is scheduled. To make things simpler, it can be done on all worker nodes. This must be completed before the anticipated crash.

systemd-coredump installed - the helper binary that receives the core image from the kernel and writes it to disk.
/etc/systemd/coredump.conf configured - controls storage location (Storage=external), compression (Compress=yes), and disk space limits (MaxUse, KeepFree, ProcessSizeMax, ExternalSizeMax).
kernel.core_pattern set to pipe crashes through systemd-coredump.
systemd-coredump.socket active.

A ready-to-use setup for GKE, OKE and OpenShift is provided below. On other platforms, apply these four steps using the OS package manager and systemd tooling available on the node.

Set up core dump collection¶

The manifests below handle all four setup steps via a single container on each ScyllaDB node. The container performs the setup once at startup and then sleeps, keeping the pod alive so that the DaemonSet re-applies the settings whenever the pod is evicted or rescheduled.

Oracle Linux nodes on OKE and RHCOS nodes on OpenShift ship systemd-coredump with systemd. GKE Ubuntu nodes do not, so the container installs it with apt-get. The container sets kernel.core_pattern from the /usr/lib/sysctl.d/50-coredump.conf file shipped with the node’s systemd-coredump, because the arguments it expects differ between systemd versions.

1. Create the ConfigMap¶

# Recommended systemd-coredump configuration for nodes running ScyllaDB.
#
# Apply this ConfigMap alongside the setup-systemd-coredump DaemonSet so that
# the setup container writes these settings to /etc/systemd/coredump.conf on
# each node before activating kernel.core_pattern.
#
# Key tuning choices (adjust to your environment, refer to `man 5 coredump.conf`):
#   Storage=external  - write core dump files to /var/lib/systemd/coredump/
#                      (as opposed to the systemd journal or tmpfs).
#   Compress=yes      - compress dumps with zstd (saves significant disk space).
#   ProcessSizeMax=0 - do not truncate core dumps; ScyllaDB needs full cores.
#   ExternalSizeMax=0
#   MaxUse=20G        - cap the total space used for stored dumps.
#   KeepFree=10G      - always leave at least this much free on the target filesystem.
#
# IMPORTANT: ScyllaDB processes may allocate hundreds of gigabytes of memory.
# Even compressed core dumps can be very large. Adjust MaxUse and KeepFree to
# match the size of your host boot disk (or a dedicated volume if you mount one
# at /var/lib/systemd/coredump/).
apiVersion: v1
kind: ConfigMap
metadata:
  name: scylladb-coredump-conf
  namespace: scylla-operator
  labels:
    app.kubernetes.io/name: scylladb-coredump-setup
data:
  coredump.conf: |
    [Coredump]
    # Store core dumps as files on the host filesystem.
    Storage=external

    # Compress stored core dumps using zstd.
    Compress=yes

    # Do not truncate core dumps. ScyllaDB requires full core images for analysis.
    ProcessSizeMax=0
    ExternalSizeMax=0  
  
    # Maximum total disk space to use for all stored core dumps.
    # Increase if your nodes have larger disks or if you expect many simultaneous crashes.
    MaxUse=20G

    # Always keep at least this much disk space free on the target filesystem.
    KeepFree=10G

Download the manifest and edit MaxUse and KeepFree to match your environment before applying - see Storage considerations.

curl -fLO https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/common/coredumps/coredump-conf.configmap.yaml

vi coredump-conf.configmap.yaml

kubectl apply --server-side -f=coredump-conf.configmap.yaml

2. Create the ServiceAccount¶

The setup DaemonSet runs under a dedicated ServiceAccount.

kubectl apply --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/common/coredumps/setup-systemd-coredump.serviceaccount.yaml

3. Grant the privileged SCC (OpenShift only)¶

This step is only required on OpenShift.

The setup container runs privileged with access to the host, which on OpenShift requires the privileged SecurityContextConstraints. Grant it to the ServiceAccount:

# The setup DaemonSet runs privileged Pods with host PID and a hostPath mount.
# On OpenShift, this requires the privileged SecurityContextConstraints.
# On other platforms, this rule has no effect.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: scylladb-coredump-setup
  namespace: scylla-operator
  labels:
    app.kubernetes.io/name: scylladb-coredump-setup
rules:
- apiGroups:
  - security.openshift.io
  resources:
  - securitycontextconstraints
  resourceNames:
  - privileged
  verbs:
  - use

kubectl apply --server-side \
  -f=https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/common/coredumps/setup-systemd-coredump.role.yaml \
  -f=https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/common/coredumps/setup-systemd-coredump.rolebinding.yaml

4. Deploy the setup DaemonSet¶

# This DaemonSet configures systemd-coredump on nodes running ScyllaDB so that
# core dumps are captured and stored on the host filesystem at
# /var/lib/systemd/coredump/.
#
# A single long-running container performs the following setup steps:
#   1. Install systemd-coredump if the host doesn't ship it. Only the
#      installation method depends on the host operating system.
#   2. Apply the recommended /etc/systemd/coredump.conf configuration.
#   3. Set kernel.core_pattern to pipe core dumps through systemd-coredump.
#   4. Start systemd-coredump.socket so the helper can connect to it when a
#      crash occurs.
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: scylladb-coredump-setup
  namespace: scylla-operator
  labels:
    app.kubernetes.io/name: scylladb-coredump-setup
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: scylladb-coredump-setup
  template:
    metadata:
      labels:
        app.kubernetes.io/name: scylladb-coredump-setup
    spec:
      # Target only the nodes that run ScyllaDB workloads.
      nodeSelector:
        scylla.scylladb.com/node-type: scylla
      tolerations:
      - key: scylla-operator.scylladb.com/dedicated
        operator: Equal
        value: scyllaclusters
        effect: NoSchedule
      serviceAccountName: scylladb-coredump-setup
      # hostPID is required so that "nsenter -t 1" targets the host's systemd
      # (PID 1) rather than the container's init process. This is needed for
      # systemctl to communicate with the host's D-Bus and start
      # systemd-coredump.socket.
      hostPID: true
      containers:
      - name: setup-coredump
        image: docker.io/library/ubuntu:24.04
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/setup-complete
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:
            cpu: 1m
            memory: 32Mi
          limits:
            cpu: 100m
            memory: 128Mi
        command:
        - /bin/bash
        - -euEo
        - pipefail
        - -O
        - inherit_errexit
        - -c
        - |
          # Run a command inside the host's mount + UTS namespaces using nsenter
          # so that package managers and sysctl operate on the real host.
          host_exec() {
            nsenter --mount=/host/proc/1/ns/mnt --uts=/host/proc/1/ns/uts -- "$@"
          }

          # Some host operating systems ship systemd-coredump with systemd
          # (e.g. Oracle Linux on OKE, RHCOS on OpenShift), others need it installed
          # (e.g. Ubuntu on GKE).
          if host_exec test -x /usr/lib/systemd/systemd-coredump; then
            echo "systemd-coredump is already installed on the host."
          else
            HOST_OS_ID="$(host_exec sh -c '. /etc/os-release && echo "${ID}"')"
            # Add a case for each host operating system that needs systemd-coredump installed.
            case "${HOST_OS_ID}" in
            ubuntu)
              echo "Installing systemd-coredump via apt-get..."
              host_exec apt-get update -y -qq
              host_exec apt-get install -y -qq systemd-coredump
              ;;
            *)
              echo "systemd-coredump is not installed on the host and installing it on ${HOST_OS_ID@Q} is not supported." >&2
              exit 1
              ;;
            esac
          fi

          # Apply the coredump configuration from the mounted ConfigMap.
          if [ -f /config/coredump.conf ]; then
            echo "Applying custom /etc/systemd/coredump.conf..."
            cp /config/coredump.conf /host/etc/systemd/coredump.conf
            nsenter -t 1 --mount --uts --ipc --net -- systemctl daemon-reload || true
          fi

          # The arguments systemd-coredump expects in kernel.core_pattern differ
          # between systemd versions, so use the sysctl configuration shipped
          # with the host's systemd-coredump.
          SYSCTL_CONF='/usr/lib/sysctl.d/50-coredump.conf'
          echo "Applying ${SYSCTL_CONF}..."
          host_exec sysctl -p "${SYSCTL_CONF}"

          # Pass a large hardcoded value in place of %c (the core file size rlimit
          # of the crashing process) to prevent truncation. Newer systemd versions
          # already ship it this way.
          CORE_PATTERN="$(host_exec sysctl -n kernel.core_pattern)"
          if [[ "${CORE_PATTERN}" != *systemd-coredump* ]]; then
            echo "Unexpected kernel.core_pattern after applying ${SYSCTL_CONF}: ${CORE_PATTERN@Q}" >&2
            exit 1
          fi
          host_exec sysctl -w "kernel.core_pattern=${CORE_PATTERN//%c/9223372036854775808}"

          echo "kernel.core_pattern is now:"
          host_exec sysctl -n kernel.core_pattern

          # Start systemd-coredump.socket on the host so the helper can hand off
          # the core image for processing. Without an active socket the helper
          # exits silently and no core file is written.
          # nsenter -t 1 with mount+UTS+IPC+net namespaces makes the host binaries
          # and the D-Bus socket visible to systemctl.
          echo "Starting systemd-coredump.socket on the host..."
          nsenter -t 1 --mount --uts --ipc --net -- systemctl start systemd-coredump.socket
          echo "systemd-coredump.socket is now active."

          # Keep the pod running so that the DaemonSet re-applies the settings
          # on eviction or reschedule.
          echo "Setup complete. Sleeping indefinitely..."
          touch /tmp/setup-complete
          exec sleep infinity
        volumeMounts:
        - name: host
          mountPath: /host
        - name: coredump-config
          mountPath: /config
          readOnly: true
      volumes:
      - name: host
        hostPath:
          path: /
          type: Directory
      - name: coredump-config
        configMap:
          name: scylladb-coredump-conf
  updateStrategy:
    type: RollingUpdate

kubectl apply --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/common/coredumps/setup-systemd-coredump.daemonset.yaml

Wait for the DaemonSet to roll out on all ScyllaDB nodes:

kubectl -n scylla-operator rollout status daemonset/scylladb-coredump-setup

5. Verify the configuration¶

After the DaemonSet rolls out, confirm kernel.core_pattern is correctly set on each node. List the dedicated ScyllaDB nodes:

kubectl get nodes -l scylla.scylladb.com/node-type=scylla -o name

Run the following command for each node, replacing <node-name> with the actual name:

kubectl debug node/<node-name> -it --profile=sysadmin --image=docker.io/library/ubuntu:24.04 -- \
  nsenter --mount=/proc/1/ns/mnt -- sysctl -n kernel.core_pattern

The output must start with |/usr/lib/systemd/systemd-coredump. The arguments that follow depend on the node’s systemd version, e.g. on GKE Ubuntu nodes:

|/usr/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d

Also verify that systemd-coredump.socket is active:

kubectl debug node/<node-name> -it --profile=sysadmin --image=docker.io/library/ubuntu:24.04 -- \
  nsenter --mount=/proc/1/ns/mnt -- systemctl is-active systemd-coredump.socket

The output must be active.

Verify that core dump collection works end to end¶

The steps below trigger a test crash of a running ScyllaDB process and confirm the dump was captured by systemd-coredump.

Warning

This procedure intentionally crashes a ScyllaDB node. Only run it when the cluster can tolerate losing one member temporarily.

1. Find the pod and its node¶

NAMESPACE=<namespace>
kubectl get pods -n "${NAMESPACE}" -l scylla-operator.scylladb.com/pod-type=scylladb-node -o wide

Store the pod name and the node it is scheduled on:

POD_NAME=<pod-name>
NODE_NAME=<node-name>

2. Trigger the crash¶

Inside a pod managed by ScyllaDB Operator, the sidecar is PID 1 and the scylla binary runs as a child process. Send a SIGABRT signal to the scylla process to trigger a crash and core dump:

kubectl exec -n "${NAMESPACE}" "${POD_NAME}" -c scylla -- sh -c 'kill -ABRT $(pgrep -x scylla)'

ScyllaDB logs a backtrace and terminates. The pod stays running because the ScyllaDB Operator sidecar (PID 1) is unaffected; ScyllaDB Operator will restart the ScyllaDB process automatically. The dump is written to the node’s host filesystem before the process exits.

3. Confirm the dump was captured¶

Confirm the dump was captured using coredumpctl list - see Retrieving core dumps from nodes for details.

Retrieve core dumps from nodes¶

Core dumps are stored at /var/lib/systemd/coredump/ on the host.

1. List available dumps¶

kubectl debug "node/${NODE_NAME}" -it --profile=sysadmin --image=docker.io/library/ubuntu:24.04 -- \
  nsenter --mount=/proc/1/ns/mnt -- coredumpctl list

Store the PID of the desired dump from the output:

DUMP_PID=<pid>

2. Export a specific dump¶

Start a debug pod on the node so that we can use kubectl exec to retrieve the dump file:

kubectl debug "node/${NODE_NAME}" --profile=sysadmin --image=docker.io/library/ubuntu:24.04 -- sleep 3600

Store the debug pod name:

DEBUG_POD_NAME=<debug-pod-name>

Pull the dump file from the node to your local machine (it can be very large, so this may take some time):

kubectl exec "${DEBUG_POD_NAME}" -- \
    nsenter --mount=/proc/1/ns/mnt -- coredumpctl dump "${DUMP_PID}" \
    > scylla.core

You can verify the dump with file scylla.core - it should show ELF 64-bit LSB core file.

Storage considerations¶

Take into account that ScyllaDB core dumps can be very large. You will need spare disk space larger than that of ScyllaDB’s RAM. Core dump storage is controlled by the [Coredump] section of /etc/systemd/coredump.conf.

Note

systemd-coredump will automatically delete the oldest dump files when the MaxUse or KeepFree thresholds are exceeded, so some dumps may be lost if a node generates many crashes in a short period of time and the disk is nearly full.

To avoid losing dumps due to insufficient disk space, consider the following:

Attach a dedicated disk to each ScyllaDB node at /var/lib/systemd/coredump/ so core dumps do not compete with the OS for disk space.
Offload dumps to object storage - the IBM core-dump-handler project provides a Helm chart that installs a similar kernel.core_pattern pipe handler and automatically uploads dumps to an S3-compatible bucket. This is a good option if you need centralized, long-term dump storage.

Was this page helpful?