ScyllaDB Docs Scylla Operator Support Gathering data with must-gather

Gathering data with must-gather¶

must-gather is an embedded tool in Scylla Operator that helps collecting all the necessary info when something goes wrong. The tool talks to the Kubernetes API, retrieves a predefined set of resources and saves them into a folder in your current directory.

By default, sensitive resources (Secrets and bitnami.com.SealedSecrets) are omitted from the collection. If you come across a clear need to include them (not recommended unless the default settings are deemed insufficient for investigation), please refer to the include sensitive resources section below. If you have other resources that you wouldn’t like to include in the collection, refer to the excluding resources section below.

Given that must-gather needs to talk to the Kubernetes API, at the very least, you need to supply the --kubeconfig flag with a path to the kubeconfig file for your Kubernetes cluster, or set the KUBECONFIG environment variable.

Running must-gather¶

There is more than one way to run must-gather. Here are some examples of how you can run the tool.

Prerequisites¶

All examples assume you have exported KUBECONFIG environment variable that points to a kubeconfig file on your machine. If not, you can run this command to export the common default location. Please make sure such a file exists.

export KUBECONFIG=~/.kube/config
ls -l "${KUBECONFIG}"

Note

There can be slight deviations in the arguments for your container tool, depending on the container runtime, whether you use SELinux or similar factors.

As an example, the need for the Z option on volume mounts depends on whether you use SELinux and what context is applied on your file or directory. If you get an error mentioning Error: lsetxattr <path>: operation not supported, try it without the Z option.

Let’s also check whether your kubeconfig uses external authentication plugin. You can determine that by running

kubectl config view --minify

and checking whether it uses an external exec plugin by looking for this pattern (containing the exec key)

users:
- name: <user_name>
  user:
    exec:

If not, you can skip the rest of this section.

In case your kubeconfig depends on external binaries, you have to take a few extra steps because the external binary won’t be available within our container to authenticate the requests.

Similarly to how Pods are run within Kubernetes, we’ll create a dedicated ServiceAccount for must-gather and use it to run the tool. (When you are done using it, feel free to remove the Kubernetes resources created for that purpose.)

kubectl create namespace must-gather
kubectl -n must-gather create serviceaccount must-gather
kubectl create clusterrolebinding must-gather --clusterrole=cluster-admin --serviceaccount=must-gather:must-gather
export MUST_GATHER_TOKEN
MUST_GATHER_TOKEN=$( kubectl -n must-gather create token must-gather --duration=1h )
kubeconfig=$( mktemp )
# Create a copy of the existing kubeconfig and
# replace user authentication using yq, or by adjusting the fields manually.
kubectl config view --minify --raw -o yaml | yq -e '.users[0].user = {"token": env(MUST_GATHER_TOKEN)}' > "${kubeconfig}"
KUBECONFIG="${kubeconfig}"

Note

If you don’t have yq installed, you can get it at https://github.com/mikefarah/yq/#install or you can replace the user authentication settings manually.

Podman¶

podman run -it --pull=always --rm -v="${KUBECONFIG}:/kubeconfig:ro,Z" -v="$( pwd ):/workspace:Z" --workdir=/workspace docker.io/scylladb/scylla-operator:latest must-gather --kubeconfig=/kubeconfig

Docker¶

docker run -it --pull=always --rm -v="${KUBECONFIG}:/kubeconfig:ro" -v="$( pwd ):/workspace" --workdir=/workspace docker.io/scylladb/scylla-operator:latest must-gather --kubeconfig=/kubeconfig

Limiting must-gather to a particular namespace¶

If you are running a large Kubernetes cluster with many ScyllaClusters, it may be useful to limit the collection of ScyllaClusters to a particular namespace. Unless you hit scale issues, we advise not to use this mode, as sometimes the ScyllaClusters affect other collected resources, like the manager or they form a multi-datacenter.

scylla-operator must-gather --namespace="<namespace_with_broken_scyllacluster>"

Note

The --namespace flag affects only ScyllaClusters. Other resources related to the operator installation or cluster state will still be collected from other namespaces.

Collecting every resource in the cluster¶

By default, must-gather collects only a predefined subset of resources. You can also request collecting every resource in the Kubernetes API, if the default set wouldn’t be enough to debug an issue.

scylla-operator must-gather --all-resources

Including sensitive resources¶

By default, must-gather omits sensitive resources like Secrets or bitnami.com.SealedSecrets. You can turn this behavior off by passing the --include-sensitive-resources flag.

scylla-operator must-gather --all-resources --include-sensitive-resources

Excluding resources¶

If you have resources that you don’t want to include in the collection other than the default ones, you can exclude them by passing the --exclude-resource flag multiple times.

Note

The format for the resource is kind (for core resources) or kind.group. Examples: LimitRange or DeviceClass.resource.k8s.io.

scylla-operator must-gather --all-resources --exclude-resource="LimitRange" --exclude-resource="DeviceClass.resource.k8s.io"

Was this page helpful?