ScyllaDB Docs ScyllaDB Operator Deploy ScyllaDB Set up monitoring Set up ScyllaDB Monitoring

Set up ScyllaDB Monitoring¶

This guide will walk you through setting up a complete monitoring stack for your ScyllaDB clusters using the ScyllaDBMonitoring custom resource and an external Prometheus instance.

The guide assumes you have read the overview of ScyllaDB monitoring and are familiar with the concepts of Prometheus and Grafana. It doesn’t cover every possible configuration option, but it will highlight the most important ones. For a complete reference of all configuration options, see the ScyllaDBMonitoring API reference.

Requirements¶

Before you can set up your ScyllaDB monitoring, you need the following components installed in your Kubernetes cluster:

Prometheus Operator - must be installed before ScyllaDB Operator. ScyllaDB Operator detects Prometheus Operator CRDs (monitoring.coreos.com/v1) at startup and enables its monitoring controller only when they are present. You can install Prometheus Operator using the bundled manifest:
```
kubectl apply --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/v1.21/examples/third-party/prometheus-operator.yaml
```
If you install Prometheus Operator after ScyllaDB Operator, you must restart the ScyllaDB Operator for detection to take effect.
```
kubectl rollout restart -n scylla-operator deployment/scylla-operator
```
ScyllaDB Operator - for more information on how to deploy it, see the installation guide.
A ScyllaCluster - the ScyllaDB cluster you want to monitor.

Deploy external Prometheus¶

Note

In this guide, we will deploy an external Prometheus instance that will be managed outside of ScyllaDBMonitoring by the Prometheus Operator. If you already have an existing Prometheus instance in your cluster that you want to use, you can skip this step.

Create ServiceAccount for Prometheus¶

Prometheus needs a ServiceAccount to operate. You can create it using the following manifest:

kubectl create serviceaccount prometheus -n scylla

Create ClusterRole and ClusterRoleBinding for Prometheus¶

Prometheus needs certain permissions to operate. You can create a ClusterRole and a ClusterRoleBinding that will bind the role to the ServiceAccount created above using the following manifests:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/metrics
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - configmaps
    verbs: ["get"]
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: scylla

Apply them using the following commands:

kubectl apply -n scylla --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/v1.21/examples/monitoring/v1alpha1/prometheus.clusterrole.yaml
kubectl apply -n scylla --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/v1.21/examples/monitoring/v1alpha1/prometheus.clusterrolebinding.yaml

Create Service for Prometheus¶

Prometheus needs a Service to be accessible within the cluster. You can create it using the following manifest:

apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  type: ClusterIP
  selector:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/instance: prometheus
      prometheus: prometheus
  ports:
    - name: web
      protocol: TCP
      port: 9090

Apply it using the following command:

kubectl apply -n scylla --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/v1.21/examples/monitoring/v1alpha1/prometheus.service.yaml

Deploy Prometheus instance¶

When deploying a Prometheus, you need to ensure serviceAccountName and serviceName are set correctly to the values created above:

spec:
  serviceAccountName: "prometheus"
  serviceName: "prometheus"

You also need to ensure that your Prometheus instance is configured to discover and scrape the ServiceMonitor and PrometheusRule resources created by ScyllaDBMonitoring. This requires setting the serviceMonitorSelector and ruleSelector in the Prometheus resource to match the labels used by ScyllaDBMonitoring, like so:

serviceMonitorSelector:
  matchLabels:
    scylla-operator.scylladb.com/scylladbmonitoring-name: "example"
ruleSelector:
  matchLabels:
    scylla-operator.scylladb.com/scylladbmonitoring-name: "example"

Monitor multiple ScyllaClusters with a single Prometheus¶

If you have more than one ScyllaCluster in your Kubernetes cluster, and you want to monitor all of them using the same Prometheus instance, you can customize the selectors to match ServiceMonitors and PrometheusRules created for multiple ScyllaClusters.

Assuming you’re going to create two ScyllaDBMonitoring objects named cluster-1-monitoring and cluster-2-monitoring for two different ScyllaClusters in two distinct namespaces, you can set the selectors like this so that the Prometheus instance will scrape both of them:

serviceMonitorNamespaceSelector: {} # Select ServiceMonitors in all namespaces.
serviceMonitorSelector: # Select ServiceMonitors created by multiple ScyllaDBMonitoring instances.
  matchExpressions:
    - key: scylla-operator.scylladb.com/scylladbmonitoring-name
      operator: In
      values:
        - "cluster-1-monitoring"
        - "cluster-2-monitoring"
ruleNamespaceSelector: {} # Select Rules in all namespaces.
ruleSelector: # Select Rules created by multiple ScyllaDBMonitoring instances.
  matchExpressions:
    - key: scylla-operator.scylladb.com/scylladbmonitoring-name
      operator: In
      values:
        - "cluster-1-monitoring"
        - "cluster-2-monitoring"

In this case, we’re setting serviceMonitorNamespaceSelector and ruleNamespaceSelector to an empty selector, which means that Prometheus will look for ServiceMonitors and PrometheusRules in all namespaces.

Warning

If you want to monitor multiple ScyllaClusters using a single Prometheus instance, ensure all selected ScyllaClusters have unique names across the entire Kubernetes cluster (not just within their namespaces). Our monitoring setup cannot currently distinguish metrics collected from ScyllaClusters with colliding names, even if they are in different namespaces.

Please refer to the Prometheus Operator documentation for more details on how to configure selectors in the Prometheus specification.

Deploy Prometheus¶

You can deploy a Prometheus instance by executing the following command:

kubectl apply -n scylla --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/v1.21/examples/monitoring/v1alpha1/prometheus.yaml

Wait for Prometheus to roll out¶

kubectl -n scylla rollout status --timeout=5m statefulset.apps/prometheus-prometheus

Configure ScyllaDBMonitoring¶

To set up the monitoring stack, create a ScyllaDBMonitoring object.

Note

If you want to customize the ScyllaDBMonitoring configuration, we will explain the important configuration options in the following sections.

Endpoints selector¶

endpointsSelector:
  matchLabels:
    app.kubernetes.io/name: scylla
    scylla-operator.scylladb.com/scylla-service-type: member
    scylla/cluster: scylla # Replace with your ScyllaCluster name.

This field is used to select the ScyllaDB nodes that will be monitored. In matchLabels you should specify the labels that match the ScyllaDB Services’ labels that are added by ScyllaDB Operator:

scylla-operator.scylladb.com/scylla-service-type: member - This is a static label that’s always added to each ScyllaDB node Service.
scylla/cluster - This label’s value depends on your ScyllaCluster name. Replace it with the actual name of your ScyllaCluster.

Prometheus configuration¶

By default, ScyllaDBMonitoring deploys a Prometheus instance for you (Managed mode). However, this mode is deprecated, and we recommend using an existing Prometheus instance instead (External mode). External mode is used in the example manifest.

components:
  prometheus:
    mode: External

Grafana configuration¶

You can configure various aspects of the Grafana instance that will be deployed for you under spec.components.grafana). You can set resource requests and limits (under resources), persistency (under storage), workload placement (under placement), Grafana authentication (under authentication), and serving certificate (under servingCertSecretName).

If you use the External Prometheus mode, you should configure a datasource (under datasources) that points to your existing Prometheus instance. In the datasource configuration, you should use the internal Kubernetes Service name of your Prometheus instance as the URL (http://prometheus.scylla.svc.cluster.local:9090 in this example). Depending on your Prometheus configuration, you may also need to configure TLS (under tls) and authentication (under auth).

datasources:
  - name: prometheus
    type: Prometheus
    url: http://prometheus.scylla.svc.cluster.local:9090
    prometheusOptions: {}

Deploy ScyllaDBMonitoring¶

Deploy the monitoring using kubectl:

kubectl apply -n scylla --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/v1.21/examples/monitoring/v1alpha1/scylladbmonitoring.yaml

ScyllaDB Operator will notice the new ScyllaDBMonitoring object, and it will reconcile all necessary resources.

Wait for ScyllaDBMonitoring to roll out¶

kubectl wait -n scylla --timeout=5m --for='condition=Available=True' scylladbmonitorings.scylla.scylladb.com/example

Note

ScyllaDBMonitoring’s Available condition will be set to True when all managed resources are created. However, it doesn’t guarantee that all resources are fully rolled out. You should still wait for Grafana to be fully rolled out before accessing it.

Wait for Grafana to roll out¶

kubectl rollout status -n scylla --timeout=5m deployments.apps/example-grafana

At this point, you should have a fully functional monitoring stack for your ScyllaDB cluster.

To learn how to access Grafana, see the Expose Grafana guide.

Was this page helpful?