ScyllaDB Docs Scylla Operator Management Monitoring Setting up ScyllaDB Monitoring on OpenShift

Caution

You're viewing documentation for an unstable version of Scylla Operator. Switch to the latest stable version.

Setting up ScyllaDB Monitoring on OpenShift¶

This guide will walk you through setting up a monitoring stack for your ScyllaDB clusters using the ScyllaDBMonitoring custom resource and an external Prometheus instance that is already deployed in your Kubernetes cluster in an OpenShift cluster using User Workload Monitoring (UWM).

The guide assumes you have read the overview and setup of ScyllaDB monitoring and are familiar with the concepts of Prometheus and Grafana.

Requirements¶

This guide assumes you have Scylla Operator and a ScyllaCluster already installed in your OpenShift cluster. For more information on how to deploy Scylla Operator, see the installation guide.

Note

The Scylla Operator installation process on OpenShift is the same as on vanilla Kubernetes. However, unlike Kubernetes, OpenShift includes a built-in Prometheus Operator and User Workload Monitoring (UWM) for user workloads. Therefore, instead of deploying Prometheus using ScyllaDBMonitoring, we configure it to use the external Prometheus instance provided by OpenShift UWM.

We also assume you have the oc CLI tool installed and configured to access your OpenShift cluster, and have the necessary permissions to create ServiceAccounts and ClusterRoleBindings.

Enable User Workload Monitoring in OpenShift¶

OpenShift provides a built-in Prometheus instance that can be used for monitoring user workloads. To use this Prometheus instance, you need to enable User Workload Monitoring in your OpenShift cluster. You can do this by following the official OpenShift documentation.

Configure OpenShift metrics access for Grafana datasource¶

Create ServiceAccount and ClusterRoleBinding¶

To allow ScyllaDBMonitoring-managed Grafana to access the OpenShift User Workload Monitoring Prometheus instance, you need to create a ServiceAccount and a ClusterRoleBinding that grants the necessary permissions. We will use its ServiceAccount token for configuring Grafana datasource. See the OpenShift’s Accessing metrics as a developer article for more details.

Note

We assume you have ScyllaCluster deployed in scylla namespace/project. Replace scylla with your namespace/project name if it’s different.

You can create the ServiceAccount and ClusterRoleBinding using the following commands:

oc create serviceaccount scylla-grafana-monitoring-viewer
oc create clusterrolebinding scylla-monitoring-grafana-cluster-monitoring-view --clusterrole=cluster-monitoring-view --serviceaccount=scylla:scylla-grafana-monitoring-viewer

Create ServiceAccount token Secret¶

Next, you need to create a Secret that contains the ServiceAccount token. The following manifest will create such a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: scylla-monitoring-grafana-token
  namespace: scylla
  annotations:
    kubernetes.io/service-account.name: scylla-grafana-monitoring-viewer
type: kubernetes.io/service-account-token

You can create it using the following command:

kubectl apply -n scylla -f https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/monitoring/v1alpha1/openshift/sa-token.secret.yaml

The Secret will be populated with the token automatically by Kubernetes and should be available under the token key of this Secret. Verify it by running:

kubectl -n scylla get secret scylla-monitoring-grafana-token -o=jsonpath='{.data.token}'

You should see the encoded token printed to the console.

Create service CA certificate ConfigMap¶

OpenShift uses a self-signed CA to sign the certificates for its internal services. To allow Grafana to trust the OpenShift UWM Prometheus instance, you need to create a ConfigMap that, when properly annotated, will be populated with the OpenShift service CA certificate. Please refer to the OpenShift documentation for more details on this mechanism.

The following manifest will create such a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-openshift-service-ca
  annotations:
    "service.beta.openshift.io/inject-cabundle": "true"

You can create it using the following command:

kubectl apply -n scylla -f https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/monitoring/v1alpha1/openshift/service-ca.configmap.yaml

You can verify that the ConfigMap has been populated with the service CA certificate under the service-ca.crt key by running:

kubectl -n scylla get configmap example-openshift-service-ca -o=jsonpath='{.data.service-ca\.crt}'

You should see the PEM-encoded CA certificate printed to the console.

Deploy ScyllaDBMonitoring¶

The following ScyllaDBMonitoring configuration will set up the monitoring stack to use the OpenShift UWM Prometheus instance as an external Prometheus datasource for Grafana:

apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBMonitoring
metadata:
  name: example
  namespace: scylla
spec:
  type: Platform
  endpointsSelector:
    matchLabels:
      app.kubernetes.io/name: scylla
      scylla-operator.scylladb.com/scylla-service-type: member
      scylla/cluster: scylla
  components:
    prometheus:
      mode: External
    grafana:
      datasources:
        # Prometheus datasource pointing to OpenShift's Thanos Querier service.
        # To make this work, `cluster-monitoring-config` ConfigMap in `openshift-monitoring` namespace must be configured
        # to contain `config.yaml` key with `enableUserWorkload: true` in its content.
        # See https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/monitoring/configuring-user-workload-monitoring#enabling-monitoring-for-user-defined-projects_preparing-to-configure-the-monitoring-stack-uwm for details.
        - type: Prometheus
          url: "https://thanos-querier.openshift-monitoring.svc:9091"
          prometheusOptions:
            tls:
              caCertConfigMapRef:
                # This is the ConfigMap reference for OpenShift's injected Service CA bundle.
                name: example-openshift-service-ca
                key: service-ca.crt
            auth:
              type: BearerToken
              bearerTokenOptions:
                secretRef:
                  # This is a `kubernetes.io/service-account-token` type of Secret created for a ServiceAccount bound to
                  # `cluster-monitoring-view` ClusterRole.
                  name: scylla-monitoring-grafana-token
                  key: token

You can apply it using kubectl:

kubectl apply -n scylla --server-side -f=https://raw.githubusercontent.com/scylladb/scylla-operator/master/examples/monitoring/v1alpha1/openshift/uwm.scylladbmonitoring.yaml

See the Setting up ScyllaDBMonitoring guide for more details on deploying ScyllaDBMonitoring.

Verify the setup¶

You can verify that configuration is correct by accessing Grafana and verifying you can see metrics from your ScyllaDB cluster.

Was this page helpful?