# ScyllaDB Monitoring overview ## Architecture ScyllaDB [exposes](https://monitoring.docs.scylladb.com/stable/reference/monitoring-apis.html) its metrics in the Prometheus format. ScyllaDB Operator provides the [`ScyllaDBMonitoring`](https://operator.docs.scylladb.com/v1.22/reference/api/groups/scylla.scylladb.com/scylladbmonitorings.md) custom resource that allows you to set up a complete monitoring stack for your ScyllaDB clusters based on the following components: - [**Prometheus**](https://prometheus.io) for metrics collection and alerting (scraping ScyllaDB and host-level metrics, and alerting rules targeting ScyllaDB instances). - [**Grafana**](https://grafana.com) for metrics visualization (with pre-configured dashboards for ScyllaDB). #### NOTE The `ScyllaDBMonitoring` CRD is still in its `v1alpha1` version, yet it is considered stable and ready for production use, with the following caveats: - `spec.components.grafana.exposeOptions` and `spec.components.prometheus.exposeOptions` are deprecated and will be removed in the next API version, - the **Managed** mode is deprecated; the **External** mode for Prometheus is likely to be the only supported mode in the next API version. ## Prometheus For deploying and/or configuring Prometheus, ScyllaDB Operator relies on [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator). ScyllaDB Operator supports two modes of operation for `ScyllaDBMonitoring` regarding Prometheus deployment: **Managed** and **External**. You can choose the mode that best fits your needs by setting the `spec.components.prometheus.mode` field in the `ScyllaDBMonitoring` resource. Depending on the mode chosen, ScyllaDB Operator may deploy and manage a Prometheus instance for you, or it can be configured to use an existing Prometheus instance (managed by the Prometheus Operator) in your cluster. The following Prometheus Operator resources are created by ScyllaDB Operator when you deploy a `ScyllaDBMonitoring` resource: - [`Prometheus`](https://github.com/prometheus-operator/prometheus-operator/blob/e4c727291acc543dab531bc4aaf16637067c1b86/pkg/apis/monitoring/v1/prometheus_types.go#L1085) - the Prometheus instance itself (it may be omitted in External mode). - [`ServiceMonitor`](https://github.com/prometheus-operator/prometheus-operator/blob/e4c727291acc543dab531bc4aaf16637067c1b86/pkg/apis/monitoring/v1/servicemonitor_types.go#L41) - the resource that defines how to scrape metrics from ScyllaDB nodes. - [`PrometheusRule`](https://github.com/prometheus-operator/prometheus-operator/blob/e4c727291acc543dab531bc4aaf16637067c1b86/pkg/apis/monitoring/v1/prometheusrule_types.go#L37) - the resource that defines alerting rules for Prometheus. The Prometheus version used in the deployment is tied to the version of ScyllaDB Operator. You can find the exact version used in the [`config.yaml`](https://github.com/scylladb/scylla-operator/blob/master/assets/config/config.yaml) file under `operator.prometheusVersion` key. Alongside ScyllaDB’s own metrics, Prometheus also collects metrics from `scylladb-node-exporter`, ScyllaDB’s packaging of the [Prometheus node_exporter agent](https://github.com/prometheus/node_exporter), exposing host/OS-level metrics (CPU, memory, disk I/O, network, filesystem) from every ScyllaDB Pod. This is what feeds the OS-level dashboards in ScyllaDB Monitoring’s Grafana. For ScyllaDB version 2026.3 or later, ScyllaDB Operator deploys a dedicated `scylladb-node-exporter` sidecar Container inside the Pod; for earlier versions, the exporter runs as a process inside the main ScyllaDB Container. ### External The **External** mode plugs into an existing Prometheus that is managed by Prometheus Operator (and therefore can be configured by `ServiceMonitor`). In the **External** mode, ScyllaDB Operator will not deploy a `Prometheus` resource (will instead expect a running Prometheus in the cluster already), but it will still create the `ServiceMonitor` and `PrometheusRule` resources that an existing Prometheus Operator in your cluster will use to configure your Prometheus instance. This mode is useful if you already have a Prometheus instance deployed in your cluster, and you want to use it for monitoring your ScyllaDB clusters. If you don’t have Prometheus deployed in your cluster, you need to deploy one to proceed. When using this mode, you need to ensure that the existing Prometheus instance is configured to discover and scrape the `ServiceMonitor` and `PrometheusRule` resources created by ScyllaDB Operator. Please refer to the [Setting up ScyllaDB Monitoring](https://operator.docs.scylladb.com/v1.22/deploy-scylladb/set-up-monitoring/setup.md#deploy-prometheus-instance) guide for more details. Please note that in this mode, `ScyllaDBMonitoring` has to be configured so that Grafana can access the Prometheus instance. You can configure Grafana datasources in the `spec.components.grafana.datasources` field of the `ScyllaDBMonitoring` resource. Please refer to the [`ScyllaDBMonitoring` API reference](https://operator.docs.scylladb.com/v1.22/reference/api/groups/scylla.scylladb.com/scylladbmonitorings.md) for details. ### Managed #### NOTE This mode is deprecated and will be removed in a future version. Instead, please deploy your own Prometheus and use **External**. In the **Managed** mode, ScyllaDB Operator will deploy a Prometheus instance for you. This is the default mode. What this means is that when you create a `ScyllaDBMonitoring` resource, ScyllaDB Operator will create a `Prometheus` resource (from the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) in the same namespace as the `ScyllaDBMonitoring` resource. This Prometheus instance will be configured to scrape metrics from the ScyllaDB nodes in the cluster that `ScyllaDBMonitoring` is monitoring and will also have alerting rules configured for ScyllaDB (using `ServiceMonitor` and `PrometheusRule` CRs). ### Node exporter resource boundaries and limitations Because `scylladb-node-exporter` runs within the ScyllaDB Pod’s namespaces rather than directly on the host, the metrics it exposes reflect these isolation boundaries: | Domain | Namespace isolation | Metric effect and limitations | |-----------------------------------|------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **CPU, memory, and system state** | Reads host-level `/proc` and `/sys` filesystems without cgroup-scoped constraints. | Metrics (like CPU, memory, disk I/O, and vmstat) report global Node-wide resource usage rather than Pod-level resource limits. When running multiple ScyllaDB Pods on the same Node, their exporters report identical and duplicated Node-level values. | | **Network** | Restricted to the Pod’s network namespace. | Network and TCP metrics reflect the Pod’s virtual ethernet interface (`veth`) and namespace activity, not the physical host network interface cards (NICs). Host-level NIC ethtool allowance counters are unavailable and return zero. | | **Storage and filesystem** | Restricted to the Container’s mount namespace. | Filesystem metrics report capacity and usage for paths mounted inside the Container (such as the `/var/lib/scylla` volume). They do not reflect the physical host’s mount namespace, meaning the host’s actual root partition and other host-level mounts are completely invisible. | ## Grafana For deploying Grafana, ScyllaDB Operator doesn’t use any third-party operator. Instead, it manages the Grafana deployment directly. It preconfigures Grafana with dashboards from [scylla-monitoring](https://github.com/scylladb/scylla-monitoring/). The Grafana image used in the deployment is tied to the version of ScyllaDB Operator. You can find the exact image used in the [config.yaml](https://github.com/scylladb/scylla-operator/blob/master/assets/config/config.yaml) file under `operator.grafanaImage` key. ### Expose Grafana ScyllaDB Operator creates a `ClusterIP` Service named `-grafana` for each `ScyllaDBMonitoring`. You can access it outside the cluster using your preferred method: - Port forwarding using `kubectl port-forward` command for temporary access. - Using [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) or [Gateway API](https://gateway-api.sigs.k8s.io/) resources (e.g., HTTPRoute) for production access. You can learn more about exposing Grafana in the [Expose Grafana](https://operator.docs.scylladb.com/v1.22/deploy-scylladb/set-up-monitoring/exposing-grafana.md) guide. ## Related pages - [Set up ScyllaDB Monitoring](https://operator.docs.scylladb.com/v1.22/deploy-scylladb/set-up-monitoring/setup.md) — deploy Prometheus and configure `ScyllaDBMonitoring` for your cluster. - [Set up ScyllaDB Monitoring on OpenShift](https://operator.docs.scylladb.com/v1.22/deploy-scylladb/set-up-monitoring/external-prometheus-on-openshift.md) — use OpenShift User Workload Monitoring as an external Prometheus source. - [Expose Grafana](https://operator.docs.scylladb.com/v1.22/deploy-scylladb/set-up-monitoring/exposing-grafana.md) — make the Grafana dashboard accessible outside the cluster.