# Upgrading ScyllaDB Operator ScyllaDB Operator supports N+1 upgrades only. That means to you can only update by 1 minor version at the time and wait for it to successfully roll out and then update all ScyllaClusters that also run using the image that’s being updated. (ScyllaDB Operator injects it as a sidecar to help run and manage ScyllaDB.) We value the stability of our APIs and all API changes are backwards compatible. ## Upgrade via GitOps (kubectl) A typical upgrade flow using GitOps (kubectl) requires re-applying the manifests using ones from the release you want to upgrade to. If ScyllaDB Manager is installed, you need to upgrade it manually by re-applying its manifests using ones from the release you want to upgrade to. Please refer to the [ScyllaDB Manager installation instructions](https://operator.docs.scylladb.com/stable/deploy-scylladb/install-scylladb-manager.md) for details. Note that ScyllaDB Operator’s dependencies also need to be updated to the versions compatible with the target ScyllaDB Operator version. Please refer to the [GitOps installation instructions](https://operator.docs.scylladb.com/stable/install-operator/install-with-gitops.md) for details. ## Upgrade via Helm ### Prerequisites - Update ScyllaDB Operator dependencies to the versions compatible with the target ScyllaDB Operator version. Refer to the [Helm installation instructions](https://operator.docs.scylladb.com/stable/install-operator/install-with-helm.md) for details. - Make sure Helm chart repository is up-to-date: ```default helm repo add scylla https://scylla-operator-charts.storage.googleapis.com/stable helm repo update ``` ### Upgrade ScyllaDB Manager Replace `` with the name of your Helm release for ScyllaDB Manager and replace `` with the version number you want to install: ```default helm upgrade --version scylla/scylla-manager ``` ### Upgrade ScyllaDB Operator Replace `` with the name of your Helm release for ScyllaDB Operator and replace `` with the version number you want to install: 1. Update CRD resources. We recommend using `--server-side` flag for `kubectl apply`, if your version supports it. ```default tmpdir=$( mktemp -d ) \ && helm pull scylla-operator/scylla-operator --version --untar --untardir "${tmpdir}" \ && find "${tmpdir}"/scylla-operator/crds/ -name '*.yaml' -printf '-f=%p ' \ | xargs kubectl apply ``` 2. Update ScyllaDB Operator. ```default helm upgrade --version scylla/scylla-operator ``` ## Upgrade steps for specific versions ### 1.22 to 1.23 #### Ensure no healthy node carries a leftover decommissioned label A stale `scylla/decommissioned=true` label makes the upgraded Operator delete a healthy node’s data, so check before upgrading. List the member Services that carry the label in all namespaces: ```bash kubectl get services --all-namespaces -l scylla/decommissioned=true ``` If the command prints `No resources found`, your clusters are unaffected. Otherwise, for every Service listed, check whether the node of the same name is healthy: its Pod is ready, and `nodetool status` run on another node of the cluster lists its host ID as `UN`. The host ID is in the `internal.scylla-operator.scylladb.com/host-id` annotation of the Service. If the node is healthy, the label is stale. Remove it before upgrading: ```bash kubectl -n label service scylla/decommissioned- ``` If the node was in fact decommissioned, meaning its Pod isn’t ready and its host ID is no longer in `nodetool status`, leave the label in place. The upgraded Operator removes the node. #### NOTE **Why is this necessary?** Starting with v1.23, the Operator treats a member Service labelled `scylla/decommissioned=true` as a node that has left the cluster. It removes the node together with its PVC before the rack reconciles to the requested node count. Previous versions only acted on the label during a scale-down, so a stale label could sit on a healthy node without consequences. On a rack whose node at ordinal N carries a stale label, the upgraded Operator decommissions every node above N, one at a time or all at once with `enableParallelNodeOperations` enabled. It then removes node N together with its PVC without decommissioning it, and bootstraps the rack back to the requested count with new, empty nodes. A healthy node N loses its data this way, and the Operator rebuilds every node above it. Previous versions could leave the label behind when you reverted a scale-down before the Operator removed the node’s Service, or when you recovered a decommissioned node by hand. #### Choose the certificate key type Check whether your Operator already sets the key type: ```bash kubectl -n scylla-operator get deployment/scylla-operator -o=jsonpath='{.spec.template.spec.containers[?(@.name=="scylla-operator")].args}{"\n"}{.spec.template.spec.containers[?(@.name=="scylla-operator")].env}{"\n"}' ``` The first line of the output lists the arguments of the Operator container, and the second one its environment variables. In a default installation, neither sets the key type: ```console ["operator","--loglevel=2"] [{"name":"SCYLLA_OPERATOR_IMAGE","value":"docker.io/scylladb/scylla-operator:1.22"}] ``` | The output contains | Your certificates | After the upgrade | |----------------------------------------------------------------------------------------|---------------------|--------------------------| | `--crypto-key-type=RSA`, or `SCYLLA_OPERATOR_CRYPTO_KEY_TYPE` with the value `RSA` | RSA | Unchanged | | `--crypto-key-type=ECDSA`, or `SCYLLA_OPERATOR_CRYPTO_KEY_TYPE` with the value `ECDSA` | ECDSA | Unchanged | | Neither, as in a default installation | RSA | Replaced with ECDSA ones | The deprecated `--crypto-key-size` flag only sets the RSA key size and doesn’t keep the Operator on RSA. If your certificates are RSA, decide before upgrading whether to [keep RSA certificates]() or [switch to ECDSA certificates](). #### NOTE **Why switch to ECDSA?** With ECDSA certificates, ScyllaDB spends much less CPU on each TLS connection that clients open, so connections are set up faster, especially when many clients reconnect at once, for example after an application restart. The Operator also spends much less CPU generating keys, and doesn’t make you wait for key generation when it needs many certificates at once, for example when you create several clusters. ##### Keep RSA certificates Set `--crypto-key-type=RSA` before upgrading, and keep it in the configuration you upgrade with. ScyllaDB Operator v1.22 accepts the flag, and since RSA is its default, setting it doesn’t change any certificates. Helm Add the flag to your values file, next to any other additional arguments you already pass: ```yaml additionalArgs: - --crypto-key-type=RSA ``` GitOps Add the flag to the arguments of the `scylla-operator` container in the `scylla-operator` Deployment of the manifests you apply: ```yaml args: - operator - --loglevel=2 - --crypto-key-type=RSA ``` OLM Add an environment variable to the `config` section of your `Subscription`: ```yaml spec: config: env: - name: SCYLLA_OPERATOR_CRYPTO_KEY_TYPE value: RSA ``` To switch to ECDSA certificates later, remove the flag and follow the steps below. ##### Switch to ECDSA certificates After the upgrade, the Operator reissues every certificate that still has an RSA key, CAs included, and leaves certificates that already have ECDSA keys unchanged. ScyllaDB then serves a certificate signed by the new serving CA, which clients holding a copy of the previous serving CA bundle don’t trust. Plan the upgrade for a time when you can update your clients right after it: 1. Upgrade the Operator. 2. Wait for the serving CA of each `ScyllaCluster` to be reissued with an ECDSA key: ```bash NAMESPACE= CLUSTER_NAME= kubectl -n="${NAMESPACE}" get "secret/${CLUSTER_NAME}-local-serving-ca" --template='{{ index .data "tls.crt" }}' | base64 -d | openssl x509 -noout -text | grep 'Public Key Algorithm' ``` The output shows `id-ecPublicKey` once the CA is reissued: ```console Public Key Algorithm: id-ecPublicKey ``` 3. Give your clients the new serving CA bundles: - CQL clients: the `-local-serving-ca` ConfigMap, as described in [Connect via CQL](https://operator.docs.scylladb.com/stable/connect-your-app/connect-via-cql.md). - Alternator clients: the `-alternator-local-serving-ca` ConfigMap, as described in [Alternator](https://operator.docs.scylladb.com/stable/connect-your-app/alternator.md). - Grafana clients: the `-grafana-serving-ca` ConfigMap of the `ScyllaDBMonitoring`, unless you provide Grafana’s serving certificate yourself with `servingCertSecretName`. Clients that [mount the serving CA bundle ConfigMap](https://operator.docs.scylladb.com/stable/connect-your-app/connect-via-cql.md#mount-tls-files-in-application-pods) pick it up by themselves. 4. Give your clients the reissued client certificates, for example the admin client certificate from the `-local-user-admin` Secret. The `-local-cql-connection-configs-admin` Secret embeds the admin client certificate and key together with the serving CA bundle, so clients that use it get both at once. See [Certificate rotation](https://operator.docs.scylladb.com/stable/understand/security.md#certificate-rotation) for how the Operator reissues certificates. #### Enable parallel node operations on existing clusters #### NOTE This step is recommended, not required. Starting with v1.23, [parallel node operations](https://operator.docs.scylladb.com/stable/operate/scale-add-remove-racks.md#sequential-and-parallel-node-operations) also cover scaling down: the Operator decommissions all the leaving nodes of a rack at once instead of one at a time. Together with starting nodes in parallel, this makes scaling a cluster considerably faster, and the difference grows with the number of nodes. Enabling parallel node operations also avoids a [known issue](https://operator.docs.scylladb.com/stable/troubleshoot/recover-from-stuck-scale-down.md) in which the Pods of the leaving nodes may never be deleted. List your `ScyllaClusters` together with their ScyllaDB version and whether parallel node operations are enabled: ```bash kubectl get scyllaclusters --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.version,PARALLEL_NODE_OPERATIONS:.spec.enableParallelNodeOperations' ``` ```console NAMESPACE NAME VERSION PARALLEL_NODE_OPERATIONS legacy old-cluster 2026.1.0 scylla analytics 2026.3.1 scylla basic 2026.3.1 true scylla staging 2026.3.1 false ``` Already existing clusters created before v1.22 show `` unless you set the field explicitly. `` defaults to `false`. Enable parallel node operations on every cluster that runs ScyllaDB 2026.2 or later and doesn’t show `true`: ```bash kubectl -n patch scyllacluster --type=merge -p '{"spec":{"enableParallelNodeOperations":true}}' ``` Changing the field doesn’t restart the running nodes. #### Remove ScyllaDBCluster and RemoteKubernetesCluster objects and CRDs The experimental `ScyllaDBCluster` and `RemoteKubernetesCluster` APIs and their controllers are removed in v1.23, together with the internal `RemoteOwner` API. `ScyllaDBCluster` is also no longer a supported value of `spec.scyllaDBClusterRef.kind` in `ScyllaDBManagerTask` and `ScyllaDBManagerClusterRegistration`. Multi-datacenter ScyllaDB clusters remain supported as before, with one `ScyllaCluster` per datacenter, as described in [Deploy a multi-datacenter ScyllaDB cluster](https://operator.docs.scylladb.com/stable/deploy-scylladb/deploy-multi-datacenter-cluster.md). Complete steps 1 to 4 below **before upgrading**, while the previous Operator version is still running. #### NOTE **Why does this have to happen before the upgrade?** All of these objects carry a finalizer that only their controller removes, and the upgraded Operator no longer runs those controllers. If you upgrade with any of them left, the Operator no longer cleans up anything related to them, and you have to remove the objects, their finalizers and everything they created in the remote Kubernetes clusters by hand. ##### 1. Delete the ScyllaDBManagerTask objects referencing a ScyllaDBCluster List the tasks together with the kind they reference: ```bash kubectl get scylladbmanagertasks.scylla.scylladb.com --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,REF_KIND:.spec.scyllaDBClusterRef.kind,REF_NAME:.spec.scyllaDBClusterRef.name' ``` ```console NAMESPACE NAME REF_KIND REF_NAME scylla daily-repair ScyllaDBDatacenter basic scylla legacy-repair ScyllaDBCluster dev-cluster ``` Delete every task whose `REF_KIND` is `ScyllaDBCluster`, `legacy-repair` in the example above: ```bash kubectl -n delete scylladbmanagertask.scylla.scylladb.com ``` Deleting them now also removes the corresponding tasks from ScyllaDB Manager. Tasks referencing a `ScyllaDBDatacenter` are unaffected and must be kept. ##### 2. Delete the ScyllaDBCluster objects ```bash kubectl delete scylladbclusters.scylla.scylladb.com --all --all-namespaces ``` The Operator removes the `ScyllaDBManagerClusterRegistration` objects it created for these clusters and deregisters them from ScyllaDB Manager, so you don’t need to delete those yourself. This happens asynchronously, after the `ScyllaDBCluster` is gone, and step 4 checks that it completed. ##### 3. Delete the RemoteKubernetesCluster objects ```bash kubectl delete remotekubernetesclusters.scylla.scylladb.com --all ``` ##### 4. Verify that nothing is left, then upgrade ```bash kubectl get scylladbclusters.scylla.scylladb.com,remotekubernetesclusters.scylla.scylladb.com --all-namespaces kubectl get scylladbmanagertasks.scylla.scylladb.com,scylladbmanagerclusterregistrations.scylla.scylladb.com --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,KIND:.kind,NAME:.metadata.name,REF_KIND:.spec.scyllaDBClusterRef.kind' ``` The first command must print `No resources found`, and no row of the second one may show `ScyllaDBCluster` in `REF_KIND`. ##### 5. After upgrading, remove the CRDs Neither `kubectl apply` nor `helm upgrade` removes CRDs, so delete them explicitly in the cluster running the Operator: ```bash kubectl delete crd scylladbclusters.scylla.scylladb.com remotekubernetesclusters.scylla.scylladb.com remoteowners.scylla.scylladb.com ``` Every Kubernetes cluster that was registered as a `RemoteKubernetesCluster` also has the `remoteowners.scylla.scylladb.com` CRD and the ClusterRoles the Operator used to access it. Remove them in each of those clusters: ```bash kubectl --context="${REMOTE_CLUSTER_CONTEXT}" delete crd remoteowners.scylla.scylladb.com kubectl --context="${REMOTE_CLUSTER_CONTEXT}" delete clusterrole scylladb:controller:operator-remote scylladb:controller:aggregate-to-operator-remote ``` ### 1.20 to 1.21 #### Ensure ScyllaCluster repair and backup task names are RFC 1123 compliant Before upgrading, ensure that all `ScyllaCluster` repair (`.spec.repairs[].name`) and backup (`.spec.backups[].name`) task names conform to [RFC 1123 subdomain](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-subdomain-names) requirements: - contain no more than 253 characters, - contain only lowercase alphanumeric characters, ‘-’ or ‘.’, - start with an alphanumeric character, - end with an alphanumeric character. You can run the following snippet to check for such `ScyllaClusters`: ```bash output=$(kubectl get scyllaclusters --all-namespaces -o json | jq -r ' def is_rfc1123_subdomain_invalid: (length <= 253) and test("^[a-z0-9]([a-z0-9.-]{0,251}[a-z0-9])?$") | not; .items[] | { namespace: .metadata.namespace, name: .metadata.name, invalid_repairs: [(.spec.repairs // [] | .[].name | select(is_rfc1123_subdomain_invalid))], invalid_backups: [(.spec.backups // [] | .[].name | select(is_rfc1123_subdomain_invalid))] } | select((.invalid_repairs | length > 0) or (.invalid_backups | length > 0)) | "\(.namespace)/\(.name)\n Invalid repairs: \(if (.invalid_repairs | length) > 0 then (.invalid_repairs | join(", ")) else "(none)" end)\n Invalid backups: \(if (.invalid_backups | length) > 0 then (.invalid_backups | join(", ")) else "(none)" end)\n" ') && if [ -z "$output" ]; then echo "All ScyllaCluster repair and backup task names are RFC 1123 compliant."; else echo "$output"; fi ``` You should get an output similar to the following if there are any `ScyllaClusters` with invalid repair or backup task names: ```console scylla/example Invalid repairs: invalid_repair Invalid backups: invalid_backup ``` or the following if all `ScyllaCluster` repair and backup task names are compliant: ```console All ScyllaCluster repair and backup task names are RFC 1123 compliant. ``` #### NOTE **Why is this necessary?** Starting with v1.20.1, ScyllaDB Operator emitted warnings for `ScyllaCluster` repair and backup task names not conforming to RFC 1123 subdomain requirements. In v1.21, these warnings have been replaced with hard validation errors, and the operator will refuse to start if any existing `ScyllaClusters` have non-conforming task names. #### Ensure ScyllaCluster spec.version is not empty `ScyllaCluster` `spec.version` is now a required field. Any create or update request with an empty value will be rejected by the admission webhook. You can run the following snippet to check whether any of your existing `ScyllaClusters` have an empty `spec.version`: ```bash kubectl get scyllaclusters --all-namespaces -o json | jq -r ' .items[] | select((.spec.version // "") == "") | "\(.metadata.namespace)/\(.metadata.name)" ' ``` If the command returns no output, all your `ScyllaClusters` are unaffected. If any are listed, set their `spec.version` to a valid ScyllaDB image tag before upgrading. #### NOTE **Why is this not a breaking change?** A `ScyllaCluster` with an empty `spec.version` was never functional. The migration controller would fail to reconcile it, leaving the cluster in a permanently degraded state. Making the field required only prevents new broken clusters from being created. #### Review ScyllaDBMonitoring spec.type default change The default value of `ScyllaDBMonitoring` `spec.type` has changed from `SaaS` to `Platform`. Any existing `ScyllaDBMonitoring` object that omits `spec.type` will render `Platform` dashboards after the upgrade instead of `SaaS`. The `SaaS` value is also now deprecated and will be removed in a future release; the admission webhook will emit a warning when it is set explicitly. You can run the following snippet to list `ScyllaDBMonitoring` objects that will be affected by the default change or that still use the deprecated `SaaS` value: ```bash output=$(kubectl get scylladbmonitorings --all-namespaces -o json | jq -r ' .items[] | select((.spec.type // "SaaS") == "SaaS") | "\(.metadata.namespace)/\(.metadata.name)\t(spec.type=\(.spec.type // ""))" ') && if [ -z "$output" ]; then echo "No ScyllaDBMonitoring objects rely on the SaaS type."; else echo "$output"; fi ``` If the command returns no output, none of your `ScyllaDBMonitoring` objects are affected. Otherwise, for each listed object, decide whether you want to: - keep the previous behavior by setting `spec.type: SaaS` explicitly before upgrading (the admission webhook will emit a deprecation warning, and you will need to migrate to `Platform` before a future release removes `SaaS`), or - adopt the new default by either setting `spec.type: Platform` explicitly or leaving `spec.type` unset. #### NOTE **Why is this not a breaking change?** The `Platform` dashboards are a superset of the `SaaS` dashboards, so switching from `SaaS` to `Platform` does not remove functionality. ### 1.17 to 1.18 Upgrading from v1.17.x requires extra actions due to the removal of the standalone ScyllaDB Manager controller. The controller becomes an integral part of ScyllaDB Operator. As a result, the standalone ScyllaDB Manager controller deployment and its related resources need to be removed before upgrading ScyllaDB Operator. #### Upgrade via GitOps (kubectl) ```default kubectl delete -n scylla-manager \ clusterrole/scylladb:controller:aggregate-to-manager-controller \ clusterrole/scylladb:controller:manager-controller \ poddisruptionbudgets.policy/scylla-manager-controller \ serviceaccounts/scylla-manager-controller \ clusterrolebindings.rbac.authorization.k8s.io/scylladb:controller:manager-controller \ deployments.apps/scylla-manager-controller ``` #### Upgrade via Helm ScyllaDB Manager Helm installation has to be upgraded before upgrading ScyllaDB Operator Helm installation, following the standard [Helm upgrade procedure](). This will ensure that the ScyllaDB Manager Controller is removed before upgrading the ScyllaDB Operator.