ScyllaDB Docs ScyllaDB Operator Upgrade Upgrading ScyllaDB Operator

For AI agents: a documentation index is available at https://operator.docs.scylladb.com/master/llms.txt. A Markdown version of this page is at https://operator.docs.scylladb.com/master/upgrade/upgrade-operator.md.

Caution

You're viewing documentation for an unstable version of ScyllaDB Operator. Switch to the latest stable version.

Upgrading ScyllaDB Operator¶

ScyllaDB Operator supports N+1 upgrades only. That means to you can only update by 1 minor version at the time and wait for it to successfully roll out and then update all ScyllaClusters that also run using the image that’s being updated. (ScyllaDB Operator injects it as a sidecar to help run and manage ScyllaDB.)

We value the stability of our APIs and all API changes are backwards compatible.

Caution

If any additional steps are required for a specific version upgrade, they will be documented in the Upgrade steps for specific versions section below.

Upgrade via GitOps (kubectl)¶

A typical upgrade flow using GitOps (kubectl) requires re-applying the manifests using ones from the release you want to upgrade to.

If ScyllaDB Manager is installed, you need to upgrade it manually by re-applying its manifests using ones from the release you want to upgrade to. Please refer to the ScyllaDB Manager installation instructions for details.

Note that ScyllaDB Operator’s dependencies also need to be updated to the versions compatible with the target ScyllaDB Operator version.

Please refer to the GitOps installation instructions for details.

Upgrade via Helm¶

Prerequisites¶

Update ScyllaDB Operator dependencies to the versions compatible with the target ScyllaDB Operator version. Refer to the Helm installation instructions for details.

Make sure Helm chart repository is up-to-date:

helm repo add scylla https://scylla-operator-charts.storage.googleapis.com/stable
helm repo update

Upgrade ScyllaDB Manager¶

Replace <release_name> with the name of your Helm release for ScyllaDB Manager and replace <version> with the version number you want to install:

helm upgrade --version <version> <release_name> scylla/scylla-manager

Upgrade ScyllaDB Operator¶

Replace <release_name> with the name of your Helm release for ScyllaDB Operator and replace <version> with the version number you want to install:

Update CRD resources. We recommend using --server-side flag for kubectl apply, if your version supports it.

tmpdir=$( mktemp -d ) \
  && helm pull scylla-operator/scylla-operator --version <version> --untar --untardir "${tmpdir}" \
  && find "${tmpdir}"/scylla-operator/crds/ -name '*.yaml' -printf '-f=%p ' \
  | xargs kubectl apply

Update ScyllaDB Operator.

helm upgrade --version <version> <release_name> scylla/scylla-operator

Upgrade steps for specific versions¶

Caution

The below instructions are supplementary to the standard upgrade procedure and don’t fully replace it. Make sure to familiarize yourself with both the standard upgrade procedure and the additional steps for the specific version you are upgrading to, if applicable, and follow them accordingly.

1.22 to 1.23¶

Ensure no healthy node carries a leftover decommissioned label¶

A stale scylla/decommissioned=true label makes the upgraded Operator delete a healthy node’s data, so check before upgrading. List the member Services that carry the label in all namespaces:

kubectl get services --all-namespaces -l scylla/decommissioned=true

If the command prints No resources found, your clusters are unaffected. Otherwise, for every Service listed, check whether the node of the same name is healthy: its Pod is ready, and nodetool status run on another node of the cluster lists its host ID as UN. The host ID is in the internal.scylla-operator.scylladb.com/host-id annotation of the Service.

If the node is healthy, the label is stale. Remove it before upgrading:

kubectl -n <namespace> label service <service-name> scylla/decommissioned-

If the node was in fact decommissioned, meaning its Pod isn’t ready and its host ID is no longer in nodetool status, leave the label in place. The upgraded Operator removes the node.

Note

Why is this necessary? Starting with v1.23, the Operator treats a member Service labelled scylla/decommissioned=true as a node that has left the cluster. It removes the node together with its PVC before the rack reconciles to the requested node count. Previous versions only acted on the label during a scale-down, so a stale label could sit on a healthy node without consequences. On a rack whose node at ordinal N carries a stale label, the upgraded Operator decommissions every node above N, one at a time or all at once with enableParallelNodeOperations enabled. It then removes node N together with its PVC without decommissioning it, and bootstraps the rack back to the requested count with new, empty nodes. A healthy node N loses its data this way, and the Operator rebuilds every node above it. Previous versions could leave the label behind when you reverted a scale-down before the Operator removed the node’s Service, or when you recovered a decommissioned node by hand.

Choose the certificate key type¶

Caution

Skipping this step can cause downtime for your applications. Starting with v1.23, the Operator issues ECDSA P-384 certificates by default instead of RSA ones. Unless it runs with --crypto-key-type=RSA, the upgraded Operator replaces every RSA certificate it manages with an ECDSA one right after the upgrade, CAs included, in all ScyllaClusters and ScyllaDBMonitorings. Clients that verify ScyllaDB against a copy of the serving CA bundle can’t connect until they get the new bundle.

Check whether your Operator already sets the key type:

kubectl -n scylla-operator get deployment/scylla-operator -o=jsonpath='{.spec.template.spec.containers[?(@.name=="scylla-operator")].args}{"\n"}{.spec.template.spec.containers[?(@.name=="scylla-operator")].env}{"\n"}'

The first line of the output lists the arguments of the Operator container, and the second one its environment variables. In a default installation, neither sets the key type:

["operator","--loglevel=2"]
[{"name":"SCYLLA_OPERATOR_IMAGE","value":"docker.io/scylladb/scylla-operator:1.22"}]

The output contains	Your certificates	After the upgrade
`--crypto-key-type=RSA`, or `SCYLLA_OPERATOR_CRYPTO_KEY_TYPE` with the value `RSA`	RSA	Unchanged
`--crypto-key-type=ECDSA`, or `SCYLLA_OPERATOR_CRYPTO_KEY_TYPE` with the value `ECDSA`	ECDSA	Unchanged
Neither, as in a default installation	RSA	Replaced with ECDSA ones

The deprecated --crypto-key-size flag only sets the RSA key size and doesn’t keep the Operator on RSA.

If your certificates are RSA, decide before upgrading whether to keep RSA certificates or switch to ECDSA certificates.

Note

Why switch to ECDSA? With ECDSA certificates, ScyllaDB spends much less CPU on each TLS connection that clients open, so connections are set up faster, especially when many clients reconnect at once, for example after an application restart. The Operator also spends much less CPU generating keys, and doesn’t make you wait for key generation when it needs many certificates at once, for example when you create several clusters.

Keep RSA certificates¶

Set --crypto-key-type=RSA before upgrading, and keep it in the configuration you upgrade with. ScyllaDB Operator v1.22 accepts the flag, and since RSA is its default, setting it doesn’t change any certificates.

Add the flag to your values file, next to any other additional arguments you already pass:

additionalArgs:
- --crypto-key-type=RSA

Add the flag to the arguments of the scylla-operator container in the scylla-operator Deployment of the manifests you apply:

args:
- operator
- --loglevel=2
- --crypto-key-type=RSA

Add an environment variable to the config section of your Subscription:

spec:
  config:
    env:
    - name: SCYLLA_OPERATOR_CRYPTO_KEY_TYPE
      value: RSA

To switch to ECDSA certificates later, remove the flag and follow the steps below.

Switch to ECDSA certificates¶

After the upgrade, the Operator reissues every certificate that still has an RSA key, CAs included, and leaves certificates that already have ECDSA keys unchanged. ScyllaDB then serves a certificate signed by the new serving CA, which clients holding a copy of the previous serving CA bundle don’t trust. Plan the upgrade for a time when you can update your clients right after it:

Upgrade the Operator.

Wait for the serving CA of each ScyllaCluster to be reissued with an ECDSA key:

NAMESPACE=<namespace>
CLUSTER_NAME=<cluster-name>
kubectl -n="${NAMESPACE}" get "secret/${CLUSTER_NAME}-local-serving-ca" --template='{{ index .data "tls.crt" }}' | base64 -d | openssl x509 -noout -text | grep 'Public Key Algorithm'

The output shows id-ecPublicKey once the CA is reissued:

            Public Key Algorithm: id-ecPublicKey

Give your clients the new serving CA bundles:
- CQL clients: the <cluster-name>-local-serving-ca ConfigMap, as described in Connect via CQL.
- Alternator clients: the <cluster-name>-alternator-local-serving-ca ConfigMap, as described in Alternator.
- Grafana clients: the <scylladbmonitoring-name>-grafana-serving-ca ConfigMap of the ScyllaDBMonitoring, unless you provide Grafana’s serving certificate yourself with servingCertSecretName.
Clients that mount the serving CA bundle ConfigMap pick it up by themselves.
Give your clients the reissued client certificates, for example the admin client certificate from the <cluster-name>-local-user-admin Secret. The <cluster-name>-local-cql-connection-configs-admin Secret embeds the admin client certificate and key together with the serving CA bundle, so clients that use it get both at once.

See Certificate rotation for how the Operator reissues certificates.

Enable parallel node operations on existing clusters¶

Note

This step is recommended, not required.

Starting with v1.23, parallel node operations also cover scaling down: the Operator decommissions all the leaving nodes of a rack at once instead of one at a time. Together with starting nodes in parallel, this makes scaling a cluster considerably faster, and the difference grows with the number of nodes. Enabling parallel node operations also avoids a known issue in which the Pods of the leaving nodes may never be deleted.

List your ScyllaClusters together with their ScyllaDB version and whether parallel node operations are enabled:

kubectl get scyllaclusters --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.version,PARALLEL_NODE_OPERATIONS:.spec.enableParallelNodeOperations'

NAMESPACE   NAME          VERSION    PARALLEL_NODE_OPERATIONS
legacy      old-cluster   2026.1.0   <none>
scylla      analytics     2026.3.1   <none>
scylla      basic         2026.3.1   true
scylla      staging       2026.3.1   false

Already existing clusters created before v1.22 show <none> unless you set the field explicitly. <none> defaults to false.

Enable parallel node operations on every cluster that runs ScyllaDB 2026.2 or later and doesn’t show true:

kubectl -n <namespace> patch scyllacluster <name> --type=merge -p '{"spec":{"enableParallelNodeOperations":true}}'

Changing the field doesn’t restart the running nodes.

Remove ScyllaDBCluster and RemoteKubernetesCluster objects and CRDs¶

The experimental ScyllaDBCluster and RemoteKubernetesCluster APIs and their controllers are removed in v1.23, together with the internal RemoteOwner API. ScyllaDBCluster is also no longer a supported value of spec.scyllaDBClusterRef.kind in ScyllaDBManagerTask and ScyllaDBManagerClusterRegistration. Multi-datacenter ScyllaDB clusters remain supported as before, with one ScyllaCluster per datacenter, as described in Deploy a multi-datacenter ScyllaDB cluster.

Complete steps 1 to 4 below before upgrading, while the previous Operator version is still running.

Note

Why does this have to happen before the upgrade? All of these objects carry a finalizer that only their controller removes, and the upgraded Operator no longer runs those controllers. If you upgrade with any of them left, the Operator no longer cleans up anything related to them, and you have to remove the objects, their finalizers and everything they created in the remote Kubernetes clusters by hand.

1. Delete the ScyllaDBManagerTask objects referencing a ScyllaDBCluster¶

List the tasks together with the kind they reference:

kubectl get scylladbmanagertasks.scylla.scylladb.com --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,REF_KIND:.spec.scyllaDBClusterRef.kind,REF_NAME:.spec.scyllaDBClusterRef.name'

NAMESPACE   NAME            REF_KIND             REF_NAME
scylla      daily-repair    ScyllaDBDatacenter   basic
scylla      legacy-repair   ScyllaDBCluster      dev-cluster

Delete every task whose REF_KIND is ScyllaDBCluster, legacy-repair in the example above:

kubectl -n <namespace> delete scylladbmanagertask.scylla.scylladb.com <name>

Deleting them now also removes the corresponding tasks from ScyllaDB Manager. Tasks referencing a ScyllaDBDatacenter are unaffected and must be kept.

2. Delete the ScyllaDBCluster objects¶

kubectl delete scylladbclusters.scylla.scylladb.com --all --all-namespaces

Caution

Deleting a ScyllaDBCluster deletes the namespaces the Operator created for its datacenters in the remote Kubernetes clusters, together with the ScyllaDBDatacenters, their PersistentVolumeClaims and the data on them. Back up the data first if you need to keep it.

The Operator removes the ScyllaDBManagerClusterRegistration objects it created for these clusters and deregisters them from ScyllaDB Manager, so you don’t need to delete those yourself. This happens asynchronously, after the ScyllaDBCluster is gone, and step 4 checks that it completed.

3. Delete the RemoteKubernetesCluster objects¶

kubectl delete remotekubernetesclusters.scylla.scylladb.com --all

4. Verify that nothing is left, then upgrade¶

kubectl get scylladbclusters.scylla.scylladb.com,remotekubernetesclusters.scylla.scylladb.com --all-namespaces
kubectl get scylladbmanagertasks.scylla.scylladb.com,scylladbmanagerclusterregistrations.scylla.scylladb.com --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,KIND:.kind,NAME:.metadata.name,REF_KIND:.spec.scyllaDBClusterRef.kind'

The first command must print No resources found, and no row of the second one may show ScyllaDBCluster in REF_KIND.

5. After upgrading, remove the CRDs¶

Neither kubectl apply nor helm upgrade removes CRDs, so delete them explicitly in the cluster running the Operator:

kubectl delete crd scylladbclusters.scylla.scylladb.com remotekubernetesclusters.scylla.scylladb.com remoteowners.scylla.scylladb.com

Every Kubernetes cluster that was registered as a RemoteKubernetesCluster also has the remoteowners.scylla.scylladb.com CRD and the ClusterRoles the Operator used to access it. Remove them in each of those clusters:

kubectl --context="${REMOTE_CLUSTER_CONTEXT}" delete crd remoteowners.scylla.scylladb.com
kubectl --context="${REMOTE_CLUSTER_CONTEXT}" delete clusterrole scylladb:controller:operator-remote scylladb:controller:aggregate-to-operator-remote

1.20 to 1.21¶

Ensure ScyllaCluster repair and backup task names are RFC 1123 compliant¶

Before upgrading, ensure that all ScyllaCluster repair (.spec.repairs[].name) and backup (.spec.backups[].name) task names conform to RFC 1123 subdomain requirements:

contain no more than 253 characters,
contain only lowercase alphanumeric characters, ‘-’ or ‘.’,
start with an alphanumeric character,
end with an alphanumeric character.

You can run the following snippet to check for such ScyllaClusters:

output=$(kubectl get scyllaclusters --all-namespaces -o json | jq -r '
  def is_rfc1123_subdomain_invalid:
    (length <= 253) and test("^[a-z0-9]([a-z0-9.-]{0,251}[a-z0-9])?$") | not;
  .items[] |
  {
    namespace: .metadata.namespace,
    name: .metadata.name,
    invalid_repairs: [(.spec.repairs // [] | .[].name | select(is_rfc1123_subdomain_invalid))],
    invalid_backups: [(.spec.backups // [] | .[].name | select(is_rfc1123_subdomain_invalid))]
  } |
  select((.invalid_repairs | length > 0) or (.invalid_backups | length > 0)) |
  "\(.namespace)/\(.name)\n  Invalid repairs: \(if (.invalid_repairs | length) > 0 then (.invalid_repairs | join(", ")) else "(none)" end)\n  Invalid backups: \(if (.invalid_backups | length) > 0 then (.invalid_backups | join(", ")) else "(none)" end)\n"
') && if [ -z "$output" ]; then echo "All ScyllaCluster repair and backup task names are RFC 1123 compliant."; else echo "$output"; fi

You should get an output similar to the following if there are any ScyllaClusters with invalid repair or backup task names:

scylla/example
  Invalid repairs: invalid_repair
  Invalid backups: invalid_backup

or the following if all ScyllaCluster repair and backup task names are compliant:

All ScyllaCluster repair and backup task names are RFC 1123 compliant.

Note

Why is this necessary? Starting with v1.20.1, ScyllaDB Operator emitted warnings for ScyllaCluster repair and backup task names not conforming to RFC 1123 subdomain requirements. In v1.21, these warnings have been replaced with hard validation errors, and the operator will refuse to start if any existing ScyllaClusters have non-conforming task names.

Ensure ScyllaCluster spec.version is not empty¶

ScyllaCluster spec.version is now a required field. Any create or update request with an empty value will be rejected by the admission webhook. You can run the following snippet to check whether any of your existing ScyllaClusters have an empty spec.version:

kubectl get scyllaclusters --all-namespaces -o json | jq -r '
  .items[] | select((.spec.version // "") == "") |
  "\(.metadata.namespace)/\(.metadata.name)"
'

If the command returns no output, all your ScyllaClusters are unaffected. If any are listed, set their spec.version to a valid ScyllaDB image tag before upgrading.

Note

Why is this not a breaking change? A ScyllaCluster with an empty spec.version was never functional. The migration controller would fail to reconcile it, leaving the cluster in a permanently degraded state. Making the field required only prevents new broken clusters from being created.

Review ScyllaDBMonitoring spec.type default change¶

The default value of ScyllaDBMonitoring spec.type has changed from SaaS to Platform. Any existing ScyllaDBMonitoring object that omits spec.type will render Platform dashboards after the upgrade instead of SaaS. The SaaS value is also now deprecated and will be removed in a future release; the admission webhook will emit a warning when it is set explicitly.

You can run the following snippet to list ScyllaDBMonitoring objects that will be affected by the default change or that still use the deprecated SaaS value:

output=$(kubectl get scylladbmonitorings --all-namespaces -o json | jq -r '
  .items[] |
  select((.spec.type // "SaaS") == "SaaS") |
  "\(.metadata.namespace)/\(.metadata.name)\t(spec.type=\(.spec.type // "<unset, defaults to SaaS>"))"
') && if [ -z "$output" ]; then echo "No ScyllaDBMonitoring objects rely on the SaaS type."; else echo "$output"; fi

If the command returns no output, none of your ScyllaDBMonitoring objects are affected. Otherwise, for each listed object, decide whether you want to:

keep the previous behavior by setting spec.type: SaaS explicitly before upgrading (the admission webhook will emit a deprecation warning, and you will need to migrate to Platform before a future release removes SaaS), or
adopt the new default by either setting spec.type: Platform explicitly or leaving spec.type unset.

Note

Why is this not a breaking change? The Platform dashboards are a superset of the SaaS dashboards, so switching from SaaS to Platform does not remove functionality.

1.17 to 1.18¶

Upgrading from v1.17.x requires extra actions due to the removal of the standalone ScyllaDB Manager controller. The controller becomes an integral part of ScyllaDB Operator. As a result, the standalone ScyllaDB Manager controller deployment and its related resources need to be removed before upgrading ScyllaDB Operator.

Upgrade via GitOps (kubectl)¶

kubectl delete -n scylla-manager \
    clusterrole/scylladb:controller:aggregate-to-manager-controller \
    clusterrole/scylladb:controller:manager-controller \
    poddisruptionbudgets.policy/scylla-manager-controller \
    serviceaccounts/scylla-manager-controller \
    clusterrolebindings.rbac.authorization.k8s.io/scylladb:controller:manager-controller \
    deployments.apps/scylla-manager-controller

Upgrade via Helm¶

ScyllaDB Manager Helm installation has to be upgraded before upgrading ScyllaDB Operator Helm installation, following the standard Helm upgrade procedure. This will ensure that the ScyllaDB Manager Controller is removed before upgrading the ScyllaDB Operator.

Was this page helpful?