ScyllaDB Docs ScyllaDB Operator Operate Resize nodes in a rack

For AI agents: a documentation index is available at https://operator.docs.scylladb.com/master/llms.txt. A Markdown version of this page is at https://operator.docs.scylladb.com/master/operate/resize-nodes-in-a-rack.md.

Caution

You're viewing documentation for an unstable version of ScyllaDB Operator. Switch to the latest stable version.

Resize nodes in a rack¶

Change the CPU or memory of the ScyllaDB nodes of a rack. To grow the storage of the nodes, see Expand storage volumes instead.

How resizing works¶

The resources of a rack are the resources of the ScyllaDB container of each of its nodes. The CPU limit sets the number of shards of a node, and the memory limit sets the memory ScyllaDB uses. It’s advised to keep the requests equal to the limits, so that the Pods keep the Guaranteed QoS class that CPU pinning needs.

Make sure the Kubernetes nodes have room for the new size: a Pod that no Kubernetes node can fit stays Pending, and the change gets stuck on it. See Recover from a stuck config change.

Set the namespace and the name of your ScyllaCluster, and the name and the index of the rack to change:

NAMESPACE=scylla
CLUSTER=my-cluster
RACK=us-east-1a
# Zero-based index of the rack in spec.datacenter.racks.
RACK_INDEX=0

Increase the CPU, or change the memory¶

Set the new limits on the rack, and the requests to match:

kubectl -n "${NAMESPACE}" patch scyllacluster "${CLUSTER}" --type=json -p="
- op: replace
  path: /spec/datacenter/racks/${RACK_INDEX}/resources/limits/cpu
  value: '8'
- op: replace
  path: /spec/datacenter/racks/${RACK_INDEX}/resources/requests/cpu
  value: '8'
- op: replace
  path: /spec/datacenter/racks/${RACK_INDEX}/resources/limits/memory
  value: 64Gi
- op: replace
  path: /spec/datacenter/racks/${RACK_INDEX}/resources/requests/memory
  value: 64Gi
"

Wait for the change to roll out:

kubectl -n "${NAMESPACE}" wait --for='condition=Progressing=False' --timeout=30m "scyllacluster/${CLUSTER}"
kubectl -n "${NAMESPACE}" wait --for='condition=Available=True' --timeout=10m "scyllacluster/${CLUSTER}"

Decrease the CPU¶

ScyllaDB Operator rejects a decrease of the CPU limit unless the ScyllaCluster carries the scylla-operator.scylladb.com/force-scylladb-cpu-decrease: "true" annotation, because of the ScyllaDB limitation described below. Keyspaces use tablets by default since ScyllaDB 2025.2, so lower the CPU limit by replacing the nodes of the rack one at a time, as described below.

Only if none of your keyspaces use tablets, set the annotation as in Step 1, change the CPU as in the previous section, and remove the annotation as in Step 4 instead. To find out which keyspaces use tablets, run cqlsh from any node:

kubectl -n "${NAMESPACE}" exec my-cluster-us-east-1-us-east-1a-0 -c scylla -- cqlsh -e "SELECT keyspace_name, initial_tablets FROM system_schema.scylla_keyspaces"

 keyspace_name | initial_tablets
---------------+-----------------
            ks |               0

(1 rows)

The keyspaces with a value in initial_tablets use tablets.

Why a plain change doesn’t work with tablets¶

The CPU limit of the ScyllaDB container sets the number of shards of a node. ScyllaDB can’t reduce the shard count of a node that holds tablet-based tables: a node restarted with fewer CPUs refuses to start, and the rack gets stuck on it.

A replacement node starts with a fresh data directory and streams its data from the other replicas, so it can have any number of shards. This procedure lowers the CPU limit and replaces each node of the rack as its rollout reaches it.

Prerequisites¶

Every keyspace has a replication factor of at least 2 in the datacenter, so that the data of a replaced node is available from other replicas.
The cluster is healthy: Progressing=False, Available=True, and all nodes are UN in nodetool status.
There is no other node operation in progress in the datacenter.

The replacements stream the data of every node of the rack once. Plan for the load and the time this takes with your data volume.

Procedure¶

Step 1: Allow the CPU decrease¶

kubectl -n "${NAMESPACE}" annotate scyllacluster "${CLUSTER}" scylla-operator.scylladb.com/force-scylladb-cpu-decrease=true

Step 2: Lower the CPU limit¶

Set the new CPU limit on the rack, and the request to match if you use guaranteed resources:

kubectl -n "${NAMESPACE}" patch scyllacluster "${CLUSTER}" --type=json -p="
- op: replace
  path: /spec/datacenter/racks/${RACK_INDEX}/resources/limits/cpu
  value: '4'
- op: replace
  path: /spec/datacenter/racks/${RACK_INDEX}/resources/requests/cpu
  value: '4'
"

ScyllaDB Operator rolls the rack one node at a time, starting from the highest ordinal. The restarted node fails to start with fewer shards, which is expected, and the rollout waits on it.

Step 3: Replace each node as it fails to start¶

Find the Pod that isn’t ready:

kubectl -n "${NAMESPACE}" get pods -l scylla/cluster="${CLUSTER}",scylla/rack="${RACK}"

NAME                                READY   STATUS    RESTARTS      AGE
my-cluster-us-east-1-us-east-1a-0   5/5     Running   0             2d
my-cluster-us-east-1-us-east-1a-1   5/5     Running   0             2d
my-cluster-us-east-1-us-east-1a-2   4/5     Running   3 (2m ago)    15m

Label its member Service, which has the same name as the Pod, to replace the node:

POD=my-cluster-us-east-1-us-east-1a-2
kubectl -n "${NAMESPACE}" label svc "${POD}" scylla/replace=""

ScyllaDB Operator deletes the Pod and its data volume, and the new Pod bootstraps as a replacement with the new CPU limit, streaming its data from the other replicas. Wait for it to become ready:

kubectl -n "${NAMESPACE}" wait --for='condition=Ready' --timeout=30m "pod/${POD}"

The rollout then moves on to the next node, which fails to start the same way. Repeat this step for every node of the rack, one at a time, until all Pods are ready.

Step 4: Remove the annotation¶

kubectl -n "${NAMESPACE}" annotate scyllacluster "${CLUSTER}" scylla-operator.scylladb.com/force-scylladb-cpu-decrease-

Step 5: Verify¶

kubectl -n "${NAMESPACE}" wait --for='condition=Progressing=False' --timeout=10m "scyllacluster/${CLUSTER}"
kubectl -n "${NAMESPACE}" wait --for='condition=Available=True' --timeout=10m "scyllacluster/${CLUSTER}"
kubectl -n "${NAMESPACE}" exec "${POD}" -c scylla -- nodetool status

Datacenter: us-east-1
===========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address       Load    Tokens Owns Host ID                              Rack
UN 10.96.153.205 1.24 MB 256    ?    52d102f2-2e42-41f3-b397-9ffdb9d57a34 us-east-1a
UN 10.96.23.29   1.25 MB 256    ?    e7a019b1-f1c7-494f-826b-6dec95644772 us-east-1a
UN 10.96.245.235 1.26 MB 256    ?    c9978ee0-014c-49a8-96cb-882e9a10d5fa us-east-1a

All nodes report UN, and the replaced nodes have new host IDs.

Was this page helpful?