Scale, add, remove racks¶

How scaling works¶

Each rack in a ScyllaDB cluster maps to a single Kubernetes StatefulSet. Scaling changes the replica count of that StatefulSet:

• Scale up — new pods are appended at the end of the ordinal sequence (highest index). After the new node joins the token ring, the Operator automatically triggers a data cleanup on affected nodes.

• Scale down — the Operator decommissions the nodes above the new count, highest ordinal first. It waits for them to stream their data to the remaining nodes, then lowers the replica count and deletes their PVCs and Services. Whether the Operator decommissions the leaving nodes one at a time or all at once depends on parallel node operations.

Because StatefulSets maintain contiguous pod ordinals and scale down from the highest ordinal, you cannot remove an arbitrary node from the middle of a rack. If a specific node is unhealthy, use node replacement instead.

For background on the StatefulSet-per-rack architecture, see StatefulSets and racks.

Sequential and parallel node operations¶

Parallel node operations control whether the Operator acts on nodes one at a time or all at once. This covers starting nodes when you create a cluster, add a rack or scale a rack up, and decommissioning nodes when you scale a rack down.

With parallel node operations disabled, the Operator starts ScyllaDB nodes one at a time. Within a rack, each Pod must become ready before the Operator starts the next one. The Operator creates racks one at a time. Bringing up a cluster takes as long as the sum of every node’s startup time. Likewise, scaling down removes one node at a time. The next decommission starts only after the previous node’s Pod is gone, and no rack in the datacenter scales up or down in the meantime.

With parallel node operations enabled, the Operator starts all ScyllaDB nodes at once. The Operator starts Pods of a rack without waiting for the previous ones to become ready. The Operator creates all racks at the same time. Bringing up a cluster is faster, and the difference grows with the number of nodes. Likewise, scaling down decommissions all the nodes above the new count at once. The other racks can still scale up or down while it does. Configuration updates and version upgrades still wait until no node in the datacenter is leaving, in both modes.

Parallel node operations provide better performance when your keyspaces are backed by tablets (the default since ScyllaDB 2025.2), as opposed to vnodes. Therefore, we strongly recommend that you only use tablets for your data keyspaces. This is because with vnode-based keyspaces, a joining node streams its data before it finishes joining, which slows down bringing up new nodes. With tablets, the data is moved in the background after the node joins, so the time to bring up new nodes is not affected by data streaming. The same holds for scaling down. ScyllaDB migrates tablets off all the leaving nodes at the same time. It streams vnode data off one leaving node at a time, no matter how many nodes are leaving, so a parallel scale-down of vnode data takes about as long as a sequential one. Consider setting tablets_mode_for_new_keyspaces to enforced in your ScyllaDB configuration to prevent individual keyspaces from opting out of tablets.

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: scylla
  namespace: scylla
spec:
  enableParallelNodeOperations: true

Bootstrap synchronisation¶

In Kubernetes, Pods can start simultaneously, and a new node could attempt to bootstrap while another node is still restarting and appears down to its peers. ScyllaDB denies such a join request and leaves the new node in a state that is not recoverable automatically.

Enabling the BootstrapSynchronisation feature gate protects against this by holding each node’s startup until all nodes in the cluster are UP. It is recommended that you enable it whether or not parallel node operations are enabled. See Bootstrap synchronisation for details on the mechanism and Feature gates for instructions on enabling feature gates.

Scale a ScyllaCluster¶

Change spec.datacenter.racks[].members to the desired node count and apply:

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: scylla
  namespace: scylla
spec:
  datacenter:
    name: us-east-1
    racks:
      - name: us-east-1a
        members: 3          # was 1, now 3
        storage:
          capacity: 500Gi

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: scylla
  namespace: scylla
spec:
  datacenter:
    name: us-east-1
    racks:
      - name: us-east-1a
        members: 1          # was 3, now 1
        storage:
          capacity: 500Gi

Wait for the operation to complete:

kubectl -n scylla wait --timeout=10m --for='condition=Available' scyllaclusters.scylla.scylladb.com/scylla

While scaling down, you can list the nodes that are still leaving:

kubectl -n scylla get scyllacluster scylla -o json | jq '.status.racks | map_values([.decommissioningMembers[]?.name])'

Verify with nodetool status:

kubectl -n scylla exec -it scylla-us-east-1a-0 -c scylla -- nodetool status

kubectl -n scylla get scyllacluster scylla -o jsonpath='{range .status.conditions[?(@.status=="True")]}{.type}: {.reason}: {.message}{"\n"}{end}'

Add a rack to a ScyllaCluster¶

Append a new entry to the spec.datacenter.racks array. The Operator creates racks in the order they appear and waits for each rack to be fully ready before creating the next.

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: scylla
  namespace: scylla
spec:
  datacenter:
    name: us-east-1
    racks:
      - name: us-east-1a
        members: 3
        storage:
          capacity: 500Gi
      - name: us-east-1b         # new rack
        members: 3
        storage:
          capacity: 500Gi

# Zero-based index of the rack being removed in spec.datacenter.racks. Change it for your cluster.
RACK_INDEX=0

kubectl -n scylla patch scyllacluster scylla --type=json --patch-file=/dev/stdin <<EOF
[{"op": "replace", "path": "/spec/datacenter/racks/${RACK_INDEX}/members", "value": 0}]
EOF

kubectl -n scylla wait --timeout=30m \
  --for='condition=Available=True' scyllacluster/scylla

kubectl -n scylla get pods -l scylla/rack=<rack-name>

kubectl -n scylla describe scyllacluster scylla

  Racks:
    us-east-1a:
      Available Members:  0
      Members:            0
      Ready Members:      0
      Stale:              false
      Updated Members:    0
      Version:            2026.2.5

  Racks:
    us-east-1a:
      Available Members:  0
      Conditions:
        Status:  True
        Type:    MemberLeaving
      Decommissioning Members:
        Name:           scylla-us-east-1-us-east-1a-0
      Members:          0
      Ready Members:    0
      Stale:            false
      Updated Members:  0
      Version:          2026.2.5

# The same rack index as in step 1.
RACK_INDEX=0

kubectl -n scylla patch scyllacluster scylla --type=json --patch-file=/dev/stdin <<EOF
[{"op": "remove", "path": "/spec/datacenter/racks/${RACK_INDEX}"}]
EOF

The ScyllaCluster "scylla" is invalid: spec.datacenter.racks[0]: Forbidden: rack "us-east-1a" can't be removed because it still has members leaving the cluster: scylla-us-east-1-us-east-1a-0; they have to finish decommissioning and be removed first, please retry later

kubectl -n scylla wait --timeout=5m \
  --for='condition=Available=True' scyllacluster/scylla

Key considerations¶

Related pages¶

• StatefulSets and racks — how StatefulSets map to racks and why mid-set removal is not possible

• Bootstrap synchronisation — the mechanism gating bootstrap on the cluster’s node statuses

• Replace nodes — replacing a specific unhealthy node without scaling

• Migrate a rack to a new node pool — scaling up a new rack and scaling down the old one to migrate infrastructure

• Perform a rolling restart — restarting all nodes without changing the cluster size

• Data distribution with tablets — how tablets distribute data and why they speed up topology changes