Was this page helpful?
Restore from backup¶
Restore from backup taken using ScyllaDB Manager to a fresh empty cluster of any size.
Prerequisites¶
A running ScyllaDB cluster managed by ScyllaDB Operator.
ScyllaDB Manager installed and the cluster registered with Manager.
A backup snapshot stored in object storage (Amazon S3, Google Cloud Storage, or Azure Blob Storage).
The target cluster has access to the backup bucket (same object storage credentials as the source).
Warning
Restoring schema with ScyllaDB OS 5.4.X or ScyllaDB Enterprise 2024.1.X and consistent_cluster_management isn’t supported.
When creating the target ScyllaDB cluster, configure it with consistent_cluster_management: false.
When following the steps for schema restore, ensure you follow the additional steps dedicated to affected ScyllaDB versions.
In the following example, the ScyllaCluster, which was used to take the backup, is called source. Backup will be restored into the ScyllaCluster named target.
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
name: source
spec:
version: 6.2.2
developerMode: true
backups:
- name: foo
location:
- s3:source-backup
keyspace:
- '*'
datacenter:
name: us-east-1
racks:
- name: us-east-1a
members: 1
storage:
capacity: 1Gi
resources:
limits:
cpu: 1
memory: 1Gi
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
name: target
spec:
version: 6.2.2
developerMode: true
datacenter:
name: us-east-1
racks:
- name: us-east-1a
members: 1
storage:
capacity: 1Gi
resources:
limits:
cpu: 1
memory: 1Gi
Make sure your target cluster is already registered in ScyllaDB Manager. To get a list of all registered clusters, execute the following command:
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool cluster list
+--------------------------------------+---------------------------------------+---------+-----------------+
| ID | Name | Port | CQL credentials |
+--------------------------------------+---------------------------------------+---------+-----------------+
| af1dd5cd-0406-4974-949f-dc9842980080 | scylla/target | default | set |
| ebd82268-efb7-407e-a540-3619ae053778 | scylla/source | default | set |
+--------------------------------------+---------------------------------------+---------+-----------------+
Identify the tag of a snapshot which you want to restore. To get a list of all available snapshots, execute following command:
kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool backup list -c <CLUSTER_ID> --all-clusters -L <BACKUP_LOCATION>
Where:
CLUSTER_ID- the name or ID of a registered cluster with access toBACKUP_LOCATION.BACKUP_LOCATION- the location in which the backup is stored.
In this example, BACKUP_LOCATION is s3:source-backup. Use the name of cluster which has access to the backup location for CLUSTER_ID.
In this example, it’s scylla/target.
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool backup list -c scylla/target --all-clusters -L s3:source-backup
backup/ff36d7e0-af2e-458c-afe6-868e0f3396b2
Snapshots:
- sm_20240105115931UTC (409MiB, 1 nodes)
Keyspaces:
- system_schema (15 tables)
- users (9 tables)
Restore schema¶
In the below commands, we are restoring the sm_20240105115931UTC snapshot. Replace it with a tag of a snapshot that you want to restore.
Restoring consist of two steps. First, you’ll restore the schema, and then the data.
To restore schema, create a restore task manually on target ScyllaCluster by executing following command:
kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c <CLUSTER_ID> -L <BACKUP_LOCATION> -T <SNAPSHOT_TAG> --restore-schema
Where:
CLUSTER_ID- a name or ID of a cluster you want to restore into.BACKUP_LOCATION- the location in which the backup is stored.SNAPSHOT_TAG- a tag of a snapshot that you want to restore.
When the task is created, the command will output the ID of a restore task.
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c scylla/target -L s3:source-backup -T sm_20240105115931UTC --restore-schema
restore/57228c52-7cf6-4271-8c8d-d446ff160747
Use the following command to check progress of the restore task:
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool progress -c scylla/target restore/57228c52-7cf6-4271-8c8d-d446ff160747
Restore progress
Run: 0dd20cdf-abc4-11ee-951c-6e7993cf42ed
Status: DONE - restart required (see restore docs)
Start time: 05 Jan 24 12:15:02 UTC
End time: 05 Jan 24 12:15:09 UTC
Duration: 6s
Progress: 100% | 100%
Snapshot Tag: sm_20240105115931UTC
+---------------+-------------+----------+----------+------------+--------+
| Keyspace | Progress | Size | Success | Downloaded | Failed |
+---------------+-------------+----------+----------+------------+--------+
| system_schema | 100% | 100% | 214.150k | 214.150k | 214.150k | 0 |
+---------------+-------------+----------+----------+------------+--------+
As suggested in the progress output, you will need to execute a rolling restart of the ScyllaCluster if you are using ScyllaDB 5.4/2024.1 or older. For ScyllaDB 2024.2 and newer, a rolling restart is not required after restoring the schema.
For more details, refer to the ScyllaDB Manager Restore documentation:
Old Restore Schema Documentation (for ScyllaDB 5.4/2024.1 or older)
New Restore Schema Documentation (for ScyllaDB 2024.2 and newer)
kubectl patch scyllacluster/target --type=merge -p='{"spec": {"forceRedeploymentReason": "schema restored"}}'
Use the following commands to wait until restart is finished:
$ kubectl wait --for='condition=Progressing=False' scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
$ kubectl wait --for='condition=Degraded=False' scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
$ kubectl wait --for='condition=Available=True' scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
Caution
Restoring schema with ScyllaDB OS 5.4.X or ScyllaDB Enterprise 2024.1.X and consistent_cluster_management
After you’ve followed the above steps with a ScyllaDB target cluster with consistent_cluster_management disabled, you’ll need to enable Raft by configuring the target cluster with consistent_cluster_management: true.
You will then need to execute a rolling restart of the ScyllaCluster for the change to take effect.
kubectl patch scyllacluster/target --type=merge -p='{"spec": {"forceRedeploymentReason": "raft enabled"}}'
Use the following commands to wait until restart is finished:
$ kubectl wait --for='condition=Progressing=False' scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
$ kubectl wait --for='condition=Degraded=False' scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
$ kubectl wait --for='condition=Available=True' scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
Restore tables¶
To restore the tables content, create a restore task manually on target ScyllaCluster by executing the following command:
kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c <CLUSTER_ID> -L <BACKUP_LOCATION> -T <SNAPSHOT_TAG> --restore-tables
Where:
CLUSTER_ID- a name or ID of a cluster you want to restore into.BACKUP_LOCATION- the location in which the backup is stored.SNAPSHOT_TAG- a tag of a snapshot that you want to restore.
When the task is created, the command will output the ID of a restore task.
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c scylla/target -L s3:source-backup -T sm_20240105115931UTC --restore-tables
restore/63642069-bed5-4def-ba0f-68c49e47ace1
Use the following command to check progress of the restore task:
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool progress -c scylla/target restore/63642069-bed5-4def-ba0f-68c49e47ace1
Restore progress
Run: ab015cef-abc8-11ee-9521-6e7993cf42ed
Status: DONE
Start time: 05 Jan 24 12:48:04 UTC
End time: 05 Jan 24 12:48:15 UTC
Duration: 11s
Progress: 100% | 100%
Snapshot Tag: sm_20240105115931UTC
+-------------+-------------+--------+---------+------------+--------+
| Keyspace | Progress | Size | Success | Downloaded | Failed |
+-------------+-------------+--------+---------+------------+--------+
| users | 100% | 100% | 409MiB | 409MiB | 409MiB | 0 |
+-------------+-------------+--------+---------+------------+--------+
Post-restore repair progress
Run: ab015cef-abc8-11ee-9521-6e7993cf42ed
Status: DONE
Start time: 05 Jan 24 12:48:04 UTC
End time: 05 Jan 24 12:48:15 UTC
Duration: 11s
Progress: 100%
Intensity: 1
Parallel: 0
Datacenters:
- us-east-1
+-------------+--------------+----------+----------+
| Keyspace | Table | Progress | Duration |
+-------------+--------------+----------+----------+
| users | users | 100% | 0s |
+-------------+--------------+----------+----------+
Verify the restore¶
Connect to the target cluster and verify your data is present:
kubectl -n scylla exec -it target-us-east-1-us-east-1a-0 -c scylla -- cqlsh
DESCRIBE KEYSPACES;
SELECT COUNT(*) FROM users.users;