Data Migration with rclone in City Cloud

Stefano Cucchiella Backup Leave a Comment

Working close to customers gives you new challenges every day, and new complex solutions, workarounds, or merely small fixes are there to be offered.

One of these is data migration between two data centers mainly due to backup & restore, data mirroring, or your environment-specific reasons. 

This short article will describe how to migrate data from one City Cloud region to another. Specifically, we will use Karlskrona (Kna1) as the SOURCE region and Frankfurt (Fra1) as the TARGET region.

A full list of City Cloud regions is available here.

Also, we will use rclone server-side copy functionality to sync data between the SOURCE AND TARGET regions.

“rclone is a command line program to manage files on cloud storage. It supports a multitude of providers including OpenStack Swift and Amazon S3.” 

rclone.org

Pre-requisites

  • A City Cloud account – How-To
  • Openstack-client installed and CLI access to both SOURCE and TARGET regions through the openstack.rc file – How-To
  • In case you want to use S3 backend, EC2 credentials with both access_ and secret_key – How-To
  • A storage container and its objects available in SOURCE region – How-To

Step-by-step guide

1. Create S3 and/or Swift rclone configuration file (rclone.conf) for both regions:

[kna1-s3]
type = s3
provider = Other
env_auth = false
access_key_id = <access key>
secret_access_key = <secret>
region = swift-kna1.citycloud.com:8080
endpoint = swift-kna1.citycloud.com:8080
acl = private
 
[kna1-swift]
type = swift
user = <username>
key = <password>
auth = https://kna1.citycloud.com:5000/v3
tenant = <Project name>
region = Kna1
domain = <Domain name>
tenant_domain = <Domain name>
auth_version = 3
 
[fra1-s3]
type = s3
provider = Other
env_auth = false
access_key_id = <access key>
secret_access_key = <secret>
region = swift-fra1.citycloud.com:8080
endpoint = swift-fra1.citycloud.com:8080
acl = private
 
[fra1-swift]
type = swift
user = <username>
key = <password>
auth = https://fra1.citycloud.com:5000/v3
tenant = <Project name>
region = Fra1
domain = <Domain name>
tenant_domain = <Domain name>
auth_version = 3

An alternative way to use your Swift or S3 config without exposing your credentials in rclone.conf is by using env_auth=true and then source your OS_* and AWS_* environment variables as needed.

2. View all objects in the container, e.g. test, in the SOURCE region.

$ openstack object list test
+---------+
| Name    |
+---------+
| file.ex |
+---------+

3. Create a container, e.g. copy_from_kna1, in the TARGET region.

4. Use either S3 or Swift backend configuration to copy your files from the SOURCE to the TARGET region:

$ rclone sync --config rclone.conf kna1-s3:test fra1-s3:copy_from_kna1 -P
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s

5. Check that all objects are copied to the new container in the TARGET region:

$ openstack object list copy_from_kna1
+---------+
| Name    |
+---------+
| file.ex |
+---------+

Your data is now safely copied to a different location so that you can always rollback to a certain point in time.