How can I use AWS CLI to migrate data from one Wasabi account to another?

Performing a cross-account transfer of data from one Wasabi account (we'll call it Account A) to another Wasabi account (Account B) is achievable through the use of bucket and IAM policies, and software that allows you to target different buckets within different accounts in the same command (such as with AWS CLI - How do I use AWS CLI with Wasabi? ). It is important to note that BOTH buckets MUST be within the same region for this to work.

In order for the transfer of cross-account resources to be successful, Wasabi requires a "two-way street" permissions set between the resources. This means that if you wish for a user in Account A to access a bucket in Account B, that user MUST have permissions in BOTH Account A (via an IAM policy) AND in Account B (via a bucket policy).

Let's assume that we are using a bucket called 'sourcebucket' in Account A, and we want to transfer the data to a bucket called 'destinationbucket' in Account B. We will be using an IAM user called 'source-user' from Account A to perform the copy. Please follow the below steps to get started:

  1. Apply a bucket policy to the destination bucket allowing access for the user in Account A to access the bucket.
  2. Apply a user policy to the IAM user in Account A allowing them to perform these actions allowing in both the source and destination buckets.
  3. Using the user in Account A that has been granted access via the bucket policy and IAM policy, use either the 'cp' or 'sync' feature to migrate the data from the source bucket to the destination bucket.

Let's go through the steps in the following example:

1) The bucket policy for 'destinationbucket' should explicitly allow access to a specified user in Account A to access the bucket. For this example, we are going to allow all LIST and PUT functions. For security reason, we will not be allowing any GET or DELETE calls against the destination bucket.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AccountA:user/AccountAUserName"
},
"Action": [
"s3:ListBucket",
"s3:PutObject*"
],
"Resource": [
"arn:aws:s3:::<DESTINATION-BUCKET-NAME>",
"arn:aws:s3:::<DESTINATION-BUCKET-NAME>/*"
]
}
]
}

My bucket settings for the 'destinationbucket' now look like the following (notice I made changes to specifically reflect my account information):

destinationbucket.JPG

Now that 'destinationbucket' has the policy granting 'source-user' in Account A access to the bucket, we need to ensure that 'source-user' that has been granted in the bucket policy has sufficient permissions to access the source bucket and destination bucket as well via an IAM policy. In my example, we will be checking the user 'source-user' in Account A. At a minimum, we will need "s3:GetObject" (to access the source) and "s3:PutObject" (to access the destination) if we know the exact path of the data we are copying. If we wish to use the 'sync' command, we will also need "s3:ListBucket" for both bucket resources.

Here is an example policy for our 'source-user' in Account A, which ensure it has sufficient permissions to LIST and GET objects from 'sourcebucket' and LIST and PUT objects to  'destinationbucket'

{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject*"
],
"Resource": [
"arn:aws:s3:::<SOURCE-BUCKET-NAME>",
"arn:aws:s3:::<SOURCE-BUCKET-NAME>/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject*"
],
"Resource": [
"arn:aws:s3:::<DESTINATION-BUCKET-NAME>",
"arn:aws:s3:::<DESTINATION-BUCKET-NAME>/*"
]
}
]
}

Now that we know 'source-user' in Account A has access to both 'sourcebucket' (via a permissions policy in Account A), and 'destinationbucket' (via the bucket policy applied to the bucket AND the IAM policy applied to the user), we can move forward with moving the data from one bucket to the other.

2) Configure an API user credential key pair for Account A's 'source-user' (Creating a Wasabi API Access Key Set). Using AWS CLI, we will structure the command to move data from 'sourcebucket' to 'destinationbucket', as well as ensure we define the region in which the buckets reside. Please remember that if the buckets are in a DIFFERENT region, this will not work as you can only define one (1) endpoint (What are the service URLs for Wasabi's different storage regions?) in the CLI command. The command consists of the following URI:

aws s3 sync <SOURCE-BUCKET> <DESTINATION-BUCKET> --endpoint-url=<ENDPOINT-URL>

So in my example, as the source is called 'sourcebucket' and the destination is called 'destinationbucket' and they are both in the us-east-2 region, we will use the following:

aws s3 sync s3://sourcebucket/ s3://destinationbucket/ --endpoint-url=https://s3.us-east-2.wasabisys.com --profile source-user

NOTE: AWS CLI allows for you to have multiple profiles saved to your local machine so that you can utilize the access/secret key pairs for different users upon command. This is used in my example by utilizing the '--profile' flag. 

This results in the 'sync' command running successfully to migrate data from 'sourcebucket' in Account A to 'destinationbucket' in Account B utilizing the 'source-user' from Account A:

$ aws s3 sync s3://sourcebucket/ s3://destinationbucket/ --endpoint-url=https://s3.us-east-2.wasabisys.com --profile source
copy: s3://sourcebucket/test2.txt to s3://destinationbucket/test2.txt
copy: s3://sourcebucket/test.txt to s3://destinationbucket/test.txt
$

If you have any questions or issues when attempting this action between buckets, please reach out to our support staff at support@wasabi.com so that we may assist.

Have more questions? Submit a request