Unlocking Rapid S3 Data Mobility: A Deep Dive into Cross-Account Transfers with AWS DataSync

Transferring vast amounts of data, especially terabytes of S3 objects, between different AWS accounts can often feel like an insurmountable challenge. Traditional methods, such as direct aws s3 cp commands or even native S3 cross-account replication, quickly become bottlenecks when dealing with 4-5 TB or more. These approaches can stretch transfer times into days, significantly impacting operational efficiency and incurring unexpected costs. This is precisely where AWS DataSync emerges as a game-changer, transforming sluggish transfers into swift, automated, and secure migrations.

During a recent large-scale migration, DataSync demonstrated its prowess, delivering speeds up to 10 times faster than conventional techniques. It redefines the process of moving massive datasets within the AWS ecosystem, offering a robust and reliable pipeline instead of a trickle.

Why AWS DataSync Excels for Large-Scale S3 Migrations

AWS DataSync is purpose-built for high-volume data movement, designed to handle everything from terabytes to petabytes with efficiency and security. Its key advantages include:

• Direct Service Integration: DataSync facilitates direct transfers between AWS services, eliminating the need for intermediate storage or complex orchestration.
• Parallelized Transfers: By leveraging parallel processing, DataSync drastically accelerates data movement, ensuring that your migration completes in a fraction of the time.
• Comprehensive Metadata Handling: It intelligently preserves crucial object metadata, tags, and Access Control Lists (ACLs), ensuring data integrity throughout the transfer.
• Incremental Sync Capabilities: DataSync supports incremental transfers, allowing you to run multiple syncs to copy only changed data, minimizing downtime during the final cutover.

For migrations involving 4-5 TB of data between AWS accounts, these features are not just conveniences; they are necessities for a smooth and cost-effective operation.

Step 1: Preparing Your S3 Buckets

Before initiating any transfer, you need to set up your S3 buckets in both the source and destination AWS accounts:

• Source Bucket: This is the S3 bucket residing in your original AWS account, containing the terabytes of data you wish to migrate.
• Destination Bucket: A new or existing S3 bucket in your target AWS account, where the data will be transferred.

For optimal speed and reduced costs, it is highly recommended that both buckets are located in the same AWS region. While DataSync doesn’t mandate versioning, ensuring both buckets exist prior to starting the migration is essential.

Step 2: Configuring IAM Roles and Permissions

Cross-account data transfer with DataSync necessitates precise IAM configurations. DataSync requires explicit permissions to read from your source bucket and write to your destination bucket. This involves setting up appropriate IAM roles and bucket policies.

Source Bucket Policy (Allows DataSync Role from Destination Account to Read)

This policy on your source S3 bucket grants the DataSync role from the destination account, along with a specific user from that account, the necessary permissions to list and retrieve objects.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads"
   ],
   "Resource": "arn:aws:s3:::source_bucket"
  },
  {
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:ListMultipartUploadParts",
    "s3:PutObjectTagging",
    "s3:GetObjectTagging",
    "s3:PutObject"
   ],
   "Resource": "arn:aws:s3:::source_bucket/*"
  }
 ]
}

Destination Account IAM Role Policy (For DataSync to Access Source)

This policy, attached to the DataSync execution role in your destination account, enables DataSync to assume the necessary permissions to read from the specified source bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::source_bucket"
        },
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:PutObjectTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::source_bucket/*"
        }
    ]
}

Destination Bucket Policy (Allows DataSync Role from Destination Account to Write)

This policy on your destination S3 bucket grants DataSync and specific users from the destination account permission to write, list, and manage objects within the destination bucket.

{
 "Version": "2008-10-17",
 "Statement": [
  {
   "Sid": "DataSyncCreateS3LocationAndTaskAccess",
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads",
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:ListMultipartUploadParts",
    "s3:PutObject",
    "s3:GetObjectTagging",
    "s3:PutObjectTagging"
   ],
   "Resource": [
    "arn:aws:s3:::dist_bucket",
    "arn:aws:s3:::dist_bucket/*"
   ]
  }
 ]
}

Step 3: Defining DataSync Locations

DataSync operates by creating “locations” that represent your data sources and destinations. You’ll need to define one for your source S3 bucket and another for your destination S3 bucket. Each location specifies the bucket ARN, storage class, and the IAM role DataSync will use to access it.

Here’s an example AWS CLI command for creating a source bucket location:

aws datasync create-location-s3 
  --s3-bucket-arn arn:aws:s3:::s3-source-01 
  --s3-storage-class STANDARD 
  --s3-config BucketAccessRoleArn="arn:aws:iam::1234567890:role/datasync-role" 
  --region us-east-1

Remember to repeat this process for your destination bucket, adapting the bucket ARN accordingly.

Step 4: Setting Up the DataSync Task

With both source and destination locations defined, the next step is to create a DataSync task. This task orchestrates the actual data transfer between your specified locations. When configuring the task, you have options to:

• Enable Metadata Copy: Crucial for preserving original timestamps, object tags, and other important metadata.
• Configure Incremental Syncs: This allows for multiple runs, only copying new or changed data, leading to minimal downtime during the final switchover.
• Monitor Progress: Utilize the AWS console to track transfer speed, progress, and completion status in real-time.

Step 5: Executing and Monitoring the Transfer

For migrations involving 4-5 TB, patience is key, but the speed increase over traditional methods is undeniable. What might have taken days can now be accomplished in hours. Monitor the DataSync task’s progress closely in the AWS Management Console to observe the dramatic improvement in transfer rates and ensure a successful migration.

Expert Recommendations for Large-Scale DataSync Transfers

1. Prioritize Same-Region Transfers: Whenever feasible, ensure both source and destination buckets are in the same AWS region. This significantly reduces data transfer costs and maximizes speed.
2. Leverage Incremental Syncs: Implement multiple DataSync runs. Perform initial bulk transfers, then use subsequent runs to copy only the incremental changes, streamlining your final cutover.
3. Maintain Strict IAM Control: Cross-account access is powerful. Ensure your IAM roles and bucket policies are as restrictive as possible, granting only the necessary permissions. Crucially, remove these permissions once the migration is complete.
4. Tag for Cost Tracking: Apply specific tags to your DataSync resources and related S3 buckets. This enables detailed cost analysis in AWS Cost Explorer, helping you track and optimize migration expenses.

Conclusion

The days of choosing between painstakingly slow and potentially costly methods for migrating large S3 datasets are behind us. AWS DataSync offers a modern, high-performance pipeline for secure, repeatable, and automated cross-account S3 migrations. It transforms a daunting task into a manageable project, freeing up valuable time and resources.

Considering the efficiency and control DataSync provides, how might it revolutionize your organization’s approach to cloud data management and resilience?