🔁 What is AWS DataSync?
AWS DataSync is a fully managed data transfer service that automates and accelerates the movement of data between:
- On-premises storage (NFS, SMB, HDFS)
- AWS storage services like Amazon S3, Amazon EFS, Amazon FSx
It is ideal for one-time migrations, recurring transfers, and hybrid cloud storage operations.
🚀 Key Benefits
| Feature | Description |
|---|---|
| ⚡ 10x faster | Optimized transfer engine over standard tools like rsync
|
| 🔁 Incremental sync | Only transfers changed data after first run |
| 🛠️ No manual scripts | Fully managed, no custom code required |
| 🔐 Secure & encrypted | End-to-end encryption + VPC support |
| 📅 Scheduled transfers | Built-in cron-style scheduling |
| 🧪 Data integrity checks | Uses checksums to verify data consistency |
🧱 Common Use Cases
| Use Case | Target AWS Service |
|---|---|
| On-prem NFS backup to AWS | Amazon S3 or EFS |
| Hybrid workloads with shared datasets | Amazon FSx or EFS |
| Data migration to the cloud | S3, EFS, FSx |
| Transfer between AWS Regions or accounts | S3 → S3 or FSx → FSx |
🧭 How It Works
🔧 1. Install DataSync Agent
- Deploy a DataSync agent on-prem (VMware, Hyper-V, EC2, or Docker)
- Connect it to NFS, SMB, or HDFS storage
☁️ 2. Create Locations
- Source: NFS, SMB, S3, EFS, FSx, HDFS
- Destination: S3, EFS, FSx
📤 3. Create & Start a Task
- Configure filters, bandwidth limits, schedule
- Start task manually or on schedule
🔐 Security
| Feature | Description |
|---|---|
| TLS in transit | All data is encrypted |
| IAM policies | Control access to DataSync and storage |
| VPC support | Fully private data movement |
| CloudTrail logs | For auditing all API calls |
💵 Pricing Overview
| Item | Cost |
|---|---|
| Data transfer fee | $0.0125 per GB (source → destination) |
| Agent use | Free |
| S3/FSx storage cost | Billed separately |
📌 No charge for data transferred within the same region to/from AWS.
👉 Pricing page: AWS DataSync Pricing
🖥️ Example Architecture: On-Prem NFS → Amazon S3
+--------------------+ (1) Agent (VM) +------------------------+
| On-Prem NAS/NFS | --------------------------> | AWS DataSync Agent |
+--------------------+ +------------------------+
|
(2) Secure Transfer
|
+---------------------+
| Amazon S3 Bucket |
+---------------------+
📝 Key Config Options per Task
- Exclusions: Skip specific files/folders using patterns
- Bandwidth throttling: Limit usage during work hours
- Overwrite policies: Replace existing data or skip
- Verification: Checksum or none
- CloudWatch alarms: For monitoring success/failure
✅ Summary Table
| Feature | Details |
|---|---|
| Transfer Types | On-prem → AWS, AWS → AWS |
| Protocols Supported | NFS, SMB, HDFS |
| Destination Services | S3, EFS, FSx |
| Agent Required | Yes (except AWS → AWS) |
| Schedule Support | Yes (cron style) |
| Ideal For | Backups, sync, migrations |
| Cost | $0.0125/GB transferred |