Data Sync
Data Sync allows moving data into or out of AWS without using physical devices. It’s designed to run at huge scales: each agent can manage up to 10 Gbps (~100 TB/day) of data transfer and each job can handle up to 50 milion files.
You can also move data from an AWS service to another without the need of any agents.
You have the possibility to use . Also, you have the option to perform incremental and scheduled transfers.
You’re billed per GB of data moved.
A Data Sync Agent integrates over NFS or SMB with existing shares and starts to send data (encryption in transit) to supported services (Locations):
-
S3 storage classes
-
VPC endpoints:
-
EFS
-
FSx for Windows Server
-
Copying data happens on a schedule.
|
To sync data when you have not much bandwidth available you can use snowcone, which has a Data Sync agent pre-installed. |
Main architecture components
Task: a job that defines what to sync, from where, to where, how quickly (bandwidth throtling).
Agent: reads/writes on-prem data stores. It uses NFS or SMB.
Locations (from and to):
-
on-prem NFS
-
on-prem SMB
-
AWS EFS
-
AWS S3
-
AWS FSx
Key features
-
It allows transferring metadata (like permissions or timestamps) ⇒ this feature is UNIQUE in AWS.
-
It offers built-in data validation
-
Bandwidth Limiters to avoid link saturation
-
Incremental transfers
-
Scheduled transfers
-
[.underline]#Bidirectional transfers
-
Compression
-
Encryption
-
Automatic recovery for transit errors
-
Integrations with S3, EFS, FSx for Windows Server
-
Service-to-service transfers (like EFS ⇒ EFS), even cross-region without requiring any agents.