Tag Archives: AWS DataSync

AWS DataSync adds support for on-premises object storage

Post Syndicated from Alejandra Quetzalli original https://aws.amazon.com/blogs/aws/aws-datasync-adds-support-for-on-premises-object-storage/

AWS DataSync now supports transfers to and from self-managed object storage! 🎉

This new feature makes it easy for customers to automate copying large amounts of data between on-premises object storage and AWS Cloud Storage.

What is AWS DataSync?

If you’re new to AWS DataSync, you may be wondering how this service works.

AWS DataSync makes it simple and fast to move large amounts of data between on-premises storage and AWS Cloud Storage. AWS DataSync makes it easy for you to move data by automating both the management of data transfer processes and the infrastructure required for a high-performance and secure data transfer. The below image illustrates how you can use AWS DataSync to quickly and efficiently move data between on-premises storage systems and Amazon S3, Amazon EFS, and Amazon FSx for Windows File Server.




Let’s see a demo…

Who’s ready for a demo? I know I am! 😁

Let’s head over to the AWS Console and search for AWS DataSync.

AWS DataSync


Let’s click “Get Started.”

AWS DataSync

For the purpose of this blog post, we will pretend we’ve already Created an Agent. After you have created an agent and configured your source and destination, the next step is to Create a Task.

We want to set up a task that will transfer files from our on-premises object storage system to AWS. To do this, we select “Create Task.”


We select “Create a new location” and specify an Object storage location type for our source.


Now it’s time to select your agent. We add the IP address of our on-premises object storage server, as well as our path to the files we want to move.

Your object storage will likely require credentials to access it, so you may want to require Authentication.

We configure an Access key and Secret key to access our S3 bucket.

Now it’s time to configure our destination location. We do so by selecting “Choose an existing location” and by choosing an existing S3 bucket.

Next I pick a “Task Name.

We also have the option to schedule what frequency we wish to execute this Task. (You may want to schedule it in order to regularly pick up incremental changes until the migration cut-over.)

Now we can attach a CloudWatch LogGroup to the task and log all transferred objects and files!

The next step is to review our setup and finally create that task.

And if all goes well, we get our happy green bar. 🙌🏽


Now we head over to the S3 console, because we want to make sure that the objects were actually copied into our S3 bucket.

Voilà! There it is. 😁


Support for on-premises object storage for AWS DataSync is now globally available in 22 AWS Regions for our customers to leverage. 🌎

To learn more about AWS DataSync, visit the DataSync product page and the DataSync developer guide.


You may also enjoy…

Check out this awesome demo video of AWS DataSync made by my fellow teammate, Jerry Hargrove.


¡Gracias por tu tiempo!
~Alejandra 💁🏻‍♀️ y Canela 🐾


AWS DataSync Update – Support for Amazon FSx for Windows File Server

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-datasync-update-support-for-amazon-fsx-for-windows-file-server/

AWS DataSync helps you to move large amounts of data into and out of the AWS Cloud. As I noted in New – AWS DataSync – Automated and Accelerated Data Transfer, our customers use DataSync for their large-scale migration, upload & process, archiving, and backup/DR use cases.

Amazon FSx for Windows File Server gives you network file storage that is fully compatible with your existing Windows applications and environments (read New – Amazon FSx for Windows File Server – Fast, Fully Managed, and Secure to learn more). It includes a very wide variety of enterprise-ready features including native multi-AZ file systems, support for SQL Server, data deduplication, quotas, and the ability to force the use of in-transit encryption. Our customers use Amazon FSx for Windows File Server to lift-and-shift their Windows workloads to the cloud, where they can benefit from consistent sub-millsecond performance and high throughput.

Inside AWS DataSync
The DataSync agent is deployed as a VM within your existing on-premises or cloud-based environment so that it can access your NAS or file system via NFS or SMB. The agent uses a robust, highly-optimized data transfer protocol to move data back and forth at up to 10 times the speed of open source data transfer solutions.

DataSync can be used for a one-time migration-style transfer, or it can be invoked on a periodic, incremental basis for upload & process, archiving, and backup/DR purposes. Our customers use DataSync for transfer operations that encompass hundreds of terabytes of data and millions of files.

Since the launch of DataSync in November 2018, we have made several important updates and changes to DataSync including:

68% Price Reduction – We reduced the data transfer charge to $0.0125 per gigabyte.

Task Scheduling – We gave you the ability to schedule data transfer tasks using the AWS Management Console or the AWS Command Line Interface (CLI), with hourly, daily, and weekly options:

Additional Region Support – We recently made DataSync available in the Europe (Stockholm), South America (São Paulo), Asia Pacific (Hong Kong), Asia Pacific (Mumbai), and AWS GovCloud (US-East) Regions, bringing the total list of supported regions to 20.

EFS-to-EFS Transfer – We added support for file transfer between a pair of Amazon Elastic File System (EFS) file systems.

Filtering for Data Transfers – We gave you the ability to use file path and object key filters to control the data transfer operation:

SMB File Share Support – We added support for file transfer between a pair of SMB file shares.

S3 Storage Class Support – We gave you the ability to choose the S3 Storage Class when transferring data to an S3 bucket.

FSx for Windows Support
Today I am happy to announce that we are giving you the ability to use DataSync to transfer data to and from Amazon FSx for Windows File Server file systems. You can configure these file systems as DataSync Locations and then reference them in your DataSync Tasks.

After I choose the desired FSx for Windows file system, I supply a user name and password, and enter the name of the Windows domain for authentication:

Then I create a task that uses one of my existing SMB shares as a source, and the FSx for Windows file system as a destination. I give my task a name (MyTask), and configure any desired options:

I can set up filtering and use a schedule:

I have many scheduling options; here are just a few:

If I don’t use a schedule, I can simply click Start to run my task on an as-needed basis:

When I do this, I have the opportunity to review and refine the settings for the task:

The task starts within seconds, and I can watch the data transfer and throughput metrics in the console:

In addition to the console-based access that I just showed you, you can also use the DataSync API and the DataSync CLI to create tasks (CreateTask), start them (StartTaskExecution), check on task status (DescribeTaskExecution) and much more.

Available Now
This important new feature is available now and you can start using it today!


AWS DataSync News – S3 Storage Class Support and Much More

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-datasync-news-s3-storage-class-support-and-much-more/

AWS DataSync helps you to move large amounts of data into or out of the AWS Cloud (read my post, New – AWS DataSync – Automated and Accelerated Data Transfer, to learn more). As I explained in my post DataSync is a great fit for you Migration, Upload & Process, and Backup / DR use cases. DataSync is a managed service, and can be used to do one-time or periodic transfers of any size.

Newest Features
We launched DataSync at AWS re:Invent 2018 and have been adding features to it ever since. Today I would like to give you a brief recap of some of the newest features, and also introduce a few new ones:

  • S3 Storage Class Support
  • SMB Support
  • Additional Regions
  • VPC Endpoint Support
  • FIPS for US Endpoints
  • File and Folder Filtering
  • Embedded CloudWatch Metrics

Let’s take a look at each one…

S3 Storage Class Support
If you are transferring data to an Amazon S3 bucket, you now have control over the storage class that is used for the objects. You simply choose the class when you create a new location for use with DataSync:

You can choose from any of the S3 storage classes:

Objects stored in certain storage classes can incur additional charges for overwriting, deleting, or retrieving. To learn more, read Considerations When Working with S3 Storage Classes in DataSync.

SMB Support
Late last month we announced that AWS DataSync Can Now Transfer Data to and from SMB File Shares. SMB (Server Message Block) protocol is common in Windows-centric environments, and is also the preferred protocol for many file servers and network attached storage (NAS) devices. You can use filter patterns to control the files that are included in or excluded from the transfer, and you can use SMB file shares as the data transfer source or destination (Amazon S3 and Amazon EFS can also be used). You simply create a DataSync location that references your SMB server and share:

To learn more, read Creating a Location for SMB.

Additional Regions
AWS DataSync is now available in more locations. Earlier this year it became available in the AWS GovCloud (US-West) and Middle East (Bahrain) Regions.

VPC Endpoint Support
You can deploy AWS DataSync in a Virtual Private Cloud (VPC). If you do this, data transferred between the DataSync agent will not traverse the public internet:

The VPC endpoints for DataSync are powered by AWS PrivateLink; to learn more read AWS DataSync Now Supports Amazon VPC Endpoints and Using AWS DataSync in a Virtual Private Cloud.

FIPS for US Endpoints
In addition to support for VPC endpoints, we announced that AWS DataSync supports FIPS 140-2 Validated Endpoints in US Regions. The endpoints in these regions use a FIPS 140-2 validated cryptographic security module, making it easier for you to use DataSync for regulated workloads. You can use these endpoints by selecting them when you create your DataSync agent:

File and Folder Filtering
Earlier this year we added the ability to use file path and object key filters to exercise additional control over the data copied in a data transfer. To learn more, read about Excluding and including specific data in transfer tasks using AWS DataSync filters.

Embedded CloudWatch Metrics
Data transfer metrics are available in the Task Execution Details page so that you can track the progress of your transfer:

Other AWS DataSync Resources
Here are some resources to help you to learn more about AWS DataSync: