Azure.Storage.DataMovement 12.0.0

Azure Storage Data Movement Common client library for .NET

Azure Storage is a Microsoft-managed service providing cloud storage that is highly available, secure, durable, scalable, and redundant.

The Azure Storage Data Movement library is optimized for uploading, downloading and copying customer data.

Currently this version of the Data Movement library only supports Blobs and File Shares.

Source code | API reference documentation | REST API documentation | Product documentation

Getting started

Install the package

Install the Azure Storage client library for .NET you'd like to use with NuGet and the Azure.Storage.DataMovement client library will be included:

dotnet add package Azure.Storage.DataMovement

Prerequisites

You need an Azure subscription and a Storage Account to use this package.

To create a new Storage Account, you can use the Azure Portal, Azure PowerShell, or the Azure CLI. Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the client

Authentication is specific to the targeted storage service. Please see documentation for the individual services:

• Blob
• File Share

Permissions

Data Movement must have appropriate permissions to the storage resources. Permissions are specific to the type of storage Data Movement is connected to.

• Blob storage permissions
• File share permissions

Key concepts

The Azure Storage DataMovement client library contains shared infrastructure like TokenCredential, TransferManager and RequestFailedException.

Thread safety

We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.

Additional concepts

Client options | Accessing the response | Long-running operations | Handling failures | Diagnostics | Mocking | Client lifetime

Examples

This section demonstrates usage of Data Movement regardless of extension package. Package-specific information and usage samples can be found in that package's documentation. These examples will use local disk and Azure Blob Storage when specific resources are needed for demonstration purposes, but the topics here will apply to other packages.

Setup the TransferManager

Singleton usage of TransferManager is recommended. Providing TransferManagerOptions is optional.

TransferManager transferManager = new TransferManager(new TransferManagerOptions());

Starting New Transfers

Transfers are defined by a source and destination StorageResource. There are two kinds of StorageResource: StorageResourceSingle and StorageResourceContainer. Source and destination of a given transfer must be of the same kind.

StorageResource instances are obtained from StorageResourceProvider instances. See Initializing Local File StorageResource(s) for more information on the resource provider for local files and directories. See the Next Steps for our DataMovement extension packages for more info on their respective StorageResourceProvider types.

The sample below demonstrates StorageResourceProvider use to start transfers by uploading a file to Azure Blob Storage, using the Azure.Storage.DataMovement.Blobs package. It uses an Azure.Core TokenCredential generated from Azure.Identity DefaultAzureCredential() with permission to write to the blob.

TokenCredential defaultTokenCredential = new DefaultAzureCredential();
BlobsStorageResourceProvider blobs = new BlobsStorageResourceProvider(defaultTokenCredential);

// Create simple transfer single blob upload job
TransferOperation transferOperation = await transferManager.StartTransferAsync(
    sourceResource: LocalFilesStorageResourceProvider.FromFile(sourceLocalPath),
    destinationResource: await blobs.FromBlobAsync(destinationBlobUri));
await transferOperation.WaitForCompletionAsync();

The sample below demonstrates StorageResourceProvider use to start transfers by uploading a file to Azure Share File Storage, using the Azure.Storage.DataMovement.Files.Shares package. It uses an Azure.Core TokenCredential generated from Azure.Identity DefaultAzureCredential() with permission to write to the blob.

TokenCredential tokenCredential = new DefaultAzureCredential();
ShareFilesStorageResourceProvider shares = new(tokenCredential);
TransferManager transferManager = new TransferManager(new TransferManagerOptions());
TransferOperation fileTransfer = await transferManager.StartTransferAsync(
    sourceResource: LocalFilesStorageResourceProvider.FromFile(sourceLocalFile),
    destinationResource: await shares.FromFileAsync(destinationFileUri));
await fileTransfer.WaitForCompletionAsync();

See more examples of starting a transfer respective to each storage service:

• Blob
• Share Files

Initializing Local File or Directory StorageResource

Local filesystem resources are provided by LocalFilesStorageResourceProvider. This provider requires no setup to produce storage resources.

LocalFilesStorageResourceProvider files = new();
StorageResource fileResource = files.FromFile("C:/path/to/file.txt");
StorageResource directoryResource = files.FromDirectory("C:/path/to/dir");

Troubleshooting

See Handling Failed Transfers and Enabling Logging to assist with any troubleshooting.

See Known Issues for detailed information.

Next steps

Get started with our Blob DataMovement samples.

Get started with our Share File DataMovement samples.

For advanced scenarios regarding the TransferManager see TransferManager Samples.

For advanced scenarios regarding the TransferManager.StartTransfer() operation, see Start Transfer Samples.

For advanced scenarios and information regarding Resume, Pause and/or checkpointing see Pause and Resume, Checkpointing Samples.

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to these libraries.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.