Azure.Storage.Blobs.ChangeFeed 12.0.0-preview.36

Azure Storage Blobs Change Feed client library for .NET

Server Version: 2021-02-12, 2020-12-06, 2020-10-02, 2020-08-04, 2020-06-12 2020-04-08, 2020-02-10, and 2019-12-12

The purpose of the change feed is to provide transaction logs of all the changes that occur to the blobs and the blob metadata in your storage account. The change feed provides ordered, guaranteed, durable, immutable, read-only log of these changes. Client applications can read these logs at any time. The change feed enables you to build efficient and scalable solutions that process change events that occur in your Blob Storage account at a low cost.

Source code | Package (NuGet) | Product documentation

Getting started

Install the package

Install the Azure Storage Blobs client library for .NET with NuGet:

dotnet add package Azure.Storage.Blobs.ChangeFeed --prerelease

Prerequisites

You need an Azure subscription and a Storage Account to use this package.

To create a new Storage Account, you can use the Azure Portal, Azure PowerShell, or the Azure CLI. Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the Client

Authentication works the same as in Azure.Storage.Blobs.

Key concepts

The change feed is stored as blobs in a special container in your storage account at standard blob pricing cost. You can control the retention period of these files based on your requirements (See the conditions of the current release). Change events are appended to the change feed as records in the Apache Avro format specification: a compact, fast, binary format that provides rich data structures with inline schema. This format is widely used in the Hadoop ecosystem, Stream Analytics, and Azure Data Factory.

You can process these logs incrementally or in-full. Any number of client applications can independently read the change feed, in parallel, and at their own pace. Analytics applications such as Apache Drill or Apache Spark can consume logs directly as Avro files, which let you process them at a low-cost, with high-bandwidth, and without having to write a custom application.

Thread safety

We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.

Additional concepts

Client options | Accessing the response | Long-running operations | Handling failures | Diagnostics | Mocking | Client lifetime

Examples

Get all events in the Change Feed

// Get all the events in the change feed.
List<BlobChangeFeedEvent> changeFeedEvents = new List<BlobChangeFeedEvent>();
await foreach (BlobChangeFeedEvent changeFeedEvent in changeFeedClient.GetChangesAsync())
{
    changeFeedEvents.Add(changeFeedEvent);
}

Get events between a start and end time

// Create the start and end time.  The change feed client will round start time down to
// the nearest hour, and round endTime up to the next hour if you provide DateTimeOffsets
// with minutes and seconds.
DateTimeOffset startTime = new DateTimeOffset(2017, 3, 2, 15, 0, 0, TimeSpan.Zero);
DateTimeOffset endTime = new DateTimeOffset(2020, 10, 7, 2, 0, 0, TimeSpan.Zero);

// You can also provide just a start or end time.
await foreach (BlobChangeFeedEvent changeFeedEvent in changeFeedClient.GetChangesAsync(
    start: startTime,
    end: endTime))
{
    changeFeedEvents.Add(changeFeedEvent);
}

Resume with continuationToken

string continuationToken = null;
await foreach (Page<BlobChangeFeedEvent> page in changeFeedClient.GetChangesAsync().AsPages(pageSizeHint: 10))
{
    foreach (BlobChangeFeedEvent changeFeedEvent in page.Values)
    {
        changeFeedEvents.Add(changeFeedEvent);
    }

    // Get the change feed continuation token.  The continuation token is not required to get each page of events,
    // it is intended to be saved and used to resume iterating at a later date.
    continuationToken = page.ContinuationToken;
    break;
}

// Resume iterating from the pervious position with the continuation token.
await foreach (BlobChangeFeedEvent changeFeedEvent in changeFeedClient.GetChangesAsync(
    continuationToken: continuationToken))
{
    changeFeedEvents.Add(changeFeedEvent);
}

Troubleshooting

All Blob service operations will throw a RequestFailedException on failure with helpful ErrorCodes. Many of these errors are recoverable.

Next steps

Get started with our Change Feed samples:

1. Hello World: Get changes that have occured in your storage account (or asynchronously)
2. Auth: Authenticate with connection strings, public access, shared keys, shared access signatures, and Azure Active Directory.

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to this library.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.