MyTrout.Pipelines.Steps.Azure.ServiceBus 5.1.1

MyTrout.Pipelines.Steps.Azure.ServiceBus

Introduction

MyTrout.Pipelines.Steps.Azure.ServiceBus provides Pipeline steps to receive and send messages using Azure ServiceBus.

MyTrout.Pipelines.Steps.Azure.ServiceBus targets .NET 6.0

For more details on Pipelines, see Pipelines.Core

Pipelines.Hosting runs the Pipeline in a Console Application using Generic Host.

List of Available Steps

Installing via NuGet

Install-Package MyTrout.Pipelines.Steps.Azure.ServiceBus

Software dependencies

1. Azure.Messaging.ServiceBus 7.5.1 minimum, less than 7.6.x
2. MyTrout.Pipelines.Steps 2.3.0 minimum, less than 2.4.x

All software dependencies listed above use the MIT License.

BREAKING CHANGES FROM v3.x to v4.x

• Supports .NET 6.0 and .NET 5.0
• ReadMessageFromAzureStep will always copy all ApplicationProperties values from a message into PipelineContext.Items with a prefix provided by ReadMessageFromAzureOptions.
• ReadMessageFromAzureOptions.ApplicationProperties has been removed permanently.
• WriteMessageToAzureStep will always copy all ApplicationProperties values from PipelineContext.Items to the message with a prefix provided by WriteMessageToAzureOptions.
• WriteMessageToAzureOptions.ApplicationProperties has been removed permanently.

BREAKING CHANGES FROM v2.x to v3.x

• Drop support for .NET Standard 2.1

How do I use the steps in this library (ReadMessageFromAzureStep) ?

sample C# code

    using MyTrout.Pipelines;
    using MyTrout.Pipelines.Hosting;
    using MyTrout.Pipelines.Steps.Azure.ServiceBus
    using System;
    using System.Linq;
    using System.Threading.Tasks;

    namespace MyTrout.Pipeline.Azure.ServiceBus.Samples
    {
        public class Program
        {
            public static async Task Main(string[] args)
            {

                var host = Host.CreateDefaultBuilder(args)
                                    .AddStepDependency<ReadMessageFromAzureOptions>()
                                    .UsePipeline(builder => 
                                    {
                                        builder
                                            // NOTE: ReadMessageFromAzureStep will continue to read (and process) messages 
                                            //       one at a time until there are no more messages on the subscription.
                                            .AddStep<ReadMessageFromAzureStep>()
                                            .AddStep<UserDefinedStepThatProcessesMessageBodyIntoPipelineContext>()
                                            .AddStep<UserDefinedStepThatDoesSomethingWithPipelineContext>();
                                    })
                                    .Build();

                //
                // IMPORTANT NOTE FOR DEVELOPERS:
                // 
                // Use StartAsync() to allow the caller to review the PipelineContext after execution.
                //
                await host.StartAsync().ConfigureAwait(false);

                var context = host.Services.GetService<PipelineContext>();

                if(context.Errors.Any())
                {
                    // TODO: Errors have already been logged, do any special error processing here.
                }

                return 0;
            }
        }
    }
}

sample appsettings.json file

ApplicationrProperties are values that should be copied from the ApplicationProperties collection of the message to the PipelineContext.

{
    "AzureServiceBusConnectionString": "user supplied ASB ConnectionString",
    "EntityName": "topic/subscription"
    "ApplicationProperties": [ "Year", "Month", "Id" ]
}

How do I configure TimeSpan via the .NET ConfigurationManager.

The appropriate format for configuring a timespan using the .NET Configuration abstraction is D.HH.mm.nn.

For example, the default timespan configurations for ReadMessageFromAzureOptions would be:

{
    "TimeToWaitForNewMessage": "0.00:01:00",
    "TimeToWaitBetweenMessageChecks": "0.00:00:02"
}

Build the software locally.

1. Clone the software from the Pipelines repository.
2. Build the software in Visual Studio 2019 to pull down all of the dependencies from nuget.org.
3. Deploy the Azure ResourceGroup Template located at [azure-arm-resourcegroup.json](./azure-arm-resourcegroup.json))  to create the Resource Group.
3. Deploy the Azure ServiceBus Template located at [azure-arm-servicebus.json](./azure-arm-servicebus.json))  to initialize the Azure Service Bus connectivity.
4. Get the Primary Connection String for the Azure Service Bus Instance.
5. Create a new machine-level environment variable named 'TEST_PIPELINE_AZURE_SERVICE_BUS_CONNECTION_STRING' and set the value to the Azure Service Bus Connection string.
6. In Visual Studio, run all tests.  All of the should pass.
7. If you have Visual Studio Enterprise 2019, analyze the code coverage; it should be 100%.

Build the software in Azure DevOps.

1. In Organization Settings, select Extensions option.
2. Install the SonarCloud Extension.
3. Login to the SonarQube instance and generate a SonarQube token with the user account to use for running analysis.
4. In Project Settings, select Service Connections option.
5. Add a Service Connection for SonarQube and enter the token.
   - Make sure you check the 'Grant access permission to all pipelines' checkbox or configure appropriate security to this connection.
6. Add a Service Connection for an Azure Subscription 
   - The Azure Service Principal must have ability to create and delete Resource Groups/Azure Service Bus namespaces.
   - Make sure you check the 'Grant access permission to all pipelines' checkbox or configure appropriate security to this connection.
7. In Artifacts, add a new Feed named mytrout.
8. On the mytrout Artifacts feed, select the gear icon to configure the feed.
9. Select the Permissions tab, and click the ...
10. Click on Allow builds and Releases (which will add Project Collection Build Services as a Contributor).
11. Click on Allow project-scoped builds (which will add Pipeline Build Service as a Contributor)
12. Create a New Pipeline and reference the azure-pipelines.yml file in the /Steps/Azure/ServiceBus directory.
13. In Pipelines....Library, set up a Variable group named 'SonarQube Analysis'
    a. Add a variable named 'sonarCloudConnectionName' with the name of the SonarQube Service Connection created in Step 5.
    b. Add a variable named 'sonarCloudEnabled' with the value of 'YES'.
    c. Add a variable named 'sonarCloudOrganization' with the value of your SonarCloud organization.
14. In Pipelines....Library, set up a Variable Group named 'Pipelines Artifacts Feed'.
    a. Add a variable named 'publishVstsFeed' with the value of the feed to which output should be published.
15. Run the newly created pipeline.

Testing

These tests are a mixture of unit and integration tests necessary to read an Azure Service Bus Message into the pipeline and write an Azure Service Bus Message from the pipeline.

API references

ReadMessageFromAzureStep

• Reads a multiple messages from an Azure Service queue or subscription and sends them to the request pipeline one message at a time.
• Copies any Application Properties from the Message into the PipelineContext based on the Options.ApplicationProperties configured.
• Creates a PipelineContext entry named INPUT_STREAM containing a MemoryStream that is wrapping the ServiceBusMessage.Body stream.

WriteMessageToAzureStep

• Writes a single message to an Azure Service queue or topic.
• Copies any Application Properties from the PipelineContext into the ServiceBusMessage based on the Options.ApplicationProperties configured.
• UTF8 encodes the Stream contained in OUTPUT_STREAM and writes it to the Message.Body.

TO DEVELOPERS: All of the tests with a ReadMessageFromAzureOptions class are written to use the options defined in TODO: ?? this class ?? for all subsequent Topic or Subscription calls. All Pull Requests should follow this pattern to enable a "one location" alteration within each test to be used throughout the rest of the test.