Conqueror.Streaming.Transport.Http.Common 0.6.0-experimental.1

Conqueror - for building scalable & maintainable .NET applications

ATTENTION: This project is currently still undergoing active development and contrary to what some of this README says, everything in here is still subject to change. Therefore please do not yet use this project for any production application.

Conqueror is a set of libraries that helps you build .NET applications in a structured way, using patterns like command-query separation, chain-of-responsibility (often also known as middlewares), publish-subscribe, and data streams.

Conqueror encourages clean architectures by decoupling your application logic from concrete transports like HTTP, and allows exposing business operations via many different transports with thin adapters. Conqueror makes it simple to build modular monoliths or distributed systems with clear contracts between different modules and applications. It also allows to transition from a modular monolith to a distributed system with minimal friction, giving teams the flexibility to start simple and delay the transition until the right time in a project's lifecycle.

See our quickstart or example projects if you want to jump right into code examples for using Conqueror. Or head over to our recipes for more detailed guidance on how you can utilize Conqueror to its maximum. Finally, if you want to learn more about the motivation behind this project (including comparisons to similar projects like MediatR), head over to the motivation section.

Conqueror only supports .NET 6 or later

Libraries

Conqueror.CQS

Split your business processes into simple-to-maintain and easy-to-test pieces of code using the command-query separation pattern. Handle cross-cutting concerns like logging, validation, authorization etc. using configurable middlewares. Keep your applications scalable by moving commands and queries from a modular monolith to a distributed application with minimal friction.

Head over to our CQS recipes for more guidance on how to use this library.

Middlewares:

Transports:

Experimental Libraries

The libraries below are still experimental. This means they do not have a stable API and are missing code documentation and recipes. They are therefore not suited for use in production applications, but can be used in proofs-of-concept or toy apps. If you use any of the experimental libraries and find bugs or have ideas for improving them, please don't hesitate to create an issue.

<details> <summary>Click here to see experimental libraries</summary>

Conqueror.Eventing

Decouple your application logic by using in-process event publishing using the publish-subscribe pattern. Handle cross-cutting concerns like logging, tracing, filtering etc. using configurable middlewares. Keep your applications scalable by moving events from a modular monolith to a distributed application with minimal friction.

Head over to our eventing recipes for more guidance on how to use this library.

Conqueror.Streaming

Keep your applications in control by allowing them to consume data streams at their own pace using a pull-based approach. Handle cross-cutting concerns like logging, error handling, authorization etc. using configurable middlewares. Keep your applications scalable by moving stream consumers from a modular monolith to a distributed application with minimal friction.

Head over to our streaming recipes for more guidance on how to use this library.

Transports:

</details>

Quickstart

This quickstart guide will let you jump right into the code without lengthy explanations (for more guidance head over to our recipes). By following this guide you'll add HTTP commands and queries to your ASP.NET Core application. You can also find the source code here in the repository.

# add relevant CQS packages
dotnet add package Conqueror.CQS
dotnet add package Conqueror.CQS.Analyzers
dotnet add package Conqueror.CQS.Middleware.Logging
dotnet add package Conqueror.CQS.Transport.Http.Server.AspNetCore

// add Conqueror CQS to your services
builder.Services
       .AddConquerorCQSTypesFromExecutingAssembly()
       .AddConquerorCQSLoggingMiddlewares();

builder.Services
       .AddControllers()
       .AddConquerorCQSHttpControllers();

// add Conqueror to your web app (just before mapping endpoints / controllers)
app.UseConqueror();
app.MapControllers();

In IncrementCounterByCommand.cs create a command that increments a named counter by a given amount (for demonstration purposes the counter is stored in an environment variable instead of a database).

using Conqueror;

namespace Quickstart;

[HttpCommand(Version = "v1")]
public sealed record IncrementCounterByCommand(string CounterName, int IncrementBy);

public sealed record IncrementCounterByCommandResponse(int NewCounterValue);

public interface IIncrementCounterByCommandHandler
    : ICommandHandler<IncrementCounterByCommand, IncrementCounterByCommandResponse>
{
}

internal sealed class IncrementCounterByCommandHandler
    : IIncrementCounterByCommandHandler, IConfigureCommandPipeline
{
    // add logging to the command pipeline and configure the pre-execution log
    // level (only for demonstration purposes since the default is the same)
    public static void ConfigurePipeline(ICommandPipelineBuilder pipeline) =>
        pipeline.UseLogging(o => o.PreExecutionLogLevel = LogLevel.Information);

    public async Task<IncrementCounterByCommandResponse> ExecuteCommand(IncrementCounterByCommand command,
                                                                        CancellationToken cancellationToken = default)
    {
        // simulate an asynchronous operation
        await Task.CompletedTask;

        var envVariableName = $"QUICKSTART_COUNTERS_{command.CounterName}";
        var counterValue = int.Parse(Environment.GetEnvironmentVariable(envVariableName) ?? "0");
        var newCounterValue = counterValue + command.IncrementBy;
        Environment.SetEnvironmentVariable(envVariableName, newCounterValue.ToString());
        return new(newCounterValue);
    }
}

In GetCounterValueQuery.cs create a query that returns the value of a counter with the given name.

using Conqueror;

namespace Quickstart;

[HttpQuery(Version = "v1")]
public sealed record GetCounterValueQuery(string CounterName);

public sealed record GetCounterValueQueryResponse(int CounterValue);

public interface IGetCounterValueQueryHandler
    : IQueryHandler<GetCounterValueQuery, GetCounterValueQueryResponse>
{
}

internal sealed class GetCounterValueQueryHandler
    : IGetCounterValueQueryHandler, IConfigureQueryPipeline
{
    // add logging to the query pipeline and configure the pre-execution log
    // level (only for demonstration purposes since the default is the same)
    public static void ConfigurePipeline(IQueryPipelineBuilder pipeline) =>
        pipeline.UseLogging(o => o.PreExecutionLogLevel = LogLevel.Information);

    public async Task<GetCounterValueQueryResponse> ExecuteQuery(GetCounterValueQuery query,
                                                                 CancellationToken cancellationToken = default)
    {
        // simulate an asynchronous operation
        await Task.CompletedTask;

        var envVariableName = $"QUICKSTART_COUNTERS_{query.CounterName}";
        var counterValue = int.Parse(Environment.GetEnvironmentVariable(envVariableName) ?? "0");
        return new(counterValue);
    }
}

Now launch your app and you can call the command and query via HTTP.

curl http://localhost:5000/api/v1/commands/incrementCounterBy --data '{"counterName":"test","incrementBy":2}' -H 'Content-Type: application/json'
# prints {"newCounterValue":2}

curl http://localhost:5000/api/v1/queries/getCounterValue?counterName=test
# prints {"counterValue":2}

Thanks to the logging middleware we added to the command and query pipelines, you will see output similar to this in the server console.

info: Quickstart.IncrementCounterByCommand[0]
      Executing command with payload {"CounterName":"test","IncrementBy":2} (Command ID: 1560c983e4856bd5, Trace ID: fe675fdbf9a987620af31a474bf7ae8c)
info: Quickstart.IncrementCounterByCommand[0]
      Executed command and got response {"NewCounterValue":2} in 4.2150ms (Command ID: 1560c983e4856bd5, Trace ID: fe675fdbf9a987620af31a474bf7ae8c)
info: Quickstart.GetCounterValueQuery[0]
      Executing query with payload {"CounterName":"test"} (Query ID: defa354e95d67ead, Trace ID: 8fdfa04f8c45ae3174044be0001a6e96)
info: Quickstart.GetCounterValueQuery[0]
      Executed query and got response {"CounterValue":2} in 2.9833ms (Query ID: defa354e95d67ead, Trace ID: 8fdfa04f8c45ae3174044be0001a6e96)

If you have swagger UI enabled, it will show the new command and query and they can be called from there.

<img src="./recipes/quickstart/swagger.gif?raw=true" alt="Quickstart Swagger" style="height: 565px" height="565px" />

Recipes

In addition to code-level API documentation, Conqueror provides you with recipes that will guide you in how to utilize it to its maximum. Each recipe will help you solve one particular challenge that you will likely encounter while building a .NET application.

For every "How do I do X?" you can imagine for this project, you should be able to find a recipe here. If you don't see a recipe for your question, please let us know by creating an issue or even better, provide the recipe as a pull request.

CQS Introduction

CQS is an acronym for command-query separation (which is the inspiration for this project and also where the name is derived from: conquer → commands and queries). The core idea behind this pattern is that operations which only read data (i.e. queries) and operations which mutate data or cause side-effects (i.e. commands) have very different characteristics (for a start, in most applications queries are executed much more frequently than commands). In addition, business operations often map very well to commands and queries, allowing you to model your application in a way that allows technical and business stakeholders alike to understand the capabilities of the system. There are many other benefits we gain from following this separation in our application logic. For example, commands and queries represent a natural boundary for encapsulation, provide clear contracts for modularization, and allow solving cross-cutting concerns according to the nature of the operation (e.g. caching makes sense for queries, but not so much for commands). With commands and queries, testing often becomes more simple as well, since they provide a clear list of the capabilities that should be tested (allowing more focus to be placed on use-case-driven testing instead of traditional unit testing).

CQS Basics

• getting started
• testing command and query handlers
• solving cross-cutting concerns with middlewares (e.g. validation or retrying on failure)
• testing command and query handlers that have middleware pipelines
• testing middlewares and reusable pipelines

CQS Advanced

• exposing commands and queries via HTTP
• testing HTTP commands and queries
• calling HTTP commands and queries from another application
• testing code which calls HTTP commands and queries
• creating a clean architecture and modular monolith with commands and queries
• moving from a modular monolith to a distributed system
• using a different dependency injection container (e.g. Autofac or Ninject) (to-be-written)
• customizing OpenAPI specification for HTTP commands and queries (to-be-written)
• re-use middleware pipelines to solve cross-cutting concerns when calling external systems (e.g. logging or retrying failed calls) (to-be-written)

CQS Expert

• store and access background context information in the scope of a single command or query (to-be-written)
• propagate background context information (e.g. trace ID) across multiple commands, queries, events, and streams (to-be-written)
• accessing properties of commands and queries in middlewares (to-be-written)
• exposing and calling commands and queries via other transports (e.g. gRPC) (to-be-written)

CQS Cross-Cutting Concerns

• authenticating and authorizing commands and queries (to-be-written)
• logging commands and queries (to-be-written)
• validating commands and queries (to-be-written)
• caching query results for improved performance (to-be-written)
• making commands and queries more resilient (e.g. through retries, circuit breakers, fallbacks etc.) (to-be-written)
• executing commands and queries in a database transaction (to-be-written)
• timeouts for commands and queries (to-be-written)
• metrics for commands and queries (to-be-written)
• tracing commands and queries (to-be-written)

Recipes for experimental libraries

<details> <summary>Click here to see recipes for experimental libraries</summary>

Eventing Introduction

Eventing is a way to refer to the publishing and observing of events via the publish-subscribe pattern. Eventing is a good way to decouple or loosely couple different parts of your application by making an event publisher agnostic to the observers of events it publishes. In addition to this basic idea, Conqueror allows solving cross-cutting concerns on both the publisher as well as the observer side.

Eventing Basics

• getting started (to-be-written)
• testing event observers (to-be-written)
• testing code that publishes events (to-be-written)
• solving cross-cutting concerns with middlewares (e.g. logging or retrying on failure) (to-be-written)
• testing event observers with pipelines (to-be-written)
• testing event publisher pipeline (to-be-written)
• testing middlewares (to-be-written)

Eventing Advanced

• using a different dependency injection container (e.g. Autofac or Ninject) (to-be-written)
• execute event observers with a different strategy (e.g. parallel execution) (to-be-written)
• enforce that all event observers declare a pipeline (to-be-written)
• creating a clean architecture with loose coupling via events (to-be-written)
• moving from a modular monolith to a distributed system (to-be-written)

Eventing Expert

• store and access background context information in the scope of a single event (to-be-written)
• propagate background context information (e.g. trace ID) across multiple commands, queries, events, and streams (to-be-written)
• accessing properties of events in middlewares (to-be-written)

Eventing Cross-Cutting Concerns

• logging events (to-be-written)
• retrying failed event observers (to-be-written)
• executing event observers in a database transaction (to-be-written)
• metrics for events (to-be-written)
• tracing events (to-be-written)

Streaming Introduction

For data streaming Conqueror uses a pull-based approach where the consumer controls the pace (using IAsyncEnumerable), which is a good approach for use cases like paging and event sourcing.

Streaming Basics

• getting started (to-be-written)
• testing streaming request handlers (to-be-written)
• solving cross-cutting concerns with middlewares (e.g. validation or retrying on failure) (to-be-written)
• testing streaming request handlers that have middleware pipelines (to-be-written)
• testing middlewares (to-be-written)

Streaming Advanced

• using a different dependency injection container (e.g. Autofac or Ninject) (to-be-written)
• reading streams from a messaging system (e.g. Kafka or RabbitMQ) (to-be-written)
• exposing streams via HTTP (to-be-written)
• testing HTTP streams (to-be-written)
• consuming HTTP streams from another application (to-be-written)
• using middlewares for streaming HTTP clients (to-be-written)
• optimize HTTP streaming performance with pre-fetching (to-be-written)
• enforce that all streaming request handlers declare a pipeline (to-be-written)
• re-use middleware pipelines to solve cross-cutting concerns when consuming streams from external systems (e.g. logging or retrying failed calls) (to-be-written)
• authenticating and authorizing streaming requests (to-be-written)
• moving from a modular monolith to a distributed system (to-be-written)

Streaming Expert

• store and access background context information in the scope of a single streaming request (to-be-written)
• propagate background context information (e.g. trace ID) across multiple commands, queries, events, and streams (to-be-written)
• accessing properties of streaming requests in middlewares (to-be-written)
• exposing and consuming streams via other transports (e.g. SignalR) (to-be-written)
• building test assertions that work for HTTP and non-HTTP streams (to-be-written)

Streaming Cross-Cutting Concerns

• authenticating and authorizing streaming requests (to-be-written)
• logging streaming requests and items (to-be-written)
• validating streaming requests (to-be-written)
• retrying failed streaming requests (to-be-written)
• timeouts for streaming requests and items (to-be-written)
• metrics for streaming requests and items (to-be-written)
• tracing streaming requests and items (to-be-written)

</details>

Motivation

Modern software development is often centered around building web applications that communicate via HTTP (we'll call them "web APIs"). However, many applications require different entry points or APIs as well (e.g. message queues, command line interfaces, raw TCP or UDP sockets, etc.). Each of these kinds of APIs need to address a variety of cross-cutting concerns, most of which apply to all kinds of APIs (e.g. logging, tracing, error handling, authorization, etc.). Microsoft has done an excellent job in providing out-of-the-box solutions for many of these concerns when building web APIs with ASP.NET Core using middlewares (which implement the chain-of-responsibility pattern). However, for other kinds of APIs, development teams are often forced to handle these concerns themselves, spending valuable development time.

One way many teams choose to address this issue is by forcing every operation to go through a web API (e.g. having a small adapter that reads messages from a queue and then calls a web API for processing the message). While this works well in many cases, it adds extra complexity and fragility by adding a new integration point for very little value. Optimally, there would be a way to address the cross-cutting concerns in a consistent way for all kinds of APIs. This is exactly what Conqueror does. It provides the building blocks for implementing business functionality and addressing those cross-cutting concerns in an transport-agnostic fashion, and provides extension packages that allow exposing the business functionality via different transports (e.g. HTTP).

A useful side-effect of moving the handling of cross-cutting concerns away from the concrete transport, is that it allows solving cross-cutting concerns for both incoming and outgoing operations. For example, with Conqueror the exact same code can be used for adding retry capabilities for your own command and query handlers as well as when calling an external HTTP API.

On an architectural level, a popular way to build systems these days is using microservices. While microservices are a powerful approach, they can often represent a significant challenge for small or new teams, mostly for deployment and operations (challenges common to most distributed systems). A different approach that many teams choose is to start with a modular monolith and move to microservices at a later point. However, it is common for teams to struggle with such a migration, partly due to sub-optimal modularization and partly due to existing tools and libraries not providing a smooth transition journey from one approach to another (or often forcing you into the distributed approach directly, e.g. MassTransit). Conqueror addresses this by encouraging you to build modules with clearly defined contracts and by allowing you to switch from having a module be part of a monolith to be its own microservice with minimal code changes.

In summary, these are some of the strengths of Conqueror:

• Providing building blocks for many different communication patterns: Many applications require the use of different communication patterns to fulfill their business requirements (e.g. request-response, fire-and-forget, publish-subscribe, streaming etc.). Conqueror provides building blocks for implementing these communication patterns efficiently and consistently, while allowing you to address cross-cutting concerns in a transport-agnostic fashion.

• Excellent use-case-driven documentation: A lot of effort went into writing our recipes. While most other libraries have documentation that is centered around explaining what they do, our use-case-driven documentation is focused on showing you how Conqueror helps you to solve the concrete challenges your are likely to encounter during application development.

• Strong focus on testability: Testing is a very important topic that is sadly often neglected. Conqueror takes testability very seriously and makes sure that you know how you can test the code you have written using it (you may have noticed that the Conqueror.CQS recipe immediately following getting started shows you how you can test the handlers we built in the first recipe).

• Out-of-the-box solutions for many common yet often complex cross-cutting concerns: Many development teams spend valuable time on solving common cross-cutting concerns like validation, logging, error handling etc. over and over again. Conqueror provides a variety of pre-built middlewares that help you address those concerns with minimal effort.

• Migrating from a modular monolith to a distributed system with minimal friction: Business logic built on top of Conqueror provides clear contracts to consumers, regardless of whether these consumers are located in the same process or in a different application. By abstracting away the concrete transport over which the business logic is called, it can easily be moved from a monolithic approach to a distributed approach with minimal code changes.

• Modular and extensible architecture: Instead of a big single library, Conqueror consists of many small (independent or complementary) packages. This allows you to pick and choose what functionality you want to use without adding the extra complexity for anything that you don't. It also improves maintainability by allowing modifications and extensions with a lower risk of breaking any existing functionality (in addition to a high level of public-API-focused test coverage).

Comparison with similar projects

Below you can find a brief comparison with some popular projects which address similar concerns as Conqueror.

Differences to MediatR

The excellent library MediatR is a popular choice for building applications. Conqueror takes a lot of inspirations from its design, with some key differences:

• MediatR allows handling cross-cutting concerns with global behaviors, while Conqueror allows handling these concerns with composable middlewares in independent pipelines per handler type.
• MediatR uses a single message sender service which makes it tricky to navigate to a message handler in your IDE from the point where the message is sent. With Conqueror you call handlers through an explicit interface, allowing you to use the "Go to implementation" functionality of your IDE.
• MediatR is focused building single applications without any support for any transports, while Conqueror allows building both single applications as well as distributed systems that communicate via different transports implemented through adapters.

Differences to MassTransit

MassTransit is a great framework for building distributed applications. It addresses many of the same concerns as Conqueror, with some key differences:

• MassTransit is designed for building distributed systems, forcing you into this approach from the start, even if you don't need it yet (the provided in-memory transport is explicitly mentioned as not being recommended for production usage). Conqueror allows building both single applications as well as distributed systems.
• MassTransit is focused on asynchronous messaging, while Conqueror provides more communication patterns (e.g. synchronous request-response over HTTP).
• MassTransit has adapters for many messaging middlewares, like RabbitMQ or Azure Service Bus, which Conqueror does not.
• MassTransit provides out-of-the-box solutions for advanced patterns like sagas, state machines, etc., which Conqueror does not.

If you require the advanced patterns or messaging middleware connectors which MassTransit provides, you can easily combine it with Conqueror by calling command and query handlers from your consumers or wrapping your producers in command handlers.