Unified Message Bus Framework - Part 1

• Download UnifiedMessageBus.zip - 9.9 MB

WARNING

This is not a simple demo project.  There is a lot of architecture and advanced concepts that span 12 projects.  If you choose to create your own Azure Service Bus (which I recommend) the demo will fully function as described, but without it there's still a lot to gain from the demo if you dig around in it.  Most of the UI elements are vanilla as I'm trying to focus on the pub/sub aspects and not how to make pretty applications.

Introduction

Most development is going multiplatform nowadays, so I wanted to expand my previous article on global service bus architecture to include a framework for base functionality to support messaging between all platforms.

I previously discussed establishing a transport provider architecture using the facade and adapter patterns to simplify consumption of messages in a universal way.  Transports in this case are technically the underlying phsyical mechanisms to move messages (i.e. MSMQ, sockets, etc.); however, I'm using that term a bit more loosely.  In this case I use transport to cover any service bus or higher level messaging implementation as well.

For the new article and demo (and potentially open source project to come) I've started off with five transports:

• Azure Service Bus
  • NOTE:  You will have to create your own Azure account and Azure Service Bus to use this.
• SignalR
• WebApi
• ZeroMQ
  • NOTE:  Due to older NuGet package issues this one can be cumbersome.
• Reactive Extensions

With these transports you can send messages between any platform.  For example, a web client can send a message that gets consumed by a WPF application and vice versa.

In upcoming updates I'll work on making it cleaner, but I wanted to put out what I had before going into Google's Cloud Pub/Sub for the first time.  Additionally ZeroMQ and SignalR had some unexpected quirks from lack of recent NuGet updates so I didn't want to invest too much into making them cleaner as hopefully updates will resolve those issues.

I'll dive deeper into the actual provider implementations in future articles.  For now let's talk about the basic concepts, architecture, and the demo projects...

Background

Before reading this artcile you should be familiar with:

• IOC development
• MEF
• Pub/Sub
• Asynchronous Programming
• Observables
• Adapter Pattern
• Facade Pattern
• Azure Service Bus

Please read the prior article for additional background on the concepts of the architecture.

There are three different applications you can run to send messages:

• Console
• Web
• WPF

If you want to work with the web projects you'll need background in ASP.NET and WebApi.  If you want to work with the WPF project you'll need to be familiar with PRISM.

Architecture

I'm not going to rehash the entire prior article, but I do want to go deeper into some of the architecture with the multiplatform consideration.

I have expanded the Scope of a provider to the following:

• Global
• Local Machine
• Application

Global messages are meant to issue system wide commands or send updates to domain level objects.  While the demo is simply passing text data, the messaging framework could even be hooked up in the IoT realm to be used in an advanced controller.

Since global messages are the most costly to send and consume it's extremely important to have a centralized service to process global messages in each application.  The message service should be responisble for broadcasting the message via the Bus to local IUnifiedMessageBusProviders or by pushing to local observables.

The Bus also ensures that each message is only picked up once.  If you were issuing a lengthy or resource intensive command you obviously only want to do it once.  The benefit of this design is allowing multiple IUnifiedMessageBusProviders for either redundancy or specialized throughput concerns without any consumer having to filter out duplicates themselves.

Application scope messages are meant to be messages only pertinent to the application.  As mentioned earlier, this includes rebroadcasting global messages to each application consumer where necessary.  Another example would be user preference settings.  The user preferences for a particular application (i.e. configuring layout, control sizes, etc) are usually only going to be pertinent to that particular application.  Obviously you don't want to process layout changes from a different application.

The bigest benefit of application scoped messages is they don't have to be serialized.  Since the object is already in memory you simply pass it around.  The pitfall of application scoped messages is causing memory leaks.  Subscriptions are normally on static or long lived objects so you need to make sure to explicitly dispose all subscriptions when not needed.

Local machine messages are the most limited scope, but applicable for distributed applications.  The standard recommendation would be to communicate via services and APIs, but this could be replaced by pub/sub.  An advantage over exposing methods publically is the message type can be constrained to particular applications therefore creating exclusive, hidden communication between applications.

In the end you connect the applications via a network of providers.  Use global service buses like Azure Service Bus to be the backbone, then push down to the appropriate local IUnifiedMessageBusProviders.  As long as two applications share at least one IUnifiedMessageBusProvider, they can rebroadcast any message to other IUnifiedMessageBusProviders the originator has no awareness of (or capibility to support).

Each base IUnifiedMessageBusProvider implementation is abstract and should be implemented by a shared concrete instance - you don't want two separate pieces of code trying to pull messages off the same resources.  By all means make multiple concrete IUnifiedMessageBusProvider<span style="color: rgb(17, 17, 17); font-family: "Segoe UI", Arial, sans-serif; font-size: 14px;"> implementations</span> off the same base, just configure it to use a different topic/subscription/resource.

Additionally the concrete implementation may need to be plugged into different parts of the application for things like startup and coordination, so the application should have knowledge of what is being exported.

Keeping it simple...

Most service buses have different types of messages that are supported.  While the sematics may differ between providers there are three basic concepts of messages:

• Topic
  • A single message is being broadcast through mutiple providers to any subscribers with little to no discretion.
• Queue
  • Messages that should only be consumed once amongst all subscribers.
• Workflow  (Saga)
  • Messages that have a designated workflow and/or cordination with other messages and responses.

The current bus is topic based because it's the most widely applicable, widely supported, and simplest to implement.  Various mechanisms like database level locks can be used to logically implement queued messages and the application message services can coordinate workflow within a topic based architecture.

That being said it isn't difficult to integrate queued messages in this architecture, but for now that's a complexity this solution isn't meant to tackle.  Messages constrained to a workflow are typically coordinated within the provider and do not fit well into a framework meant for general consumption.

The trick is...

The biggest trick to the architecture is how the serialization/deserialization is handled.  Each IUnifiedMessageBusProvider has it's own requirements on how the message must be constructed going accross the transport.

• Azure Service Bus
  • BrokeredMessage which contains a binary serialized object that can only be retrieved once.
• WebApi
  • JSON friendly serialization.
• ZeroMQ
  • Each message requires specific serialization attributes.

The MessageWrapper class contains the message metadata to include what datatype the message object is.  However the object is sent, this message metadata must go accross the wire to ensure the receiving end can generically process the object for deserialization.  If you don't do this the issue becomes having to create class specific consumers of messages which kinda defeats the purpose of an IOC based bus (IMHO).

NOTE:  This is one of the reasons why I wasn't fond of MassTransit as it requires consumer implementations.

Listeners are expected to pass back the MessageWrapper to the bus, not the deserialized object.  Each IUnifiedMessageBusProvider may have to reconstruct the MessageWrapper, but the ID is particularly important.  While you might be thinking this is unnecessary, the other factor is the message might actually be sending a different data type altogether!

For the web we're not going to send binary serialized data right, or the object itself right?  We're going to want to serialized the data to a JSON string over any web IUnifiedMessageBusProvider; however, the non-web providers may need to send the actual body.

To support this case the MessageWrapper class has a generic, virtual method Unwrap so any specialized implementation can provide it's own custom logic.  Providing the method on the MessageWrapper allows other providers to have access to the real data type.

The other way around this issue is to have two different messages - one that is to be consumed generically and one that pre-converts that same exact data to JSON.  The problem I have with this approach is then you have to have inner knowledge about each provider and what types it will work with, as well as having explicity knowledge about each format a given message may be in.  The overall goal of this design is so that any consumer only has to know what message types it needs.

Since the subscriptions and serializations are based on the actual message Type, the classes need to be available in a globally available library.  I recommend that DTO messages be created from POCO models in the same library.

Particularly when working with WPF, developers may be used to more complex model objects; however, I personally don't recommend that approach for global applications consuming domain level model types.  Although it got very poor initial reception you can check out my article on using these types of models in MVVM here.  Perhaps it will make a little more sense in this context...

Using the Code

There are four main demo projects:

• A console application (TestClient).
  • Web provider
  • ZeroMQ
• A MVC web application (DemoWeb).
  • Azure
  • SignalR
• A WebApi to support the web based provider (DemoWebApi).
  • Azure
• A WPF application using PRISM (DemoWpf).
  • Azure
  • Reactive

The main demo projects implement concrete providers, and other application specific support, that won't be included in the framework.  

The code is IOC based using MEF.  That means you would normally output or copy any assembly into the main applications directory (or a configured directory) that is being exported to be injected.  If you want to change the IUnifiedMessageBusProviders available to your application simple add/remove them from the designated directory.  

For the simplicity of the demo there are direct project references from the main application to the providers in the console application and the web application.  You manually create and coordinate classes for any assemblies meant to be consumed with IOC, it's just you have to go through all the pains that IOC is meant to resolve.  It's not recommended in general with the exception of simple/concept/demo projects.

The web application and WPF do use IOC.  Both projects have some good starting examples for how MEF and IOC works in their respective environment if you have a lower level of familiarity with IOC and/or MEF.  The WPF demo shows a more complete extent of features of MEF and IOC and uses PRISM as well.

If you want to use the Azure Service Bus IUnifiedMessageBusProvider you'll need to setup your own Azure account and host your own service bus.  Replace the endpoints in the concrete implemenations with your service bus endpoint.  Next, uncomment the Export attribute and you'll be good to go.

//If using this provider, uncomment the line below then change to your Azure Service Bus endpoint.
//[Export(typeof(IUnifiedMessageBusProvider))]
public class WpfAzureProvider : TopicProvider, IDisposable
{
    private static readonly string connectionString = "YOUR_ENDPOINT";
    ...
}

If you're using the console application you'll need to change the web provider address to the WebApi project address and remove the exception:

    public class ConsoleWebProvider : WebProvider
    {
        public ConsoleWebProvider() 
            : base("http://localhost:12345/", Guid.NewGuid(), TimeSpan.FromSeconds(10))
        {
            throw new Exception("Change localhost port!");
        }
    }

From a logical standpoint, here's how the framework in the demo looks:

The actual publishers and subscribers have no specific interaction or deleaings with how to appropriately send data.  All that is needed is the instance of the Bus <font color="#111111" face="Segoe UI, Arial, sans-serif"><span style="font-size: 14px;">then the </span></font>Type and Scope of the messages they are interested in.  

Underneath the hood there is a lot more complexity.  The console application can talk to other console applications directly via it's ZeroMQ provider.  When talking with the WPF application it's going to the WebApi which then relays the message to it's Azure provider.  Since the WPF application has an Azure provider it can pick it up.

Even though the demo application may appear to have a lot of knowledge of some inner workings, it's simply because that's the focus of the demo.  Normally this would be more plug and play with completely separate assemblies for the teams working on the communication aspect.  Additionally, large scale applications could have 10 - 100 different consumers of messages.  Each of those instances should only be working with application services or the Bus, so the lack of complexity for a consumer would be much more apparent.

NOTE:  Just to clarify in case by consumer I do mean application specific code requiring a particular subscription, and not clients.  Active clients could reach 100,000+.

The web application in the demo does not have a web provider.  It's using the Azure provider for global communication.  The web provider is only used in the console application to demonstrate it works and how different platforms can be connected.  It's really not a good demo if everything had a connection to the Azure provider.

Even if you only have a single transport technology there are still advantages to the Bus.  For example, if it was desired to load balance the Azure Service Bus you could create multiple concrete providers with different endpoints.  There are several potential solutions to only send messages to select Azure Service Bus providers reducing the message load for each of them.  In the simplest implementation you would still listen to all instances through the bus.  It's also possible to create intermediate applications to reduce the subscribers for a given transport if that was a goal.

So for now at this point in the demonstrations just play around with everything a bit.  There's still a good deal of work to make this a true framework, but I'd be happy to hear your thoughts on everythig if you want to take a moment to share.

In the coming articles (with interest) I'll take deep dives into particulars of the implementations to include each transport and my experiences with them.  Enjoy!

Points of Interest

The Azure Service Bus is a breeze to setup and start using.  It is a reliable and durable service bus, but that comes at the price of throughput and latency.  Most high throughput providers are not reliable or durable.  Think of it as the difference between TCP and UDP (and sometimes that's literally the reason behind the differences).  Unless you're doing stock tickers or have another need for high throughput this is the global service bus I recommend.  For advice on improving performance for your Azure Service Bus needs check out this article.

At the time of development SignalR wasn't supporting ASP.NET Core, so the latest supported framework was used.  Something in the WebApi was causing problems with SignalR so I had to pull it out to a seperate project, but that's more realistic anyway.  Just would have kept it in the same project for demo simplicity.

SignalR has the concept of a backplan meant to coordinate messages on load balanced servers; however, this framework handles that feature - just put a global provider on each server.  I don't know about SignalR's efficiency on the backplan, but the huge advantage of the framework is the capacity to send messages accross any IUnifiedMessageBusProvider that may be on any box.

The WebApi based provider is polling based, but I created it as most any platform supports RESTful API calls (specifically mobile).

I do intend to make the framework available on NuGet and turn this into an open source project.  There's a bit of cleanup needed I want to do first, but if by chance you would be interested in contributing drop a comment.  If there is a specific transport provider you would like to implement, I'd be happy to take a look or if you finish I'd be glad to add it to the framework (except for MassTransit).

I just checked out the Google Cloud Pub/Sub and it seems very promising.  Never worked with it before, but it appears to be designed with mobile in mind.  I'll have to check it out.  It claims to be able to send up to 1mil messages per second compared to Azure's 2,000 per second so this could also be a choice for high throughput too.  Anticipate an update with a Google provider...

History

2016-09-05  Initial release.  Slightly rushed.  Anxious to work on Google Cloud Pub/Sub.  Preparring for insults...