Designing a Large Network Connected Distributed System using a Message Bus - Part 1

Introduction

A message bus is a fundamental component of modern distributed systems, serving as a vital communication infrastructure that facilitates the exchange of messages between applications or services within a network. It plays a crucial role in enabling efficient and reliable communication, coordination, and collaboration among various system components. By acting as a central hub for message transmission and receipt, the message bus provides a flexible and decoupled approach to inter-application or inter-service communication, allowing for the seamless integration of diverse technologies and systems, enabling ease of addition or removal of systems without impact on others. Message bus implementations may vary, with a typical example being one found on a trading floor, as shown below.

To implement a message bus architecture, several essential components need to be established:

1. Messaging infrastructure: The messaging infrastructure serves as the foundational system for facilitating message exchange between applications. It provides the necessary mechanisms and protocols for sending and receiving messages across the network.

2. Message format: A standardized message format is crucial to ensure interoperability between applications. By defining a common structure and data format, all participating applications can understand and interpret the messages accurately. This ensures seamless communication and avoids data misinterpretation or errors.

3. Command set: Establishing a common set of commands is essential for enabling applications to communicate with each other in a standardized manner. This command set defines the actions or operations that applications can perform when sending or receiving messages. It promotes consistency and simplifies the integration process by providing a uniform interface for interaction.

4. Message router: The message router plays a critical role in the message bus architecture. It is responsible for intelligently routing messages between applications based on the content of each message. The router examines the message attributes, such as destination address or topic, and determines the appropriate destination application(s) for the message. This ensures that messages are delivered to the intended recipients efficiently and reliably.

By establishing these key components within the message bus architecture, organizations can create a robust and efficient communication infrastructure that enables seamless message exchange and collaboration between applications or services.

Here are a few examples of popular message brokers:

1. Apache Kafka: Kafka is a distributed streaming platform known for its high throughput, fault tolerance, and scalability. It provides durable message storage and enables real-time data processing and stream processing.

2. RabbitMQ: RabbitMQ is a robust and flexible message broker that implements the Advanced Message Queuing Protocol (AMQP). It supports various messaging patterns, including point-to-point, publish-subscribe, and request-reply.

3. ActiveMQ: ActiveMQ is an open-source message broker that implements the Java Message Service (JMS) standard. It offers features like message persistence, message filtering, and support for multiple communication protocols.

4. Apache Pulsar: Pulsar is a distributed messaging and streaming platform that combines messaging queues and publish-subscribe semantics. It focuses on scalability, multi-tenancy, and strong durability guarantees.

5. AWS Simple Queue Service (SQS): SQS is a fully managed message queuing service provided by Amazon Web Services (AWS). It decouples applications and enables reliable communication between different components in a distributed system.

A very basic flow of a message bus communication is as shown below:

(picture from Wikipedia.org)

At a high level, a message bus design typically involves one machine broadcasting messages to multiple machines, resulting in a one-to-many message distribution. However, in practice, there are various combinations of bi-directional message flows that can be used, including one-to-one, one-to-many, and one-to-group scenarios.

When implementing a Message Bus, the "Observer" design pattern is commonly employed, and there are three ways to implement it from scratch:

1. Using core Windows/Linux socket APIs: This approach relies on the native socket APIs provided by the operating system. While it is platform-dependent, it offers complete control over the implementation. Developers can directly utilize socket functions and manage low-level network communication. This method provides flexibility, but requires more effort and expertise in socket programming.

2. Utilizing a portable library like Boost: This approach adopts a platform-independent approach by leveraging a portable library such as Boost. Boost provides a comprehensive set of libraries, including ones for networking, threading, and interprocess communication. It offers a balanced and popular approach, combining ease of use with portability across different operating systems.

3. Using a messaging-specific library like ZeroMQ: ZeroMQ is a high-performance messaging library that provides practical solutions for building message bus architectures. It offers advanced messaging patterns, including publish-subscribe, request-reply, and pipeline, with excellent performance characteristics. This approach is particularly suitable for commercial applications that require high throughput and low-latency messaging.

//A simple interface can be implemented as shown below.

//singleton controller to manage messages and broadcast them to clients
class MsgQMgr
{
public:
    //create thread-safe singleton object
    static MsgQMgr* getInstance();
    //start processing messages
    void start();
    //register new clients
    void addClient(std::shared_ptr<Proxy> proxy);
    //un-register new clients
    void removeClient(std::shared_ptr<Proxy> proxy);
    //add messages to a message queue
    void addMsg(const string& msg);
    //broadcast messages to all clients
    void broadcastMsg(const string& msg);
private:
    MsgQMgr() {}

private:
    static MsgQMgr * instance_;
    // Define a deque to hold pending messages
    static std::deque<string> messages_;
    static std::mutex msg_q_mutex_;
    static vector<std::shared_ptr<Proxy>> clients_;
    static std::mutex clients_mutex_;
    static std::condition_variable condition_;
};

//client proxy
class Proxy : public std::enable_shared_from_this<Proxy>
{
public:
    //create client proxy
    Proxy(tcp::socket socket);
    //start proxy
    void start();
    //update message
    void update(const char* msg, size_t length);
    virtual ~Proxy() {}

private:
    //read message
    void read();
    
private:

    tcp::socket socket_;
    enum { max_length = 1024 };
    char data_[max_length];
    bool is_connected_;    
    std::shared_ptr<Proxy> proxy_;
};

Ultimately, the choice of messaging infrastructure depends on the specific needs and requirements of the application. Careful evaluation of available options is essential to select the most suitable approach that aligns with the project's requirements, platform compatibility, performance, and development effort.

References

1. Design and implementation of a decentralized message bus for micro-services
   [Published in: 2016 13^th International Joint Conference on Computer Science and Software Engineering – (JCSSE)] (https://ieeexplore.ieee.org/document/7748869)
2. Message Bus/Broker - Python and C++ (https://www.youtube.com/watch?v=F-XtGVo_BU0)

History

• 8^th May 2023: Initial version