CQRS Architecture with Kafka

Apache Kafka is an amazing tool for microservice communication, it's easy to send and receive messages in a variety of programming languages and frameworks. And with Confluent Cloud's really-quite-good Kafka SaaS, we don't even need to bother setting up brokers.

With such an easy road to getting started, we have more time to consider the architecture of our services, and the naming of topics and messages. This is especially important if we're starting to build a system that will eventually grow quite large, with lots of inter-depencency.

How does CQRS help?

Command Query Responsibility Segregation is an architectural pattern to help us avoid a spider's web of dependency chaos as system complexity grows.

Many of us have seen this before, especially in the microservice early days; while developers rejoiced at the death of the monolith, they were often confronted with large set of microservices that was just as hard to extend and maintain as the monolith was.

Segregating the responsibility of requesting an action from querying state it is a simple way to add clarity and ease extensibility and maintainability.

How do we clearly segregate these responsibilities?

Naming is the hardest thing in software development. The following naming principals help us to keep this as simple as possible;

'Command' and 'Event' topics

In a CQRS system, there are only two types of topics. Command topics carry messages that request an action and are usually short lived. Event topics carry messages that contain the result of an action, and are usually long lived.

'Control', 'Logic' and 'View' services

It can also be useful to have separate services for sending Commands, for processing them, and for providing RESTful endpoints for querying the state of any created entities.

Control services: Send Commands

Logic services: Receive the Commands, perform any actions, and send Events

View services: Receive Events and provide RESTful endpoints to query state

Let's take a simple example for a 'create account' use case...

The above diagram shows the three CQRS services, and two topics; account-commands and account-events. This is a simple example, but this design provides the flexibility to cope with the complexity that might build over time. For example:

• The View service can easily listen to other Event topics (users or permissions for example), to build richer REST endpoints as required.

• Other view services can also be interested in these account-events.

• The control service can send other commands as part of a 'Saga' of operations that might be needed as part of an account opening process (for example, sending a confirmation email, performing a fraud check).

Finally, it's worth noting that for some simple operations this might be overkill - it may be perfectly reasonable to combine a control and logic service, removing the need for a command topic.