Introduction

In this part, we’ll go through the changes that happen mainly to data once a monolithic system has been pulled apart into separate services each with its own domain. Every service abides by the bounded context rule in Domain Driven Design where each domain has its own application code and data together.

I made several articles about this book where we discussed different chapters of it. I’d recommend checking them out to make sure you have a complete understanding of what’s going on. In a nutshell we’re pulling apart a huge monolithic application into several coarse grained services, and pulling apart the data as well.

When data is pulled apart it needs to be stitched back together to make the system work. So the main hiccup is figuring out which service owns what data and how to manage distributed transactions. Also how any service can access data they need (not own)

Assigning Data Ownership

The main question that arises here is: which service owns which data?

A general rule of thumb for assigning table ownership is that services that perform write operations to a table own that table. This works well if a single service writes to the table but it gets messy when multiple services have to write to the same table.

A simple example is as follows;

We have 3 services and 3 databases. As we observe all 3 services write to the Audit Table.

The catalog service writes to the product table and the inventory service does too.

These examples make assigning data ownership a very complex task.

There are 3 common scenarios encountered when assigning data ownership to services:

1. Single ownership

2. Common ownership

3. Joint ownership

We’re going to dive into them and explore several techniques for resolving these scenarios

Single Ownership

Occurs when only one service writes to the table. Very straightforward and very easy to resolve. For example above the wishlist service is the only one that writes to the wishlist table making it a single ownership scenario.

Knowing this information makes us conclude that the wishlist table is part of the bounded context for the wishlist service.

It’s a lot easier to address single table relationships first when approaching these kinds of problems before moving on to complex scenarios

Common Ownership

Occurs when most or all of the services need to write to the same table. Looking back at our example we see all 3 services writing to the audit table). Since all 3 write to it’s really difficult to say who owns this table domain wise.

A proposed solution is to put the audit table in a shared database or schema since it receives writes from everywhere. However this has its own set of problems:

1. Change control where if the schema changes you’ll have to revisit all the services writing to it and update them accordingly

2. Connection starvation due to the amount of services connecting to it

3. Scalability and fault tolerance where if it goes down it causes chaos where a lot of services encounter issues related to auditing

A popular technique for addressing this is to assign a service for auditing that owns the audit table.

So any services that need to write audits go through the audit service to write data and not access the database directly. This has a huge impact on the design and a lot of benefits

1. If no acknowledgment is required a buffer (queue) can sit between the services and the audit service. Where it can process audits at its own pace

2. Also becomes more fault tolerant where it’ll be easier to scale the audit service independently.

Applying what was said our design should look like this;

Joint Ownership

Occurs just like the common ownership but the catch is only a couple of services within the same domain write to the same table. Not most of them as previously described in the common ownership.

Looking back at our first example, only the Catalog and Inventory services perform write operations on the product table.

There exists several techniques to solve this ownership problem

1. Table split

2. Data Domain

3. Delegation

4. Service Consolidation

Let’s go one by one discuss them

Table Split Technique

The table split breaks the table into multiple tables where each service owns a part of the data it’s responsible for.

Looking at our example if we were able to break down the product table into 2 tables where the inventory can own the data it manipulates and so can the catalog that means we did a table split technique. This highly depends on the nature of what the inventory service writes so if it updates counts for example we can extract that column and add a product id foreign key as a reference so the inventory has its own table.

It moves the joint ownership to a single table ownership. However the overhead is consistent communication between both services to ensure data is synced correctly between them and that it remains in a consistent state.

If a new product got added for example, the catalog service needs to communicate that to the inventory service sending the id and inventory counts to it. If removed vice versa also.

But a lot of questions arise when syncing data between 2 tables:

1. Should the communication be synchronous or asynchronous between both services?

2. What happens if the catalog service want to communicate to the inventory service and found that its not available? It’s a availability versus consistency question

Choosing availability means that the catalog service must always add or remove regardless if the inventory service is working or not.

Choosing consistency means that adding or removing would fail if one of both services is down.

So it depends on the business requirements and what you need in order to be able to answer or make a decision.

Delegate Technique

In this method, one service is assigned a single ownership of the table and becomes the delegate. Any other service communicates with the delegate to perform updates on its behalf.

The main challenge here is to know which service to assign as the delegate (the sole owner of the table) we have two options;

1. Primary domain priority

   Where we assign the table to the service that most closely represents the primary domain of the data (the service that does most of the CRUD operations for a entity in that domain)

2. Operational characteristics

   Assigning the table to the service needing higher operational characteristics such as performance, scalability, availability and throughput

If we look at the following joint ownership scenario

The catalog service performs most of the CRUD operations on the product table because it creates, updates and removes products. Getting product information whilst the inventory service is responsible for retrieving and updating inventory count as well as knowing when to restock if the inventory count is too low.

Applying the priority domain technique results in the following

The catalog service would be assigned as the single owner of the table. The Inventory service must communicate with the catalog service to access that table.

Delegate techniques always force inter service communication between the services forcing them to communicate to update data. Also type of communication is key here because in synchronous communication the inventory service must wait for the inventory to be updated by the catalog service. This impacts performance but ensures data consistency. Using asynchronous communication boosts performance but makes data eventually consistent.

With the operational characteristics priority option, the ownership rules would be reversed because the inventory updates occur at a much faster rate than static product data. In this case the ownership would be assigned to the inventory service.

With this option updates to the inventory can use direct database calls instead of remote access protocols. Therefore making inventory operations much faster also the most volatile data is kept consistent (inventory counts)

However one major problem here is the domain management responsibility. The inventory service is responsible for managing inventory counts, not creating, deleting and updating the products (and potentially error management too).

Service Consolidation Technique

The delegate approach highlights the primary issue with joint ownership which is service dependency

The service consolidation technique solves this by combining multiple table owners into a single consolidated service.

Combining multiple services into one creates a coarse grained service. Which increases the testing scope as well as the deployment risk (breaking something else in the service when a new feature is added or a bug is fixed). Consolidation might affect the overall fault tolerance of the system since the whole service fails together.

One of the caveats is that they both have to scale equally as well even if it wasn’t necessary for one service to do so.

Summary

This part covered different data ownership scenarios and techniques used to choose which service owns which data.

Summarizing everything we said, the design should look like this now:

1. We used single table ownership for the wishlist service

2. For common ownership we created an audit service with all other services sending messages via a queue (asynchronous).

3. Finally for the joint ownership between the catalog and the inventory with the product table, we chose the delegate technique domain priority assigning the table to the catalog service where the inventory service sends update requests to the catalog service.

That’s it for part 1. In the next part we’ll go through distributed transactions and the caveats that happen when data is pulled apart. Stay tuned!