Before reading, watch this.

To stay nimble as our R&D grows, the microservices architecture was chosen to allow teams to deploy code to production without any concern to what other teams are currently doing.

Some extra more minor advantages:

  • Horizontal Scaling - You can scale different parts of the system more easily. One service is CPU bound? Put it on a CPU optimized machine on AWS. Another service is doing work on data coming in from a queue? Scale it up & down by monitoring the number of items in the queue.
  • Availability - A service crashes because of a bad nil dereference, and it doesn’t crash the whole system.
  • Best tool for the job - You can use the most appropriate programming language for the problem at hand, if for example it has some specific library you need. Or maybe you just need real time performance so you use C / Rust for example.

The important thing to remember though, is that microservices introduce a lot of complexity to a system.

This page was created to discuss the complexities, and increase your awareness in the designing step before writing code.

The problem with a distributed system

The difference between a monolith and microservices boils down to how you execute an API. In a monolith you call a function, and in microservices you communicate over a socket. This hard boundary is exactly what allows you to deploy code separately from other teams’ code.

Another name for this kind of system is called a distributed system. While a distributed system introduces network partitioning, it must lose either availability or consistency, famously named the CAP theorem.

The CAP theorem states that a system can guarantee only 2 of the following 3:

  • Consistency - Reads receive the most recent write.
  • Availability - All requests succeed, no matter the failures.
  • Partition Tolerance - The system continues to operate despite dropped / delayed messages between nodes.

To understand why this is, imagine a system operating on a single machine. It is definitely partition tolerant, as messages in the system are not sent through something like a network, but through function calls operating on the same hardware (CPU / memory). It is also consistent, as the state of the data is saved on the same hardware (memory / disk) that all other read / write requests operate on. Once the machine fails (be it software failures like SIGSEGV or hardware failures like the disk overheating) all new requests to it fail, violating availability.

Now imagine a system operating on 2 machines with separate CPUs, memories, and disks, connected through some cable. When a request to one of the machines fails, for whatever reason, the system can choose to do one of the following:

  • Cancel the request, thus sacrificing availability for consistency.
  • Allow the request to continue only on the working machine, meaning our other machine will now have inconsistent data (reads from it will not return the most recent write), thus sacrificing consistency for availability. When a system does this, it is called eventually consistent.

Most microservices architectures choose to allow the request to continue, becoming eventually consistent, by separating the data in different tables and sometimes even different databases altogether (for performance for example).

Distributed state

You could counter argue the following: “let’s store all the state in the same single DB instance”.

This is a valid argument and I love you for thinking that, but because database models change pretty often, you lose the most important microservices advantage, which is the independence between teams.

If one team’s migration causes another team’s services to be needed to upgrade & deploy, your teams are not isolated and flexible enough.

This is the main reason we use gRPC. The protobuf format is designed for backwards and forwards compatibility. In protobuf, fields are given explicit numbers to them (like tags), which allows adding new fields or removing old ones in a backward and forward compatible manner:

  • Backwards compatibility - Old clients can communicate with newer versions of the service, ignoring any fields they don’t understand.
  • Forwards compatibility - Newer clients can communicate with older versions of a service, as long as they don’t require any of the newer fields that the older service doesn’t understand.

You can also utilize other features for even better compatibility, like:

  • Optional fields - Allow you to introduce new fields without breaking older clients.
  • Default values - When a client sends a protobuf message without setting all of its fields, the receiving service can rely on default values to set them.

How to deal with eventual consistency

Eventual consistency introduces complexity. Because state is distributed, if you need to JOIN / UNION states of 2 or more tables owned by different services, you need a distributed transaction.

You can run a distributed transaction using either Two-phase commit or SAGA, but they are notoriously complex and difficult to debug.

There are other ways to handle eventual consistency.

The simplest is by ensuring proper service separation, where there’s no case of 2 services that should actually just be 1 service, for example. Consolidating services and “by accident” their state, will allow you to make a non distributed transaction (simple DB query) on the data.

Another solution is to simply not at all. For some use cases, it might be ok to read stale data. If you’re aware about it, and think about the different scenarios that might happen, understanding that there’s no issue in reading stale state, then you’re fine.

E-commerce example

Imagine you’re running an e-commerce platform including these two services:

  • Inventory - Tracks which products exist in the inventory.
  • Recommendation - Suggests related products based on what’s in the customer’s cart.

Now, let’s say a customer adds a product to their cart, which makes it go out of stock. At the same time, to another user, the recommendation service suggests to buy this item.

Instead of enforcing strong consistency and make a distributed transaction to both service’s states, you can let the recommendation service read stale inventory data. Now that the recommended product is temporarily out of stock, the user might click on it and receive a notification like “Back in stock soon!”. This is acceptable for user experience, and the system avoids the complexity of distributed transactions.

Everyone is happy!

Stateless services

A specific service can be horizontally scaled to run multiple instances (pods in k8s), which means that state should live inside an external service, most likely a database.

Sometimes it might be tempting to query state from the DB, storing it in memory for caching purposes. This is dangerous and should be handled with care, as again, it further increases the eventual consistency problem. You just introduced more distributed state, for the benefit of performance, but at the cost of everything we’ve already discussed.

Unless you have the perfect cache eviction strategy, where you are always notified when data is modified, right as it happens or you don’t care reading stale data, think twice before doing it.

When should you create a new service?

If it’s not obvious why you need another service, like separating a GPU heavy function to run on a machine with a strong GPU, you’ve got an abstract service organization problem.

Here you’ve got to be honest with yourself, asking the following questions:

  • Am I working on completely new data, never seen before in the system?
    • If yes - Lean towards a new service.
  • Do I need to create foreign keys / pointers to data of existing services?
    • If yes - Think again. Maybe you simply want to add a new gRPC endpoint to an existing service.

From the questions, you can start to tell, the abstract service organization problem is usually just an empirical data problem in disguise.

Don’t follow SOLID blindly. If you do single responsibility hardcore, what happens is you end up with 10k services with 1 function each.

~ Follow the data.