Quite often in technical interviews, when the conversation turns to distributed systems, you will hear a question about a pattern called Idempotency. In short, an Idempotency Key is a unique identifier of an operation that lets a system recognize that it has already seen this operation and not execute it again.
I have already covered idempotency in the context of HTTP requests, so I recommend reading about it as well. But in an interview, a definition alone is usually not enough. It is much better to show what real problem this pattern solves.
Let's walk through a simple payment example.
A system without Idempotency
Imagine a user buying a product on a website. The browser sends a request to our microservice. The microservice accepts it and publishes a message to a broker, for example RabbitMQ.
Suppose the message delivery model is at-least-once delivery. This means the broker guarantees that a message is delivered at least once, but under certain circumstances the same message may be delivered again.
In the diagram above, RabbitMQ passes the message to one of our workers. The worker receives it and performs the business operation, which in our case is charging the user's card.
After successful processing, the worker must send the broker an ACK, a confirmation that the message was processed successfully.
Now imagine the following situation:
The cause can be anything: the worker crashed, a network timeout occurred, the process restarted, and so on. And here is the important point.
RabbitMQ does not know that the money has already been charged. All it knows is that the message was sent but no ACK was received. Since at-least-once delivery is used, the broker will redeliver this message after some time, possibly even to a different worker.
As a result, a single logical operation may be executed multiple times. For payments, this is obviously unacceptable.
This is where the Idempotency Key comes in
Now let's look at the same system, but this time using an Idempotency Key.
The architecture barely changes. The main difference is that every logical operation now has a unique idempotencyKey.
For example: payment:order-123
Before executing the business logic, the worker tries to register this key in a fast store, for example Redis.
The logic goes roughly like this:
What Idempotency gives you in the end
It is important to understand that Idempotency does not guarantee that RabbitMQ will not deliver a message twice.
The goal of Idempotency is to ensure that redelivery of the same message does not result in the business operation being executed again.
In our example, RabbitMQ may deliver the payment message several times, but thanks to the Idempotency Key, the system recognizes the duplicates and does not charge the money again.