Resilient applications with SAGA

Building resilient applications means understanding that, as the architecture becomes broader and more complex, the chances of something going wrong increase. Even when setbacks happen, a resilient application can keep operating and recover from those failures.

When working with software, we deal with many kinds of transactions, which we can easily illustrate in a database context. Consider, for example, a simple bank transfer between two accounts:

BEGIN TRANSACTION;

INSERT INTO bank.transaction (id, type, amount) VALUES (1, 'debit', 100);
INSERT INTO bank.transaction (id, type, amount) VALUES (2, 'credit', 100);

COMMIT TRANSACTION;

What happens if the first insert succeeds and the second one fails? In that case, the transaction is not committed, so none of the inserts are persisted in the database, keeping it in the consistent state it had before the transaction.

A common way to think about transactions is through the ACID concept: Atomicity, Consistency, Isolation, and Durability. These principles are fundamental for ensuring the reliability and integrity of operations in database environments, especially when concurrency and system failures are involved.

But not every transaction happens inside a database.

Working with transactions

Transaction is the abstraction of a set of operations that must be treated as a single logical unit, where, for the transaction to succeed, all operations must succeed or be undone.

When working with distributed transactions, we can use a fast food app order as an example. It is expected that every service involved in the transaction is available, from the Order service to the Delivery service.

If those two services have 99.5% availability, the transaction availability will be 99%, and each additional service will reduce the transaction’s availability percentage. This leads us to the CAP theorem, created by Eric Brewer, which says that a system can only have two of the following three properties:

• Consistency: In this context, it means that all services involved in a transaction, such as placing an order and debiting inventory, present a consistent view of the data. In other words, all read operations reflect the most recent version of the data after a write. Maintaining consistency means avoiding divergent responses to queries, regardless of which node is queried.
• Availability: Refers to the ability of a distributed system to respond to requests, even when failures happen. In our example, availability would be crucial to ensure that services remain accessible to accept orders, process payments, and update data. Even if a specific service fails, the remaining services continue operating so the application remains functional.
• Partition Tolerance: Means that the system keeps operating even when communication failures happen between nodes, resulting in network partitions. For example, even if the payment service cannot communicate directly with the order service, both services continue to work, ensuring that customers can place orders and make payments.

Today, it is common to choose availability over consistency. To handle the complexity around data consistency in microservice architectures, we use mechanisms for building loosely coupled applications with asynchronous services.

Saga

Saga is a mechanism for maintaining data consistency in a microservice architecture without using distributed transactions. The Saga pattern is a sequence of local transactions where each transaction updates the data of a single service.

With Saga, we work with three types of transactions:

• Pivot Transaction: Refers to a strategy in which a transaction performs a series of operations but only confirms those operations after the successful execution of a crucial or “pivot” step. If the crucial step fails, the transaction is rolled back.
• Compensable Transaction: Compensation transactions are used to undo operations performed by a previous transaction that failed or was interrupted. Instead of directly reverting the original operations, a compensation transaction executes inverse operations to restore the system to its previous consistent state.
• Retriable Transactions: Retriable transactions are transactions that can be safely repeated if a temporary failure happens. If a transaction fails due to transient conditions, such as a network failure, it can be retried without causing problems in the system.

Considering our fast food app, below we can see the creation of an Order using Saga:

We can consider the following transaction flow:

1. Order Service: Creates an order with the APPROVAL_PENDING status.
2. Consumer Service: Checks whether the customer can place an order.
3. Kitchen Service: Validates the order details and creates a ticket with the CREATE_PENDING status.
4. Accounting Service: Authorizes the customer’s credit card payment.
5. Kitchen Service: Changes the ticket status to AWAITING_ACCEPTANCE.
6. Order Service: Changes the order status to APPROVED.

Saga uses compensation transactions to perform rollback changes, and this flow can fail for several reasons:

• The customer is not allowed to create orders;
• The restaurant is not able to receive orders;
• The customer’s payment method is declined.

The first three steps of the flow can end in compensation transactions, while the fourth step is a pivot transaction because it is the go/no-go point for the rest of the transaction. The final two steps are retriable transactions because they will be executed and eventually succeed.

Implementing Saga consists of coordinating its steps, and there are two ways to do that: orchestration or choreography.

Orchestrated

In this scenario, there is centralized logic that coordinates each Saga step. Below, we can see how the steps would execute in the order creation flow.

Pros:

• Simple dependencies: This model avoids cyclic dependencies. Since the orchestration class knows and triggers the Saga steps, the same is not true for the services. As a result, the orchestrator depends on the participants, but the reverse does not happen, avoiding cyclic dependencies.
• Loose coupling: Each service implements an API called by the orchestrator, so it does not need to know about events from other participants published in the Saga.
• Better separation of concerns and simpler business rules: The coordination logic is centralized in the orchestrator, so Saga participants only need to care about their own business rules.

Cons:

• Centralization risk: There is a risk of centralizing too much business logic in the orchestrator. To avoid that, we should keep in mind that this class should only be responsible for sequencing Saga steps and should not contain business logic.

Choreographed

Here, each participant publishes and handles events independently, deciding how to perform its part.

Pros:

• Simplicity: Each service publishes events when entities are created, updated, or deleted.
• Loose coupling: Participants listen to events without knowing who creates them.

Cons:

• Harder to understand: Without an orchestrator, it is harder to understand the Saga flow.
• Cyclic dependency between services: Saga participants may listen to each other’s events, creating cyclic dependencies.
• Coupling risk: Each Saga participant needs to listen to the events that affect it. For example, the Account Service must listen to every event where the customer’s credit card is charged or refunded. As a result, there is a risk of changing the implementation of the order lifecycle.

Conclusion

This is a good way to build resilient applications while keeping low coupling, high availability, and eventual consistency.

Orchestrated Saga can be used in different scenarios, from the simplest to the most complex, and it serves each of them well. Choreographed Saga fits very well with event-driven architectures; outside that context, it is recommended for simpler flows. There are also tools that help trace events in a choreographed Saga, using OpenTelemetry for example.