Implementing Idempotence

We have been and would be arguing in this book for building a system consisting of smaller business components which do not share their database with each other. This mean that we would have multiple databases. In any system where data needs to be consistently stored in multiple data stores we need to worry about transactions. But this problem goes beyond that as most (interesting) application we make needs to integrate with other application over the network or Internet. Since two-phase commit (as it leads to distributed locks) is not our favourite solution for this any more we need an alternative, as the problem still exists even if we don't like the solution. Idempotent operations used along-side single resource transactions is quite commonly proposed as a solution. While this as a high level direction is useful but when it comes to implementation we need more detail than this. In this chapter we would look at implementation techniques in various scenarios.

Before we dive any further we need to differentiate between transaction from business and system perspective. As the name, business transaction is something we as human user care about. For example, when we book our tickets online, we expect that the money would be withdrawn from our account and whatever we bought would be delivered to us. We as a user don't know nor care about how many software transactions this results in. But as software people we might see these are multiple software transactions (or just transactions), order, payment and ticket-delivery. A business transaction encapsulates multiple software transactions. In an two-phase commit world we would be able to perform all or none and let the user know if it all succeeds or not.

A software transaction may deal with one or more data stores. In this chapter we would look at techniques to implement various scenarios when a software transaction performed with multiple resources, with or without user involvement, with all resources supporting transaction but without using 2-phase commit between them. On multiple resources we can perform either create, update or delete, leading to multiple permutations. A software transaction can be performed on request of a user or triggered via another transactional resource (e.g. a message sitting in a queue).

What is an idempotent operation?

Idempotence is the property of certain operations in mathematics and computer science, that they can be applied multiple times without changing the result beyond the initial application.

-- Borrowed directly from Wikipedia

Updating a row in the database is good example of it. If we don't care about database's transaction log, auditing logs etc, then an update performed any number times with the same data doesn't change the state of database.

User operation resulting in state change of multiple data stores

Lets start-off with the example of user confirming payment for a seat in a movie theatre. From the business perspective two things are important (triggered) here reservation of a seat and successful payment. Since we do not have atomic transaction across booking and payment databases, there are two failure scenarios which need to be considered. Assuming we do booking first and payment next.

1. Booking is successful but payment fails

2. Booking fails but payment is successful

We can easily reduce it to just one scenario by making sure that when the booking fails we do not try payment at all. (There is additional scenario where both of them pass but something fails before we tell this to the user. We are not considering this scenario as this is something one would have to deal with even when one has two-phase commit). The code below illustrates how we would implement such a business transaction.

public BookingResponse book(BookingRequest bookingRequest) {

    String businessTransactionId = bookingRequest.hasBusinessTransactionId() ? bookingRequest.businessTransactionId() : UUID.randomUUID().toString();

    BookingResponse bookingResponse = new BookingResponse(businessTransactionId);

    String reservationId = reservationService.reserve(businessTransactionId, bookingRequest.showId(), bookingRequest.seats());

    bookingResponse.reservationId(reservationId);

    try {

        String paymentId = paymentService.pay(businessTransactionId, bookingRequest.paymentDetails());

        bookingResponse.paymentId(paymentId);

    } catch (Exception e) {

        bookingResponse.paymentException(e);

    }

    return bookingResponse;

}

The book method other than doing business as usual also does few things differently.

Business Transaction Id

A unique number identifying a business transaction. All individual operations within a business transaction can use this for correlation. Here this number is passed on to reservation service so that it can check whether if this is a retry of a reservation already done before. This number is also returned in BookingResponse so that the caller can use it when calling the book method again if payment fails. e.g. In a web application one can put this number in user session.

It doesn't do any error handling for reservation as it is the first operation and a failure would fail the book operation (this would depend on your application). Lets look at how do we make reserve method idempotent.

public String reserve(String businessTransactionId, long showId, List<String> seats) {

    Reservation existingReservation = allReservations.findByBusinessTransaction(businessTransactionId);

    if (existingReservation != null) return existingReservation.id();

    Reservation reservation = allReservations.new();

    reservation.businessTransactionId(businessTransactionId);

    Show show = allShows.get(showId);

    for (String seatId : seats) {

        Seat seat = allSeats.get(seatId);

        reservation.add(show, seat);

    }

    allReservations.add(reservation);

    return reservation.id();

}

This is pretty straightforward. If a reservation for same business transaction id does exist then we don't need to do anything.

Since business transaction id is used in both payment and booking, it makes it easier to resolve scenarios like which bookings should be purged because there has been no payment for it. This can arise when the user doesn't retry when payment fails once.

Idempotence is also useful for a single transactional resource, because errors can happen anywhere in the code after transaction commit resulting in a retry.

If you do not want to pollute all your persisted data with business transaction ids then you can choose to use your own business natural keys for finding a reservation. For example we could have used a combination of customerId, showId and seatIds.