Service Architecture for OLTP --- Part 1. Introduction

OLTP (Online Transaction Processing) imposes the most challenging requirements for Micro-service based Web application architecture, in terms of the following properties:

• ACID on state changes of the system.
• Scalability of data over commodity hardware.
• High throughput and low latency on online request handling.

In this series, I will analyze those requirements and describe the architecture and programming patterns of OLTP Web service and the underlying data store. The following view of such system will greatly help understanding both the requirements and the architecture:

An OLTP system is a distributed system that encapsulates a set of states (system states) under an interface for querying states as well as to changing system states. A request to such system may read a view of the system states, perform change of states change and/or trigger eventual state changes. Both view of states and change of states are performed as ACID transactions.

General Architecture

1. Synchronous Request Processing
   This is the mechanism that handles the interactions with users, in other words, the code path that processes a request from user. In general, a user request incurs both read of system states as well as mutations on system states. For example, in the convention of REST over HTTP, a GET request sometimes only encapsulate reads, while a POST/PUT/PATCH/DELETE sometimes encapsulate both reads and mutations on the server side. I will refer to the former as RO (read-only) user request and the later as RW (read-write) user request.
   In an OLTP Web Service, it could be required that both RO request and RW request are ACID. It means the following:
• A RO user request performs a READ on system states with some ACID requirements. For example, it may need to be a snapshot READ with serializable isolation level (I will cover isolation level later).
• A RW user request performs a Read-Write operation on system states with some ACID requirements. For example, it may need to be a RW transaction with serializable isolation level.


2. Asynchronous Processing
   Asynchronous processing occurs as as an eventual effect of synchronous processing of a user request. Using an example of order checkout, user clicks a button "Confirm Checkout" at the final step of checkout and this sends a "checkout" request to Web Service. All the following steps may occur within the synchronous processing of such request: deducting inventories of products, payment, creating an order object, etc. But the post-purchase processing, including order processing and fulfillment, will occur eventually but automatically as a result of successful handling of "checkout" request but outside the context of handling that request. Asynchronous processing usually have the following properties:
   • It occurs eventually.
   • It may be multi-stepped. And each step may be an ACID transaction on the system states.
   • It needs to occur exactly once or it needs to be idempotent. And this requirement applies to each step.
   Async processing is usually implemented with lower level abstraction of event driven processing or higher level abstractions such as Workflow, which will be covered by a post in this series.


3. Distributed Data Store
   Both implementations of synchronous processing of request and asynchronous processing are mostly stateless in a typical architecture. And it leaves only the data store to be the stateful component in this architecture to store all the system states. Most people think of the data store as a relational database, which could be true for most implementations. There is further a perception that ACID transactions are only implemented within such data store, but this is not really true for a typical Microservice OLTP architecture, where transactions are actually implemented at both the microservice level as well as database. The perception actually came from the convenience implied in the traditional architecture of monolithic data store, where all computations are performed on a single centralized logical database. But in modern architecture, data store is distributed, in particular, sharded (partitioned) for scalability over commodity hardware. The distributed data store has the following properties:
• System states are partitioned into different logical stores (see more in "Partitioning of the System"). And each logical store is further sharded based on computation locality for scalability.
• Each shard is replicated for HA and DR. In a typical architecture, replication is implemented based on quorum-based cluster within a data center. Paxos and Raft are two common algorithms for forming consensus on the states replicated over a cluster. MySQL's group replication is one example implementation. Asynchronous replication is usually performed across data centers for data center level disaster recovery.
• ACID transaction is supported within the scope of a shard.
• Transaction beyond the scope of a shard is supported either as a distributed DB transaction across multiple shards of a DB or a global transaction across multiple microservice instances (, each of which performs read and write on its own DB shards).

Partitioning of the System

Partitioning of system states based on separation of business logic into domains is one of the most important characteristics of such system architecture. Such partition not only enables efficiency of locality of data and computation, but also enable the bounded context for Domain-Driven Design of software. Such partition is across the whole stack:

• Data store is partitioned, with each partition stores states of one business domain. I refer to such partition as a "logical DB" or a "DB".
  For instance, an Ecommerce site might have the following logical DBs: Products (including categories, products, inventories, etc), Users, etc. And those different logical DBs may be backed by different data store implementations. For example different data stores may be selected based on the requirement of transactions, processing latency, throughput, scalability and cost for each specific business domain. And we reply on microservice layer to hide the heterogeneity of data stores.
  *Note that each "logical DB" may be further sharded based on connectivity of records in the logical DB for scalability. That will be referred to as "DB shard".
• Computations including both view generation of system states and updates on system states, are also partitioned according to the partition of those states. Computations are usually partitioned into different microservices, though some of them may also be implemented as procedures running on DB for efficiency purposes (see transaction language in Part 2).

This series will cover the following topics with details, which will be analyzed and examined with the examples of ECommerce systems. As of 02/25/2020, only Part 1, 2 and 4 are published.

Part 2. Transaction in Database
The basic but fundamentally important understanding of transaction processing in modern databases.

Part 3. stributed Transaction and Global Transaction
How to implement cross-DB-shard and cross-DB, cross-microservice OLTP.

Part 4. SAGA pattern in Microservices
Explain workflow-based eventual-consistence processing as a replacement strategy for global transaction, with example code.

Part 5. Workflow for Asynchronous Transaction Processing

Continuation from the SAGA pattern. Look into workflow-based asynchronous processing and its implementation based on open source workflow engine.

Part 6. Distributed Database
Sharding and replication of databases. Look into DB cluster management.

Part 7. Schemaless and Object Stores over Relational DBs

Strategies to implement object stores with flexible schema over relational DB.