Distributed Data Management Concepts

CAP Theorem

“Distributed data management, raises new challenges. As a consequence of the CAP Theorem, distributed microservices architectures inherently trade off consistency for performance and need to embrace eventual consistency.” AWS Documentation

  • Consistency
    • Every read receives the most recent write / error
  • Availability
    • Every request receives the a non-error response
  • Partition Tolerance
    • System continues to operate despite an arbitrary number of messages being dropped by the network between nodes

Network Partition Failure

  • The choice is really between consistency and availability only when a network partition or failure happens.
  • When a network partition failure happens should we decide to
    • Cancel the operation and thus decrease the availability but ensure consistency
    • Proceed with the operation and thus provide availability but risk inconsistency

ACID Transactions

In context of databases, a transaction can be defined as a series of operations that satisfies the ACID properties.

Properties of database transactions:

  • Atomicity
    • Each transaction is treated as a single unit
    • Either succeeds completely or fails completely
    • Prevents partial updates to the database
  • Consistency
    • Transactions bring DB from one valid state to another
  • Isolation
    • Main goal of isolation is concurrency control
    • Effects of an incomplete transaction might not even be visible to other transactions
  • Durability
    • Once transaction is committed it remains committed
    • Committed transactions are recorded in non-volatile memory

Saga Pattern

  • Long Lived Transaction that can be written as a sequence of transactions that can be interleaved
  • All transactions complete or compensating transactions are ran to amend a partial execution
  • Application in Semantically Consistent state
  • Trade-Off Atomicity for Availability
  • Sagas are a Failure Management Pattern

Overview:

  • Collection of Sub-Transactions: $T_1, T_2, …T_n$
  • Each transaction has compensating transaction: $C_1, C_2, … C_n$
  • $C_n$ semantically undoes $T_n$
  • Saga Guarantee:
    • $T_1, T_2 … T_n$ or
    • $T_1, T_2 … T_n, and C_j … C_2, C_1$
  • Requests and Compensating Requests must be idempotent
  • Saga Execution Coordinator
    • Stores and interprets Saga’s state machine
    • Interprets & writes to Saga Log
    • Executes Requests of a Saga by talking to other services
    • Handles failure recovery by executing compensating requests