Replication

View as Markdown

Status: this page describes a DESIGN, not a shipped feature. Do not deploy it. Everything below the Cluster Modes table is the intended architecture. In the shipping engine, as of 2026-08-21:

  • A replica never applies streamed records to its storage. The receive loop appends into an in-memory structure whose own comment says it "simulates durability". The function that would write into real replica storage has no callers outside its tests.
  • A replica is fully writable and nothing consults replication state. There is no read-only gate and no role check, so pointing a client at a "replica" gives you an empty, writable database while the startup banner says it is replicating.
  • Automatic failover is not wired. check_health and promote have no production callers.
  • Synchronous mode does not wait for anything. The durability it advertises is not enforced.
  • Multi-Raft does not gate writes. Joining N nodes yields N independent writable databases running background elections, not one cluster.

Single-node Nucleus is unaffected — MVCC, WAL durability and crash recovery are real and tested. If you need high availability today, replicate at the storage or infrastructure layer, not with this.

This notice will be removed when there is a probe_replication harness proving a replica converges and survives a crash. Until it exists, treat the tables below as a roadmap.

Nucleus defines three cluster modes. Only Standalone is implemented.

Cluster Modes

| Mode | Nodes | Consensus | Status | |------|-------|-----------|--------| | Standalone | 1 | None | Implemented. The supported way to run Nucleus. | | PrimaryReplica | 2 | WAL streaming | Design only. Streams, never applies. Not HA. | | MultiRaft | 3+ | Raft per shard | Design only. Elections run; writes are not gated. |

Primary-Replica

Two-node setup where the primary streams WAL records to a replica.

Replication Modes

| Mode | Intended behavior | Actual behavior today | |------|-------------------|-----------------------| | Synchronous | Primary waits for replica ACK before committing | Does not wait. The advertised durability is not enforced. | | Asynchronous | Primary streams WAL in background | Streams, but the replica never applies what it receives. |

WAL Streaming

The primary's Write-Ahead Log is streamed to the replica in batches (default 64 records per batch):

Primary                          Replica
   │                                │
   │── WAL Batch (records 1-64) ──►│
   │                                │── Apply records
   │◄── ACK (confirmed_lsn=64) ────│
   │                                │
   │── WAL Batch (records 65-128) ─►│
   │                                │

WAL Payload Types

| Type | Content | |------|---------| | PageWrite | Page ID + data bytes | | Commit | Transaction ID | | Abort | Transaction ID | | Checkpoint | Marks a consistent point |

Monitoring

SELECT * FROM nucleus_replication_status;

Returns:

| Field | Description | |-------|-------------| | node_id | This node's ID | | role | primary or replica | | wal_lsn | Latest WAL sequence number | | applied_lsn | Last applied sequence number | | replication_lag | Records behind primary | | peer_connected | Whether peer is reachable |

Automatic Failover

Design only. check_health and promote have no production callers.

The FailoverManager monitors heartbeats and promotes the replica if the primary is unreachable.

Failover Timeline

  1. Primary stops responding — Heartbeat timeout (default 5 seconds)
  2. PrimaryDown detected — Failover manager triggers promotion
  3. Replica promoted — Becomes new primary, starts accepting writes
  4. Old primary rejoins — Demoted to replica, resyncs from new primary

Failover Events

| Event | Description | |-------|-------------| | PrimaryDown | Heartbeat timeout exceeded | | ReplicaPromoted | Replica took over as primary | | OldPrimaryRejoined | Previous primary reconnected | | ReplicationResumed | Streaming resumed after failover |

Multi-Raft (3+ Nodes)

Design only. The DML gate never opens, so joined nodes accept writes independently.

For horizontal scaling, Nucleus uses one Raft consensus group per shard. Each shard independently elects a leader and replicates its log.

Architecture

Node 1                Node 2                Node 3
┌──────────┐          ┌──────────┐          ┌──────────┐
│ Shard A  │◄────────►│ Shard A  │◄────────►│ Shard A  │
│ (Leader) │          │(Follower)│          │(Follower)│
├──────────┤          ├──────────┤          ├──────────┤
│ Shard B  │          │ Shard B  │          │ Shard B  │
│(Follower)│◄────────►│ (Leader) │◄────────►│(Follower)│
├──────────┤          ├──────────┤          ├──────────┤
│ Shard C  │          │ Shard C  │          │ Shard C  │
│(Follower)│◄────────►│(Follower)│◄────────►│ (Leader) │
└──────────┘          └──────────┘          └──────────┘

Raft Consensus

Each Raft group follows the standard protocol:

  1. Leader election — Candidates increment term, request votes from majority
  2. Log replication — Leader sends AppendEntries RPCs to followers
  3. Commitment — Entry committed when majority acknowledges
  4. Snapshot transfer — Slow followers receive full state snapshot

Raft Messages

| Message | Purpose | |---------|---------| | RequestVote | Candidate requests vote with term + log position | | VoteResponse | Follower grants or denies vote | | AppendEntries | Leader sends log entries + commit index | | Heartbeat | Leader maintains authority (empty AppendEntries) | | InstallSnapshot | Full state transfer to lagging follower |

Operations

The Raft log carries these operation types:

Put { key, value }         — Write a key-value pair
Delete { key }             — Remove a key
Sql(String)                — Execute SQL statement
TxnPrepare { txn_id }     — Phase 1 of 2PC
TxnCommit { txn_id }      — Phase 2 commit
TxnAbort { txn_id }       — Phase 2 abort
Noop                       — Leader confirmation

Distributed Transactions

Design only. Depends on the Raft mode above.

Cross-shard transactions use two-phase commit (2PC) coordinated through Raft:

Transaction Phases

Coordinator              Shard A Leader          Shard B Leader
     │                        │                        │
     │── TxnPrepare ────────►│                        │
     │── TxnPrepare ─────────────────────────────────►│
     │                        │                        │
     │◄── Prepared ──────────│                        │
     │◄── Prepared ──────────────────────────────────│
     │                        │                        │
     │── TxnCommit ─────────►│                        │
     │── TxnCommit ──────────────────────────────────►│
     │                        │                        │

Transaction Lifecycle

| Phase | State | Description | |-------|-------|-------------| | 1 | Active | Transaction executing, writes buffered | | 2 | Preparing | Prepare sent to all participating shards | | 3 | Committing | All shards prepared, commit in progress | | 4 | Committed | All shards committed | | — | Aborting | Any shard failed to prepare | | — | Aborted | All shards rolled back |

Configuration

[cluster]
mode = "multi_raft"           # standalone | primary_replica | multi_raft
node_id = 1
peers = ["node2:5433", "node3:5434"]

[cluster.replication]
mode = "synchronous"          # synchronous | asynchronous
batch_size = 64               # WAL records per batch

[cluster.failover]
heartbeat_timeout_ms = 5000   # Promote replica after this timeout

[cluster.tls]
cert = "/path/to/node.crt"
key = "/path/to/node.key"
ca = "/path/to/ca.crt"