anti-split-brain distributed transaction framework -- three node server processes, process-level simulation
[start] spawning the txn / user / shop node server processes ...
        user -> http://127.0.0.1:6001   shop -> http://127.0.0.1:6002   txn -> http://127.0.0.1:6003
[seed] users u1=100 / u2=500 / u_low=10; merchants m1{s1:10,s2:5} / m2{s1:3}
transaction state machine transition rules:
  INIT         -> FROZEN, MANUAL_REVIEW, ROLLBACK
  FROZEN       -> MANUAL_REVIEW, ROLLBACK, SUCCESS
  ROLLBACK     -> MANUAL_REVIEW, ROLLED_BACK
  any non-terminal    -> MANUAL_REVIEW (retries exhausted, manual review)
case 1 normal order: debit user balance + merchant stock; atomic to the caller
  initial: user u1 balance=100, merchant m1/s1 stock=10
  state transitions:
    INIT           create txn a466d2aa
    INIT           -> FROZEN         a466d2aa
    FROZEN         -> SUCCESS        a466d2aa
  result: balance=70 (expect 70), stock=9 (expect 9)
  idempotency check passed: a replay with the same key returns the existing result, no double debit.
  [PASS] case1_normal
case 2 freeze failure: insufficient user balance, freeze fails midway -> ROLLBACK -> ROLLED_BACK
  initial: user u_low balance=10, merchant m1/s1 stock=9
  state transitions:
    INIT           create txn 98fea3f9
    INIT           -> ROLLBACK       98fea3f9
    ROLLBACK       -> ROLLED_BACK    98fea3f9
  result: balance=10, stock=9 (merchant freeze released, stock unchanged)
  [PASS] case2_freeze_fail
case 3 failure during commit: user balance already debited, merchant COMMIT rejected downstream -> refund balance + release stock freeze
  state transitions:
    INIT           create txn 8fcb4f19
    INIT           -> FROZEN         8fcb4f19
    FROZEN         -> ROLLBACK       8fcb4f19
    ROLLBACK       -> ROLLED_BACK    8fcb4f19
  result: balance=500 (expect refunded to 500), stock=9 (expect unchanged 9)
  [PASS] case3_commit_fail_refund
case 4 network recovery: merchant shard isolated -> stuck in INIT -> resumed by poller/dead-letter after recovery
  submit order: merchant unreachable, txn stuck in INIT (attempt 1), next retry at 1792511456.74
  merchant shard recovered; the poller starts scanning non-terminal transactions ...
  state transitions:
    INIT           create txn 353835d0
    INIT           -> FROZEN         353835d0
    FROZEN         -> SUCCESS        353835d0
  terminal state: SUCCESS (attempts=1)
  result: balance=470 (expect 470), stock=8 (expect 8)
  [PASS] case4_network_recover
case 5 node-autonomous rollback: merchant down so the rollback cannot be pushed; the restarted node self-checks and compensates
  1) both freezes succeed -> FROZEN; merchant COMMIT hits a transient failure, txn stuck in FROZEN (attempts=1)
  2) merchant process is down; forced rollback: the debited user balance is refunded, the merchant release-freeze compensation hangs because the node is unreachable, txn stuck in ROLLBACK (attempts=2)
  3) restart the merchant node; the startup self-check plugin queries the unfinished transactions and checks the local freezes ...
  state transitions:
    INIT           create txn 4a8b618b
    INIT           -> FROZEN         4a8b618b
    FROZEN         -> ROLLBACK       4a8b618b
  terminal state: ROLLED_BACK
  result: balance=470 (expect refunded to 470), stock=8 (expect unchanged 8)
  [PASS] case5_node_selfcheck
case 6 offline reconciliation: orphan freeze auto-released + global store loss recoverable from backup
  inject orphan freeze: user/u1 holds the freeze marker orphan_txn_0001 but the global store has no such record
  global transaction state store snapshot written: <FRAMEWORK>/data/txn/backup/snapshot_1792511461479.json
  reconciliation detected the orphan freeze -> auto release + alert.
    alert: reconciliation: released orphan freeze user/orphan_txn_0001
  orphan freeze cleaned up, u1 balance unchanged (=70).
  global transaction state store main data cleared (simulating total loss) -> reconciliation reports that a backup can restore it.
  restored from snapshot: <FRAMEWORK>/data/txn/backup/snapshot_1792511461479.json, 5 transaction records restored
  [PASS] case6_reconcile_orphan
case 7 retries exhausted: merchant permanently unreachable, exponential backoff reaches the limit -> MANUAL_REVIEW with an alert
  submit order: merchant m2 permanently unreachable, txn stuck in INIT, exponential backoff retrying ...
  terminal state: MANUAL_REVIEW (attempts=3, max=3)
  alert: [13c534bcdb564faf] retries exhausted (3), entering manual review: shop_db[shard_1.json] node unreachable/isolated
  infinite retrying stopped, moved to MANUAL_REVIEW, awaiting manual review.
  [PASS] case7_retry_exhaust
run summary
  [PASS] case 1 normal order  -  INIT->FROZEN->SUCCESS, idempotent replay passed
  [PASS] case 2 freeze-failure rollback  -  merchant freeze released, stock unchanged
  [PASS] case 3 refund after mid-commit failure  -  debited balance refunded, stock unchanged
  [PASS] case 4 resume after network recovery  -  SUCCESS after poller+dead-letter reconnect
  [PASS] case 5 node-autonomous rollback  -  self-check compensation after merchant restart, txn ROLLED_BACK
  [PASS] case 6 offline reconciliation  -  orphan freeze auto-released with alert, global store restorable from backup
  [PASS] case 7 retries exhausted -> manual review  -  backoff limit -> MANUAL_REVIEW + alert
alerts raised in this run:
  [orphan_unfreeze] reconciliation: released orphan freeze user/orphan_txn_0001
  [manual_review] [13c534bcdb564faf] retries exhausted (3), entering manual review: shop_db[shard_1.json] node unreachable/isolated
the three node processes have been shut down.
overall result: all passed
