Model of Patroni Active Failure

RTO Timeline

Failure Model

Key Difference Between Active and Passive Failure:

During active failure, Patroni remains available. It detects a PostgreSQL crash and attempts a restart. If restart succeeds, service recovers locally. If recovery times out, Patroni releases the Leader Key and triggers an election.

Timing Analysis

Phase 1: Failure Detection

Each loop_wait cycle, Patroni checks PostgreSQL through pg_isready or a process check.

Timeline:
    Last check      PG crash      Next check
       |              |              |
       |←── 0~loop ──→|              |

• Best case: PG crashes right before Patroni check, detected immediately, wait 0
• Worst case: PG crashes right after check, wait for next cycle, wait loop
• Average case: loop/2

Phase 2: Restart Timeout

After detecting a crash, Patroni attempts to restart PostgreSQL. There are two possible results:

Timeline:
  Crash detected     Restart attempt     Success/Timeout
      |                  |                    |
      |←──── 0 ~ start ─────────────────────→|

Path A: Self-healing Success (Best case)

• PostgreSQL restarts and service recovers.
• No failover occurs, so RTO is very short.
• Wait: 0 relative to the failover path.

Path B: Failover Required (Average/Worst case)

• PostgreSQL does not recover before primary_start_timeout.
• Patroni releases the Leader Key.
• Wait: start.

Note: The average assumes failover is necessary. A quick local restart can substantially reduce RTO.

Phase 3: Standby Detection

Standbys check the DCS Leader Key each loop_wait cycle. After the primary releases it, they detect its absence and begin an election.

Timeline:
    Lease released    Standby wakes
       |                  |
       |←── 0~loop ──────→|

• Best case: Standby wakes right when lease is released, wait 0
• Worst case: Standby just went to sleep when lease released, wait loop
• Average case: loop/2

Phase 4: Lock & Promote

When the Leader Key is absent, standbys begin an election. The winner acquires the key and runs pg_ctl promote.

1. Query standby replication positions in parallel through REST APIs. Typical response time is 10ms; the hardcoded timeout is 2s.
2. Compare WAL positions and select the best candidate. Standbys attempt atomic CAS creation of the Leader Key.
3. Run pg_ctl promote. Its duration is usually negligible.

Election process:
  StandbyA ──→ Query replication position ──→ Compare ──→ Try lock ──→ Success
  StandbyB ──→ Query replication position ──→ Compare ──→ Try lock ──→ Fail

• Best case: Single standby or direct lock acquisition and promote, constant overhead 0.1s
• Worst case: DCS API call timeout: 2s
• Average case: 1s constant overhead

Phase 5: Health Check

HAProxy requires rise consecutive successful checks before it marks the new primary healthy.

Check timeline:
  New primary    First check    Second check   Third check (UP)
     |              |               |               |
     |←─ 0~inter ──→|←─── fast ────→|←─── fast ────→|

• Best case: New primary comes up right at check time, (rise-1) × fastinter
• Worst case: New primary comes up right after check, (rise-1) × fastinter + inter
• Average case: (rise-1) × fastinter + inter/2

RTO Formula

Add the phase durations to calculate total RTO:

Best Case (PG instant self-healing)

Average Case (Failover required)

Worst Case

Model Calculation

Apply the four RTO presets to these formulas:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'500ms'  ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'     ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1500ms' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'     ,'4s' ,3 ,3 ]  # rto < 150s

Calculation Results for Four Modes (unit: seconds, format: min / avg / max)

Comparison with Passive Failure

RTO Comparison (Average case):

Analysis: In fast and norm, active failures wait for primary_start_timeout (start), making RTO slightly longer than for passive failures. In safe and wide, start < ttl - loop, so active recovery is faster. A successful local restart can make the best-case active RTO much shorter.