Module: ETCD

• 1: Configuration
• 2: Parameters
• 3: Administration
• 4: Playbook
• 5: Monitoring
• 6: Metrics
• 7: FAQ

flowchart LR
    subgraph PGSQL [PGSQL]
        patroni[Patroni]
        vip[VIP Manager]
    end

    subgraph ETCD [ETCD]
        etcd[DCS Service]
    end

    subgraph NODE [NODE]
        node[Software Repo]
    end

    PGSQL -->|depends| ETCD -->|depends| NODE

    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style ETCD fill:#5B9CD5,stroke:#4178a8,color:#fff
    style NODE fill:#FCDB72,stroke:#d4b85e,color:#333

    style patroni fill:#2d4a66,stroke:#1e3347,color:#fff
    style vip fill:#2d4a66,stroke:#1e3347,color:#fff
    style etcd fill:#4178a8,stroke:#2d5a7a,color:#fff
    style node fill:#d4b85e,stroke:#b89a4a,color:#333

1 - Configuration

Before deployment, define the etcd cluster in the configuration inventory. Typical choices are:

• One Node: No HA, suitable for dev, test, demo, or standalone deployments using external S3 backup for PITR
• Three Nodes: Basic HA, tolerates 1 node failure, suitable for small-medium prod
• Five Nodes: Better HA, tolerates 2 node failures, suitable for large prod

An etcd cluster can have an even number of members. It has the same failure tolerance as a cluster with one fewer member. The extra member increases deployment and quorum costs. Thus, production clusters usually have one, three, or five members. Clusters with more than five members are uncommon.

One Node

To define a single etcd instance in Pigsty, use one line of configuration:

etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

All single-node configuration templates include this line. The configuration process replaces placeholder IP 10.10.10.10 with the current admin node IP.

Only etcd_seq and etcd_cluster are required. Together, these parameters uniquely identify each etcd instance.

Three Nodes

The most common configuration has 3 etcd nodes. It tolerates 1 node failure and is suitable for small and medium production environments.

Example: Pigsty’s 3-node templates trio and safe use 3-node etcd:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq (instance number) required
    10.10.10.11: { etcd_seq: 2 }  # positive integers, sequential from 0 or 1
    10.10.10.12: { etcd_seq: 3 }  # immutable for life, never recycled
  vars: # cluster-level params
    etcd_cluster: etcd    # default cluster name: 'etcd', don't change unless deploying multiple etcd clusters
    etcd_safeguard: false # enable safeguard? Enable after prod init to prevent accidental deletion
    etcd_clean: true      # force remove existing during init? Enable for testing for true idempotency

Five Nodes

A 5-node cluster tolerates 2 node failures. It is suitable for large production environments.

Example: Pigsty’s production simulation template ha/simu uses a 5-node etcd cluster:

etcd:
  hosts:
    10.10.10.21 : { etcd_seq: 1 }
    10.10.10.22 : { etcd_seq: 2 }
    10.10.10.23 : { etcd_seq: 3 }
    10.10.10.24 : { etcd_seq: 4 }
    10.10.10.25 : { etcd_seq: 5 }
  vars: { etcd_cluster: etcd    }

Services Using etcd

Services using etcd in Pigsty:

After a permanent etcd membership change, reload the related service configurations. This updates the endpoints that services use to access etcd.

Update Patroni’s etcd endpoint ref:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager’s etcd endpoint ref (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

RBAC Authentication Config

Since v4.0, Pigsty enables etcd RBAC authentication by default. Related parameters:

Production recommendations:

all:
  vars:
    etcd_root_password: 'YourSecureEtcdPassword'  # change default

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true    # enable safeguard for production

Filesystem Layout

The module creates these directories and files on target hosts:

2 - Parameters

The ETCD module has 13 parameters, divided into two sections:

• ETCD: 10 parameters for etcd cluster deployment and configuration
• ETCD_REMOVE: 3 parameters for controlling etcd cluster removal

Parameter Overview

Use the ETCD parameter group to deploy and configure etcd clusters. It controls instance identification, cluster name, data directory, ports, and authentication password.

The ETCD_REMOVE parameter group controls etcd cluster removal behavior, including safeguard protection, data cleanup, and package uninstallation.

ETCD

This section contains parameters for the etcd role, which are used by the etcd.yml playbook.

Parameters are defined in roles/etcd/defaults/main.yml

#etcd_seq: 1                      # etcd instance identifier, explicitly required
etcd_cluster: etcd                # etcd cluster & group name, etcd by default
etcd_learner: false               # run etcd instance as learner? default is false
etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
etcd_port: 2379                   # etcd client port, 2379 by default
etcd_peer_port: 2380              # etcd peer port, 2380 by default
etcd_init: new                    # etcd initial cluster state, new or existing
etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default
etcd_root_password: Etcd.Root     # etcd root user password for RBAC authentication (please change!)

etcd_seq

Parameter: etcd_seq, Type: int, Level: I

etcd instance identifier. This parameter is required. You must assign a unique identifier to each etcd instance.

Here is an example of a 3-node etcd cluster with identifiers 1 through 3:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # use odd numbers
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging

etcd_cluster

Parameter: etcd_cluster, Type: string, Level: C

etcd cluster & group name, default value is the hard-coded etcd.

You can modify this parameter when you want to deploy an additional etcd cluster for backup purposes.

etcd_learner

Parameter: etcd_learner, Type: bool, Level: I/A

Specifies whether to initialize the etcd instance as a learner. The default is false.

If this parameter is true, Pigsty initializes the instance as a learner. A learner cannot vote in etcd cluster elections.

Use Cases:

• Cluster Expansion: Add members as learners to preserve the current quorum requirements until data synchronization completes.
• Safe Migration: During a rolling upgrade or migration, add the member as a learner. Confirm data synchronization before you promote it.

Workflow:

1. Set etcd_learner: true to initialize the new member as a learner
2. Wait for data synchronization to complete (check with etcdctl endpoint status)
3. Use etcdctl member promote <member_id> to promote it to a full member

etcd_data

Parameter: etcd_data, Type: path, Level: C

etcd data directory, default is /data/etcd.

etcd_port

Parameter: etcd_port, Type: port, Level: C

etcd client port, default is 2379.

etcd_peer_port

Parameter: etcd_peer_port, Type: port, Level: C

etcd peer port, default is 2380.

etcd_init

Parameter: etcd_init, Type: enum, Level: C

Initial etcd cluster state. Set it to new or existing. The default is new.

Option Values:

Important Notes:

Usage Examples:

# Create new cluster (default behavior)
./etcd.yml

# Add new member to existing cluster
./etcd.yml -l <new_ip> -e etcd_init=existing

# Or use the convenience script (automatically sets etcd_init=existing)
bin/etcd-add <new_ip>

etcd_election_timeout

Parameter: etcd_election_timeout, Type: int, Level: C

etcd election timeout, default is 1000 (milliseconds), i.e., 1 second.

etcd_heartbeat_interval

Parameter: etcd_heartbeat_interval, Type: int, Level: C

etcd heartbeat interval, default is 100 (milliseconds).

etcd_root_password

Parameter: etcd_root_password, Type: password, Level: G

etcd root user password for RBAC authentication, default value is Etcd.Root.

Since v4.0, Pigsty has enabled etcd RBAC (Role-Based Access Control) authentication by default. During cluster initialization, the etcd_auth task automatically creates the root user and enables authentication.

Password Storage Location:

• Password is stored in /etc/etcd/etcd.pass file
• File permissions are 0640 (owned by root, readable by etcd group)
• The etcdctl environment script /etc/profile.d/etcdctl.sh automatically reads this file

Integration with Other Components:

• Patroni uses the pg_etcd_password parameter to configure the password for connecting to etcd
• If pg_etcd_password is empty, Patroni will use the cluster name as password (not recommended)
• VIP-Manager also requires the same authentication credentials to connect to etcd

Security Recommendations:

etcd_root_password: 'YourSecurePassword'

ETCD_REMOVE

This section contains parameters for the etcd_remove role, which are action flags used by the etcd-rm.yml playbook.

Parameters are defined in roles/etcd_remove/defaults/main.yml

etcd_safeguard: false             # unconditionally refuse removal when true
etcd_rm_data: true                # remove etcd data and config files during removal?
etcd_rm_pkg: false                # uninstall etcd packages during removal?

etcd_safeguard

Parameter: etcd_safeguard, Type: bool, Level: G/C/A

Removal safeguard. The default is false. If set to true, etcd-rm.yml aborts before deregistration, cluster departure, service shutdown, or data deletion. This static boolean switch does not check whether the instance is running. Override it explicitly with -e etcd_safeguard=false.

Recommended Settings:

In emergencies, you can override the configuration with command-line parameters:

./etcd-rm.yml -l etcd -e etcd_safeguard=false # Override the guard and remove the target Etcd cluster

etcd_rm_data

Parameter: etcd_rm_data, Type: bool, Level: G/C/A

Specifies whether to delete etcd data and configuration files during removal. The default is true.

If enabled, etcd-rm.yml deletes these files and directories when it removes a cluster or member:

• /etc/etcd/ - Configuration directory (including certificates and password files)
• /var/lib/etcd/ - Alternate data directory
• {{ etcd_data }} - Primary data directory (default /data/etcd)
• /etc/systemd/system/etcd.service - Systemd service unit file
• /etc/profile.d/etcdctl.sh - Client environment script
• /etc/vector/etcd.yaml - Vector log collection config

Use Cases:

# Stop service only, preserve data
./etcd-rm.yml -l etcd -e etcd_rm_data=false

etcd_rm_pkg

Parameter: etcd_rm_pkg, Type: bool, Level: G/C/A

Specifies whether to uninstall etcd packages during removal. The default is false.

If enabled, etcd-rm.yml uninstalls etcd packages when it removes a cluster or member.

Use Cases:

# Uninstall packages during removal
./etcd-rm.yml -l etcd -e etcd_rm_pkg=true

3 - Administration

Use these standard procedures to administer etcd:

• Create Cluster: Initialize an etcd cluster
• Destroy Cluster: Destroy an etcd cluster
• CLI Environment: Configure etcd client to access server cluster
• RBAC Authentication: Use etcd RBAC auth
• Reload Config: Update etcd server member list for clients
• Add Member: Add new member to existing etcd cluster
• Remove Member: Remove member from etcd cluster
• Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd }

Run etcd.yml playbook:

./etcd.yml  # initialize etcd cluster

For production etcd clusters, enable the safeguard etcd_safeguard to prevent accidental deletion.

Destroy Cluster

Use the dedicated etcd-rm.yml playbook to destroy an Etcd cluster. The default etcd_rm_data: true deletes local data and configuration. First, confirm that no PostgreSQL cluster still uses it as DCS. Verify a recent backup and check the exact target name.

./etcd-rm.yml -l etcd                            # Destroy the entire Etcd cluster after confirmation
./etcd-rm.yml -l etcd -e etcd_safeguard=false   # Override only when the inventory enables the safeguard

Or use utility script:

bin/etcd-rm                           # remove entire etcd cluster

The removal playbook respects etcd_safeguard. If this parameter is true, the playbook aborts before any cluster-leave, deregistration, service-stop, or file-deletion operation. The default is false. The absence of an explicit override does not confirm removal.

CLI Environment

Pigsty uses the etcd v3 API by default. Etcd v3.6+ removed the v2 API. Pigsty automatically configures /etc/profile.d/etcdctl.sh on etcd nodes. The shell loads this environment script at login.

Example client env config:

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

Since v4.0, Pigsty enables RBAC authentication for etcd by default. User authentication is still required:

export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

After you configure the client environment, run etcd CRUD operations:

e put a 10 ; e get a; e del a   # basic KV ops
e member list                    # list cluster members
e endpoint health                # check endpoint health
e endpoint status                # view endpoint status

RBAC Authentication

Since v4.0, Pigsty enables etcd RBAC authentication by default. During cluster initialization, the etcd_auth task creates the root user and enables authentication.

Set the root user password with etcd_root_password. The default is Etcd.Root. Pigsty stores it in /etc/etcd/etcd.pass with 0640 permissions. Root owns the file, and the etcd group can read it.

We strongly recommend that you change the default password in production:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_root_password: 'YourSecurePassword'  # change default

Client auth methods:

# Method 1: env vars (recommended, auto-configured in /etc/profile.d/etcdctl.sh)
export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

# Method 2: command line
etcdctl --user root:YourSecurePassword member list

Patroni and etcd auth:

Patroni uses pg_etcd_password as the etcd connection password. If it is empty, Patroni uses the cluster name as the password. We do not recommend this default. In production, set a separate etcd password for each PostgreSQL cluster.

Reload Config

After you add or remove etcd members, update the service endpoint references. Update these etcd references in Pigsty:

Refresh etcd member config:

./etcd.yml -t etcd_conf                           # refresh /etc/etcd/etcd.conf
ansible etcd -f 1 -b -a 'systemctl restart etcd'  # optional: restart etcd instances

Refresh etcdctl client env:

./etcd.yml -t etcd_config                         # refresh /etc/profile.d/etcdctl.sh

Update Patroni DCS endpoint config:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager endpoint config (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

Add Member

ETCD Reference: Add a member

Recommended: Utility Script

Use bin/etcd-add script to add new members to existing etcd cluster:

# First add new member definition to config inventory, then:
bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

The script attempts these operations in order:

• Validates the IP address
• Sets etcd_init=existing automatically and runs the etcd.yml playbook
• Provides safety warnings and countdown
• Shows configuration refresh commands after completion

Manual: Step-by-Step

Add new member to existing etcd cluster:

1. Update config inventory: Add new instance to etcd group
2. Notify cluster: Run etcdctl member add (optional, playbook auto-does this)
3. Initialize new member: Run playbook with etcd_init=existing parameter
4. Promote member: Promote learner to full member (optional, required when using etcd_learner=true)
5. Reload config: Update etcd endpoint references for all clients

# After config inventory update, initialize new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing

# If using learner mode, manually promote
etcdctl member promote <new_ins_server_id>

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 } # <--- only existing instance in cluster
    10.10.10.11: { etcd_seq: 2 } # <--- add this new member to inventory
  vars: { etcd_cluster: etcd }

$ bin/etcd-add 10.10.10.11

$ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380
Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f

ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true       # unstarted new member here
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing    # must add existing parameter
...
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ etcdctl member promote 33631ba6ced84cf8   # promote learner to follower
Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f

$ em list                # check again, new member promoted to full member
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

Remove Member

Recommended: Utility Script

Use bin/etcd-rm script to remove members from etcd cluster:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

The script automatically performs these operations:

• Gracefully removes members from cluster
• Stops and disables etcd service
• Cleans up data and config files
• Deregisters from monitoring system

The underlying removal role tolerates some leave and cleanup errors. After the script finishes, inspect etcdctl member list, endpoint health, and remaining quorum. Also inspect the actual service and data-directory state on the target.

Manual: Step-by-Step

Remove member instance from etcd cluster:

1. Keep the member in the inventory: The removal playbook needs its etcd_seq, cluster members, and connection endpoints
2. Clean up the instance: Run etcd-rm.yml against the target. It attempts member remove and stops the service. It then cleans up according to the removal flags.
3. Update the inventory: Comment out or delete the member only after the playbook succeeds
4. Reload references: Follow Reload Config for the remaining etcd members and the Patroni/VIP-Manager endpoints

# <ip> must still be in the etcd inventory group
./etcd-rm.yml -l <ip>                  # Leave the cluster and clean up automatically
# After success, remove the member from pigsty.yml and refresh the remaining members and clients

Do not delete the target from the inventory before you run the removal playbook. Without the target, the hosts: etcd scope in etcd-rm.yml cannot select it. The playbook also cannot get the instance identity or cluster endpoints from the inventory.

You do not need to repeat etcdctl member remove before or after the playbook.

$ bin/etcd-rm 10.10.10.12

$ ./etcd-rm.yml -l 10.10.10.12

$ etcdctl member list
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false  # <--- remove this

$ etcdctl member remove 93fcf23b220473fb # kick from cluster
Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f

Utility Scripts

v3.6+ provides utility scripts to simplify etcd cluster scaling:

bin/etcd-add

Add new members to existing etcd cluster:

bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script features:

• Validates IP addresses in config inventory
• Auto-sets etcd_init=existing parameter
• Executes etcd.yml playbook to complete member addition
• Shows configuration refresh commands after completion

bin/etcd-rm

Remove members or entire cluster from etcd:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script features:

• Provides safety warnings and confirmation countdown
• Auto-executes etcd-rm.yml playbook
• Gracefully removes members from cluster
• Cleans up data and config files

Manage Etcd Password

The etcd_root_password parameter defines the password of the etcd root user.

To change it, use an INFRA node or ETCD node that can reach an etcd endpoint. Run this command as the admin user:

e user passwd root  # change the etcd root user password

Update every reference to the etcd root password. Include the Patroni client configuration on INFRA nodes and the etcdctl client environment:

./infra.yml -t env_patroni    # refresh the etcd root password in /infra/conf/patronictl.yml
./etcd.yml  -t etcd_conf      # refresh /etc/etcd/etcd.pass and /etc/profile.d/etcdctl.sh

4 - Playbook

The ETCD module provides two core playbooks. Use etcd.yml to install and configure etcd clusters. Use etcd-rm.yml to remove clusters or members.

etcd.yml

Playbook source: etcd.yml

This playbook installs and configures an etcd cluster on the hardcoded etcd group. Then it starts the etcd service.

The following subtasks are available in etcd.yml:

• etcd_assert : Validate etcd identity parameters (etcd_seq must be defined as a non-negative integer)
• etcd_install : Install etcd packages
• etcd_dir : Create etcd data and configuration directories
• etcd_config : Generate etcd configuration
  • etcd_conf : Generate etcd main config file /etc/etcd/etcd.conf
  • etcd_cert : Generate etcd TLS certificates (CA, server cert, private key)
• etcd_member : Add new member to existing cluster (only runs when etcd_init=existing)
• etcd_launch : Launch etcd service
• etcd_auth : Enable RBAC authentication (create root user and enable auth)
• etcd_register : Register etcd to VictoriaMetrics monitoring

etcd-rm.yml

Playbook source: etcd-rm.yml

A dedicated playbook for removing etcd clusters or individual members. The following subtasks are available in etcd-rm.yml:

• etcd_safeguard : Check safeguard and abort if enabled
• etcd_pause : Pause for 3 seconds, allowing user to abort with Ctrl-C
• etcd_deregister : Remove etcd registration from VictoriaMetrics monitoring targets
• etcd_leave : Try graceful leaving etcd cluster before purge
• etcd_svc : Stop and disable etcd service with systemd
• etcd_data : Remove etcd data (disable with etcd_rm_data=false)
• etcd_pkg : Uninstall etcd packages (enable with etcd_rm_pkg=true)

The removal playbook uses the etcd_remove role with the following configurable parameters:

• etcd_safeguard: Prevents accidental removal when set to true
• etcd_rm_data: Controls whether ETCD data is deleted (default: true)
• etcd_rm_pkg: Controls whether ETCD packages are uninstalled (default: false)

Demo

Cheatsheet

Etcd Installation & Configuration:

./etcd.yml                                      # Initialize etcd cluster
./etcd.yml -t etcd_launch                       # Restart entire etcd cluster
./etcd.yml -t etcd_conf                         # Refresh /etc/etcd/etcd.conf with latest state
./etcd.yml -t etcd_cert                         # Regenerate etcd TLS certificates
./etcd.yml -l 10.10.10.12 -e etcd_init=existing # Scale out: add new member to existing cluster

Etcd Removal & Cleanup:

./etcd-rm.yml -l 10.10.10.12                       # Leave, deregister, stop, and delete local data by default
./etcd-rm.yml -l 10.10.10.12 -e etcd_rm_data=false # Leave, deregister, and stop, preserving local data/config
./etcd-rm.yml -l 10.10.10.12 -e etcd_rm_pkg=true   # Also uninstall Etcd packages
./etcd-rm.yml -l etcd                              # Destroy the entire cluster and its local data

Convenience Scripts:

bin/etcd-add <ip>                               # Add new member to existing cluster (recommended)
bin/etcd-rm <ip>                                # Remove specific member from cluster (recommended)
bin/etcd-rm                                     # Remove entire etcd cluster

Safeguard

The ETCD module provides a safeguard against accidental deletion. Control it with etcd_safeguard. The default is false, which disables the safeguard.

After you initialize a production etcd cluster, we recommend that you enable the safeguard. This helps prevent accidental deletion of existing instances:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true  # Enable safeguard protection

If etcd_safeguard is true, etcd-rm.yml aborts before any deregistration, cluster-leave, service-stop, or deletion action. This boolean guard does not check whether the instance is running. Override it with a command-line parameter:

./etcd-rm.yml -l <exact-target> -e etcd_safeguard=false  # Override only after verifying target and backups

After a real run, inspect etcdctl member list, endpoint health, and remaining quorum, regardless of the safeguard value. A successful task status does not verify the runtime state.

5 - Monitoring

Dashboards

The ETCD module provides one monitoring dashboard: Etcd Overview.

ETCD Overview Dashboard

ETCD Overview: Overview of ETCD cluster

The dashboard shows key ETCD status information. The ETCD Aliveness panel shows the overall cluster service status.

Red bands indicate instance downtime. A blue-gray area below the bands indicates that the cluster is unavailable.

Alert Rules

Pigsty provides five preset alert rules for etcd, defined in files/victoria/rules/etcd.yml:

• EtcdServerDown: etcd node down, CRIT alert
• EtcdNoLeader: etcd cluster no leader, CRIT alert
• EtcdQuotaFull: etcd quota > 90%, WARN alert
• EtcdNetworkPeerRTSlow: etcd network latency slow, INFO alert
• EtcdWalFsyncSlow: etcd disk fsync slow, INFO alert

The following excerpt matches the current rule source. Pigsty currently includes only the etcd-overview dashboard. Comments for the two latency alerts refer to /ui/d/etcd-instance, but these targets do not exist. Use /ui/d/etcd-overview to inspect cluster state. This known error in the source comments does not affect the alert expressions.

#==============================================================#
#                         Aliveness                            #
#==============================================================#
# etcd server instance down
- alert: EtcdServerDown
  expr: etcd_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdServerDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      etcd_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Error                                #
#==============================================================#
# Etcd no Leader triggers P0 alert immediately
# if dcs_failsafe mode not enabled, may cause global outage
- alert: EtcdNoLeader
  expr: min(etcd_server_has_leader) by (cls) < 1
  for: 15s
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdNoLeader: {{ $labels.cls }} {{ $value }}"
    description: |
      etcd_server_has_leader[cls={{ $labels.cls }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

#==============================================================#
#                        Saturation                            #
#==============================================================#
- alert: EtcdQuotaFull
  expr: etcd:cls:quota_usage > 0.90
  for: 1m
  labels: { level: 1, severity: WARN, category: etcd }
  annotations:
    summary: "WARN EtcdQuotaFull: {{ $labels.cls }}"
    description: |
      etcd:cls:quota_usage[cls={{ $labels.cls }}] = {{ $value | printf "%.3f" }} > 90%
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Latency                              #
#==============================================================#
# etcd network peer rt p95 > 200ms for 1m
- alert: EtcdNetworkPeerRTSlow
  expr: etcd:ins:network_peer_rt_p95_5m > 0.200
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdNetworkPeerRTSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:network_peer_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 200ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}
# Etcd wal fsync rt p95 > 50ms
- alert: EtcdWalFsyncSlow
  expr: etcd:ins:wal_fsync_rt_p95_5m > 0.050
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdWalFsyncSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:wal_fsync_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 50ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}

6 - Metrics

This snapshot records 177 monitoring metric families for the ETCD module. The metrics present at runtime vary with package version, enabled collectors, and target state.

7 - FAQ

What is etcd’s role in Pigsty?

etcd is a distributed, reliable key-value store for critical system data. Pigsty uses etcd as the DCS (Distributed Config Store) for Patroni. It stores PostgreSQL HA status.

Patroni uses etcd for cluster failure detection, automatic failover, primary-replica switchover, and cluster configuration management.

etcd is critical for PostgreSQL HA. Its own availability requires a reachable majority. Production deployments normally put three or five voting members in independent failure domains.

What’s the appropriate etcd cluster size?

If at least half the etcd instances are unavailable, the cluster becomes unavailable and refuses service.

A 3-node cluster tolerates 1 node failure while the other 2 nodes continue to serve requests. A 5-node cluster tolerates 2 node failures.

Learner instances do not count as voting members. A 3-node cluster with 1 learner has only 2 voting members. It cannot tolerate a voting member failure.

Use an odd number of instances in production. We recommend 3 or 5 nodes for production reliability.

Impact of etcd unavailability?

An etcd failure affects the PostgreSQL control plane. Existing PostgreSQL clusters continue to run, but Patroni management operations fail.

During the failure, PostgreSQL HA cannot perform automatic failover. You cannot use patronictl for operations such as configuration changes or manual failover.

Ansible playbooks for database creation, user creation, and HBA or Service configuration updates are not affected. You can operate PostgreSQL clusters directly during an etcd failure.

This behavior applies to Patroni >=3.0 (Pigsty >=2.0). With older Patroni versions (<3.0, Pigsty 1.x), an etcd or consul failure has a severe global effect. All PostgreSQL primaries become replicas and reject writes. Thus, an etcd failure causes a global PostgreSQL failure. The DCS Failsafe feature in Patroni 3.0 significantly improves this behavior.

What data does etcd store?

By default, Pigsty uses etcd as Patroni’s DCS. It stores coordination data, including leader leases, member state, and dynamic configuration. Pigsty does not store application data in etcd.

Patroni creates and manages this DCS data. During controlled maintenance, it can normally reconstruct coordination state from a healthy PostgreSQL cluster. However, etcd has state, and direct DCS deletion has risks.

An etcd rebuild interrupts automatic failover and patronictl management. It also clears the current DCS state. Before a rebuild, inspect the Patroni topology, current primary, remaining quorum, and recent backups. Use a documented recovery procedure during a maintenance window.

If etcd also stores K8s metadata or other custom data, back up this data yourself. Restore it after cluster recovery.

Recover from etcd failure?

By default, Pigsty uses etcd only as the Patroni DCS. A service restart preserves DCS data. A full cluster rebuild clears coordination state and leaves PostgreSQL HA without DCS quorum until recovery.

First, diagnose the failure and recover existing members. Rebuild only after you verify the topology, backups, and recovery procedure.

To restart the etcd cluster:

./etcd.yml -t etcd_launch

If a full reset/rebuild is necessary, do it during a maintenance window. Then verify etcdctl endpoint health, etcdctl member list, and patronictl list:

./etcd-rm.yml -l etcd          # Clean the cluster after verifying target and backups
./etcd.yml -l etcd             # Redeploy from inventory

Back up custom etcd data before recovery. Restore it after recovery.

Etcd maintenance considerations?

Do not let etcd reach its storage limit.

Pigsty enables etcd auto-compaction by default. The current backend quota is 8 GiB. This is usually sufficient, but you should monitor actual usage.

In the etcd data model, each write creates a new version. Frequent writes increase the database size, even if there are only a few keys. When etcd reaches its capacity limit, it rejects writes and PostgreSQL HA fails.

The default Pigsty etcd configuration includes these optimizations:

auto-compaction-mode: periodic      # periodic auto compaction
auto-compaction-retention: "24h"    # retain 24 hours history
quota-backend-bytes: 8589934592     # 8 GiB quota

For details, see the official etcd maintenance guide.

Enable etcd auto garbage collection?

For Pigsty v2.0-v2.5, enable etcd auto-compaction in production. This prevents unavailability caused by the storage quota.

Edit the etcd configuration template, roles/etcd/templates/etcd.conf:

auto-compaction-mode: periodic
auto-compaction-retention: "24h"
quota-backend-bytes: 17179869184

Set the related PostgreSQL clusters to maintenance mode. Then redeploy etcd with ./etcd.yml.

This change increases the default quota from 2 GiB to 16 GiB. It retains writes from the last 24 hours to prevent unlimited growth.

Where is PG HA data stored in etcd?

Patroni uses the pg_namespace prefix for all metadata keys. The default prefix is /pg. The PostgreSQL cluster name follows this prefix.

For example, the pg-meta cluster stores metadata under /pg/pg-meta:

etcdctl get /pg/pg-meta --prefix

Sample data:

/pg/pg-meta/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"primary_start_timeout":10,"maximum_lag_on_failover":1048576,"maximum_lag_on_syncnode":-1,"primary_stop_timeout":30,"synchronous_mode":false,"synchronous_mode_strict":false,"failsafe_mode":true,"pg_version":16,"pg_cluster":"pg-meta","pg_shard":"pg-meta","pg_group":0,"postgresql":{"use_slots":true,"use_pg_rewind":true,"remove_data_directory_on_rewind_failure":true,"parameters":{"max_connections":100,"superuser_reserved_connections":10,"max_locks_per_transaction":200,"max_prepared_transactions":0,"track_commit_timestamp":"on","wal_level":"logical","wal_log_hints":"on","max_worker_processes":16,"max_wal_senders":50,"max_replication_slots":50,"password_encryption":"scram-sha-256","ssl":"on","ssl_cert_file":"/pg/cert/server.crt","ssl_key_file":"/pg/cert/server.key","ssl_ca_file":"/pg/cert/ca.crt","shared_buffers":"7969MB","maintenance_work_mem":"1993MB","work_mem":"79MB","max_parallel_workers":8,"max_parallel_maintenance_workers":2,"max_parallel_workers_per_gather":0,"hash_mem_multiplier":8.0,"huge_pages":"try","temp_file_limit":"7GB","vacuum_cost_delay":"20ms","vacuum_cost_limit":2000,"bgwriter_delay":"10ms","bgwriter_lru_maxpages":800,"bgwriter_lru_multiplier":5.0,"min_wal_size":"7GB","max_wal_size":"28GB","max_slot_wal_keep_size":"42GB","wal_buffers":"16MB","wal_writer_delay":"20ms","wal_writer_flush_after":"1MB","commit_delay":20,"commit_siblings":10,"checkpoint_timeout":"15min","checkpoint_completion_target":0.8,"archive_mode":"on","archive_timeout":300,"archive_command":"pgbackrest --stanza=pg-meta archive-push %p","max_standby_archive_delay":"10min","max_standby_streaming_delay":"3min","wal_receiver_status_interval":"1s","hot_standby_feedback":"on","wal_receiver_timeout":"60s","max_logical_replication_workers":8,"max_sync_workers_per_subscription":6,"random_page_cost":1.1,"effective_io_concurrency":1000,"effective_cache_size":"23907MB","default_statistics_target":200,"log_destination":"csvlog","logging_collector":"on","l...
ode=prefer"}}
/pg/pg-meta/failsafe
{"pg-meta-2":"http://10.10.10.11:8008/patroni","pg-meta-1":"http://10.10.10.10:8008/patroni"}
/pg/pg-meta/initialize
7418384210787662172
/pg/pg-meta/leader
pg-meta-1
/pg/pg-meta/members/pg-meta-1
{"conn_url":"postgres://10.10.10.10:5432/postgres","api_url":"http://10.10.10.10:8008/patroni","state":"running","role":"primary","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"timeline":1}
/pg/pg-meta/members/pg-meta-2
{"conn_url":"postgres://10.10.10.11:5432/postgres","api_url":"http://10.10.10.11:8008/patroni","state":"running","role":"replica","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"replication_state":"streaming","timeline":1}
/pg/pg-meta/status
{"optime":184549376,"slots":{"pg_meta_2":184549376,"pg_meta_1":184549376},"retain_slots":["pg_meta_1","pg_meta_2"]}

Use external existing etcd cluster?

The configuration inventory uses the hardcoded etcd group. Its members serve as DCS servers for PGSQL. You can initialize this group with etcd.yml or use an existing external cluster.

To use external etcd, define the cluster as usual. Skip etcd.yml because the cluster already exists and does not require deployment.

The external etcd cluster certificate must use the same CA as Pigsty. Otherwise, clients cannot use Pigsty’s self-signed certificates.

Add new member to existing etcd cluster?

For the full procedure, see Add member to etcd cluster.

Recommended: Utility script

# First add new member to config inventory, then:
bin/etcd-add <ip>      # add single new member
bin/etcd-add <ip1>     # add multiple new members

Manual method:

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380 # announce new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing                                 # initialize new member
etcdctl member promote <new_ins_server_id>                                       # promote to full member

We recommend that you add one member at a time.

Remove member from existing etcd cluster?

For the full procedure, see Remove member from etcd cluster.

Recommended: Utility script

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm                   # remove entire etcd cluster

Manual method:

./etcd-rm.yml -l <ins_ip>                    # Leave, stop, and delete local data by default

etcd-rm.yml includes the etcdctl member remove step. Do not repeat this step before or after the normal procedure. Run member remove manually only for troubleshooting.

Keep the target in the inventory. You can still run the removal playbook once to stop, deregister, and clean up the target locally. Then verify the remaining quorum.

Configure etcd RBAC authentication?

Since v4.0, Pigsty enables etcd RBAC authentication by default. Set the root password with etcd_root_password. The default is Etcd.Root.

Production recommendation: Change the default password.

all:
  vars:
    etcd_root_password: 'YourSecurePassword'

Client authentication:

# On etcd nodes, env vars auto-configured
source /etc/profile.d/etcdctl.sh
etcdctl member list

# Manual auth config
export ETCDCTL_USER="root:YourSecurePassword"
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

See RBAC Authentication.