Module: VAULT

• 1: Configuration
• 2: Parameters
• 3: Playbooks
• 4: Administration
• 5: Usage
• 6: Monitoring
• 7: Metrics
• 8: FAQ

flowchart LR
    subgraph CONSUMER [Consumers]
        app[Applications]
        tde["PGSQL pg_tde<br>(planned)"]
        kes["MINIO KES<br>(planned)"]
    end

    subgraph VAULT [VAULT]
        entry["vault.pigsty:8200<br>VIP + HAProxy"]
        bao["OpenBao × 3<br>Raft consensus"]
    end

    subgraph NODE [NODE]
        node["Repositories / CA / VIP"]
    end

    CONSUMER -->|HTTPS| VAULT -->|depends on| NODE
    entry --> bao

    style CONSUMER fill:#3E668F,stroke:#2d4a66,color:#fff
    style VAULT fill:#5B9CD5,stroke:#4178a8,color:#fff
    style NODE fill:#FCDB72,stroke:#d4b85e,color:#333

    style app fill:#2d4a66,stroke:#1e3347,color:#fff
    style tde fill:#2d4a66,stroke:#1e3347,color:#fff,stroke-dasharray: 5 5
    style kes fill:#2d4a66,stroke:#1e3347,color:#fff,stroke-dasharray: 5 5
    style entry fill:#4178a8,stroke:#2d5a7a,color:#fff
    style bao fill:#4178a8,stroke:#2d5a7a,color:#fff
    style node fill:#d4b85e,stroke:#b89a4a,color:#333

all:
  children:
    vault: { hosts: { 10.10.10.10: { vault_seq: 1 } }, vars: { vault_cluster: vault } }
  vars:
    node_etc_hosts: [ '10.10.10.10 vault.pigsty' ]   # All nodes access vault through this domain

./node.yml  -t node_hosts          # Write vault.pigsty into /etc/hosts on all nodes
./vault.yml -l vault               # Deploy the vault cluster (or use bin/vault-add)

sudo -i bao status                 # Healthy: Initialized=true, Sealed=false

1 - Configuration

Before deploying VAULT, define a group named vault in the inventory. There is no profile or mode parameter: the mode is derived entirely from the number of members in vault.

• Single node: no HA; suitable for development, testing, demos, and sandboxes.
• Three nodes: tolerates one member failure and provides HA access with an L2 VIP; suitable for production.
• Two nodes: functional but has no fault tolerance; treat it as a single-node setup.

VAULT uses OpenBao’s built-in Raft storage. Only one active member handles writes at a time; the remaining standby members forward requests to it.

Single Node

A single-node VAULT instance needs only one line of configuration:

vault: { hosts: { 10.10.10.10: { vault_seq: 1 } }, vars: { vault_cluster: vault } }

Besides the IP address, the only required parameter is vault_seq. It determines the instance name vault-<seq> and which member performs initial cluster initialization: the one with the lowest vault_seq.

Single-node VAULT usually shares a machine with INFRA, ETCD, and PGSQL, so keys and data are not separated. Obtaining its disk exposes both encrypted data and the key. This is suitable for development and demos, but should not supply transparent-encryption keys in production.

Three Nodes

Three members tolerate one failure. Enable NODE’s L2 VIP on the vault group in production so the service endpoint is also highly available:

vault:
  hosts:
    10.10.10.11: { vault_seq: 1 }   # Required identity parameter; instance name vault-1
    10.10.10.12: { vault_seq: 2 }   # Nonnegative integer, unique within the group
    10.10.10.13: { vault_seq: 3 }   # Keep immutable; the lowest seq performs initial initialization
  vars:
    vault_cluster: vault            # Cluster name for monitoring labels and instance prefixes
    node_cluster: vault             # Same node cluster name ensures consistent VRRP authentication
    vip_enabled: true               # Enable an L2 VIP for the vault node cluster
    vip_vrid: 129                   # VRRP router ID, 1-254; unique on the L2 network
    vip_address: 10.10.10.8         # VIP address; vault.pigsty should resolve here

all:
  vars:
    node_etc_hosts: [ '10.10.10.8 vault.pigsty' ]   # All nodes access vault through the VIP

The resulting topology is:

                  vault.pigsty -> 10.10.10.8 (VIP, node_etc_hosts)
                                   |
        +--------------------------+--------------------------+
        |    keepalived VIP + haproxy :8200 (check /v1/sys/health == 200)
   vault-1 10.10.10.11      vault-2 10.10.10.12      vault-3 10.10.10.13
   openbao :8200/:8201      openbao :8200/:8201      openbao :8200/:8201
        +------------- Raft (quorum 2, tolerates 1 failure) -----------+

When vip_enabled is true and there is more than one member, vault.yml generates /etc/haproxy/conf.d/vault.cfg on every member:

• Binds <VIP>:8200 in TCP mode with TLS passthrough. Clients and OpenBao still perform certificate validation.
• Checks GET /v1/sys/health: active returns 200, standby 429, sealed 503, and uninitialized 501. Only a member returning 200 receives traffic.
• Health checks also use HTTPS and validate member certificates with /etc/pki/ca.crt.
• Members without the VIP must also bind <VIP>:8200, requiring net.ipv4.ip_nonlocal_bind=1. NODE’s tuned profiles set this. With node_tune: none, set it yourself or the playbook stops here.

With the VIP enabled, server certificate SANs include its address. Members must share the same node_cluster; otherwise NODE derives different VRRP authentication values and VIP failover does not work.

Two Nodes

A two-member Raft cluster requires a quorum of two and tolerates no failures. Either member failing makes the whole cluster unavailable, potentially reducing availability below that of a single node.

VAULT permits two-member deployments but prints a warning. Treat this as a temporary state during scaling, not a permanent topology.

Raft supports larger clusters, but VAULT is designed and validated for one or three members.

Service Domain

All clients should use vault_domain, defaulting to vault.pigsty. The full address is:

https://vault.pigsty:8200

Consumers such as pg_tde persist the KMS endpoint in their metadata. Using a domain allows migration from one to three members, or a change of VIP, by updating /etc/hosts without changing every consumer’s configuration.

NODE’s node_etc_hosts writes the mapping to each node’s /etc/hosts, without depending on INFRA DNS. Databases can therefore locate KMS at startup even if INFRA is unavailable.

./node.yml -t node_hosts           # Refresh /etc/hosts on all nodes after changing node_etc_hosts

Pigsty CA signs the server certificates. Every managed node has /etc/pki/ca.crt, which clients can use to verify VAULT’s identity.

Placement

In production, deploy VAULT on dedicated nodes or alongside control components such as ETCD and INFRA. Do not co-locate it with PostgreSQL or Silo data nodes that use its keys. Transparent encryption depends on separating keys from data: a leaked data disk, database backup, or object store should not expose its key in the same place.

VAULT needs few resources. Low disk latency matters most because each Raft log commit must be persisted.

Key Material

All VAULT trust roots are stored on the admin node in files/pki/vault/, alongside Pigsty CA. files/pki/ is ignored by Git by default:

Each member holds a copy of unseal.key. Before distributing it, Pigsty compares checksums. If an existing member key differs from the admin node’s key, the playbook stops immediately to avoid wrapping data with the wrong key.

Filesystem Layout

VAULT creates the following files and directories on each member:

On INFRA nodes, /infra/targets/vault/<vault_cluster>-<vault_seq>.yml defines the member’s monitoring scrape target.

2 - Parameters

VAULT provides eight parameters in two groups:

• VAULT: five parameters for deployment and configuration.
• VAULT_REMOVE: three parameters controlling cluster removal.

VAULT deliberately exposes few settings. Member count determines cluster mode, HA access reuses NODE’s VIP parameters, and other ports, directories, engines, and backup policies follow fixed conventions.

Parameter Overview

The VAULT group configures instance identity, cluster name, data directory, port, and service domain.

The VAULT_REMOVE group controls removal safeguards, data cleanup, and package removal.

VAULT

This section lists parameters for the vault role, used by the vault.yml playbook.

They are defined in roles/vault/defaults/main.yml.

#vault_seq: 1                     # Required vault instance identifier
vault_cluster: vault              # Cluster name, default vault
vault_data: /data/vault           # Data directory for Raft storage and local snapshots
vault_port: 8200                  # API port; Raft cluster port is this value +1
vault_domain: vault.pigsty        # Service domain for certificate SANs and all clients

vault_seq

Name: vault_seq, type: int, level: I

Required VAULT instance number. Assign every member a nonnegative integer unique within the group.

The instance name is <vault_cluster>-<vault_seq>, for example vault-1. It is also the Raft node ID, server certificate CN, and monitoring label ins.

The member with the lowest vault_seq performs initial cluster initialization. Keep instance numbers immutable once assigned; do not reuse them for other nodes.

vault:
  hosts:
    10.10.10.11: { vault_seq: 1 }  # vault-1 performs initial initialization
    10.10.10.12: { vault_seq: 2 }  # vault-2
    10.10.10.13: { vault_seq: 3 }  # vault-3

The playbook verifies that every vault_seq is a nonnegative integer unique in the vault group, stopping immediately on failure.

vault_cluster

Name: vault_cluster, type: string, level: C

VAULT cluster name, default vault.

Used as the instance name prefix, OpenBao cluster_name, and monitoring label cls, and in generating the seal key ID.

The Ansible group name is hardcoded as vault, independently of vault_cluster. Each Pigsty deployment has one VAULT cluster; this parameter usually needs no change.

vault_data

Name: vault_data, type: path, level: C

VAULT data directory, default /data/vault.

Contains raft/ for Raft storage and backup/ for local snapshots.

Use a dedicated absolute path at least two levels deep, such as /data/vault, with no .. components. Removal deletes the entire directory; never point it at a shared parent such as /var/lib.

vault_port

Name: vault_port, type: port, level: G

VAULT API port, default 8200.

The Raft cluster port is fixed at vault_port + 1, default 8201. This follows OpenBao’s convention and has no separate parameter.

Clients, HAProxy, and monitoring all use this port. Do not change it after creating the cluster.

vault_domain

Name: vault_domain, type: string, level: G

VAULT service domain, default vault.pigsty.

Included in every member’s server certificate SAN. All clients should use https://<vault_domain>:<vault_port>.

Pigsty does not resolve this name automatically. Use node_etc_hosts to point it to the single member or the three-member cluster’s VIP; see Service Domain.

VAULT_REMOVE

This section lists parameters for the vault_remove role, used by the vault-rm.yml playbook.

They are defined in roles/vault_remove/defaults/main.yml.

vault_safeguard: true             # Removal safeguard, default true
vault_rm_data: true               # Delete vault data and configuration? Default true
vault_rm_pkg: false               # Uninstall openbao? Default false

vault_safeguard

Name: vault_safeguard, type: bool, level: G/C/A

VAULT removal safeguard, default true. When enabled, vault-rm.yml aborts before taking any action.

Like MYSQL’s mysql_safeguard, this safeguard is enabled by default. Deleting VAULT destroys all keys it stores: databases, objects, and all their backups encrypted with those keys become permanently unreadable.

To remove it, explicitly override the parameter on the command line:

./vault-rm.yml -l <exact-target> -e vault_safeguard=false   # Override only after verifying the target and snapshot

vault_rm_data

Name: vault_rm_data, type: bool, level: G/C/A

Delete VAULT data and configuration during removal? Default true.

Deletes /etc/openbao on members, including local copies of TLS certificates, the seal key, and backup token; the vault_data directory, including local snapshots; the systemd unit, client environment, log rotation configuration, and vault-backup script.

When removing the whole cluster with this enabled, the playbook first takes a final snapshot and retrieves it to files/pki/vault/backup/ on the admin node. Failure aborts removal.

Regardless of this setting, files/pki/vault/ on the admin node and /var/log/openbao/ audit logs on members are never deleted.

./vault-rm.yml -l 10.10.10.13 -e vault_safeguard=false -e vault_rm_data=false  # Stop service while retaining data

vault_rm_pkg

Name: vault_rm_pkg, type: bool, level: G/C/A

Uninstall openbao during removal? Default false.

./vault-rm.yml -l vault -e vault_safeguard=false -e vault_rm_pkg=true  # Also uninstall the package

Related Parameters

VAULT reuses existing parameters for HA access and domain resolution:

Fixed Conventions

The following conventions have no configurable parameters:

3 - Playbooks

VAULT provides two playbooks: vault.yml creates, updates, and scales clusters, while vault-rm.yml removes members or the whole cluster. The helper scripts bin/vault-add and bin/vault-rm wrap these playbooks.

vault.yml

Source: vault.yml

Installs, configures, and starts OpenBao in the hardcoded vault group. It is idempotent: repeated runs reconcile configuration changes without reinitializing the cluster.

vault.yml provides these task subsets:

• id: derive node identity and variables through node_id.
• vault_assert: validate member IDs, data paths, ports, and VIP configuration; derive instance names and members (always runs).
• vault_install: install openbao and ensure the openbao system user exists.
• vault_dir: create configuration, data, snapshot, and audit log directories.
• vault_config: generate VAULT configuration.
  • vault_cert: issue and distribute TLS certificates.
    • vault_cert_issue: issue server certificates with Pigsty CA on the admin node.
    • vault_cert_copy: copy the CA certificate, server certificate, and private key to members.
  • vault_seal: generate and distribute the static seal key; abort if an existing member key differs from the admin node’s key.
  • vault_conf: render openbao.hcl, the systemd unit, bao client environment, and log rotation configuration.
• vault_launch: start stopped members. Members already running before execution restart one at a time only if their certificates, seal key, configuration, or unit changed, waiting for each to rejoin. Newly started members skip a second restart to avoid startup races.
• vault_init: initialize once on the member with the lowest vault_seq, then wait for all members to unseal and complete Raft election (leader_address nonempty).
• vault_provision: enable secret/, transit/, and approle; issue the backup token, reissuing it automatically if an existing token is invalid, for example from an old cluster.
• vault_haproxy: generate the <VIP>:8200 HAProxy endpoint on members when VIP is enabled; otherwise remove it.
• vault_backup: install vault-backup, its daily schedule, and take the first snapshot.
• vault_register (aliases register, add_metrics): register members with VictoriaMetrics.

vault_init runs bao operator init only when all these conditions hold:

1. The current member has the lowest vault_seq in vault.
2. Every member is reachable and reports that it is uninitialized.
3. files/pki/vault/init.json does not exist on the admin node.

Using -l <ip> to add or rebuild a member therefore cannot bootstrap a second empty cluster. The new member joins through Raft retry_join and unseals with the same key. The playbook aborts if an unreachable member prevents determining initialization state. It also aborts if all members are uninitialized but init.json exists, indicating a previously initialized cluster whose data was lost. See Restore a Snapshot.

The generated root token and recovery keys are saved to files/pki/vault/init.json on the admin node. If saving fails, they remain temporarily in /root/vault-init.json on the member and the playbook stops; move them to the admin node manually.

vault-rm.yml

Source: vault-rm.yml

Removes members or the whole cluster. vault-rm.yml provides these task subsets:

• vault_safeguard: check the removal safeguard, enabled by default; abort if enabled.
• vault_consumer: before whole-cluster removal, list tde/* and kes/* consumers whose data will become unreadable.
• vault_pause: pause five seconds to allow cancellation with Ctrl-C.
• vault_snapshot: before whole-cluster removal, take a final snapshot and retrieve it to files/pki/vault/backup/ on the admin node; abort on failure without deleting data.
• vault_leave: for partial removal, run bao operator raft remove-peer on a surviving member; abort on failure without deleting data.
• vault_deregister (aliases deregister, rm_metrics): remove members from VictoriaMetrics targets.
• vault_svc: remove the backup cron job and metrics, stop and disable openbao, and remove the HAProxy endpoint.
• vault_data: delete data, configuration, local TLS certificates, seal key and backup token copies, and local snapshots; disable with vault_rm_data=false.
• vault_pkg: uninstall openbao; explicitly enable with vault_rm_pkg=true.

The playbook uses vault_remove and supports:

• vault_safeguard: block removal when true; default true.
• vault_rm_data: delete VAULT data; default true.
• vault_rm_pkg: uninstall openbao; default false.

If the run targets every member of vault, it is treated as whole-cluster removal. Otherwise, it is partial removal and the selected members must first leave Raft. Every selected member must be reachable before removal begins; an unreachable member aborts the playbook. Peer removal uses the root token in the admin node’s init.json, so members being removed must remain in the inventory’s vault group during execution.

The admin node’s files/pki/vault/ (seal key, initialization material, initial and final snapshots) and members’ /var/log/openbao/ audit logs are never deleted.

Command Reference

Installation and configuration:

./vault.yml -l vault                              # Create or update the vault cluster
./vault.yml -l vault --check                      # Preview proposed changes
./vault.yml -l vault -t vault_conf,vault_launch   # Refresh configuration; restart members sequentially as needed
./vault.yml -l vault -t vault_cert,vault_launch   # Reissue/distribute certificates; restart sequentially as needed
./vault.yml -l vault -t vault_haproxy             # Refresh the HAProxy VIP endpoint
./vault.yml -l vault -t vault_backup              # Refresh snapshot script, schedule, and backup token
./vault.yml -l vault -t vault_register            # Register monitoring targets again
./vault.yml -l 10.10.10.14                        # Add a new member to the existing cluster

Removal and cleanup:

./vault-rm.yml -l 10.10.10.13 -e vault_safeguard=false                       # Remove peer, stop service, delete local data
./vault-rm.yml -l 10.10.10.13 -e vault_safeguard=false -e vault_rm_data=false # Remove peer and stop service, retain local data
./vault-rm.yml -l vault -e vault_safeguard=false                             # Destroy the whole cluster
./vault-rm.yml -l vault -e vault_safeguard=false -e vault_rm_pkg=true        # Destroy cluster and uninstall package

Helper Scripts

bin/vault-add                  # Create/update the whole cluster; equivalent to ./vault.yml -l vault
bin/vault-add 10.10.10.14      # Add members (multiple IPs accepted)
bin/vault-rm                   # Destroy the whole cluster; type vault to confirm
bin/vault-rm 10.10.10.13       # Remove specified members (multiple IPs accepted)

Both scripts accept only full IPv4 addresses, provide a countdown, and suggest follow-up actions. For example, after adding or removing members, they prompt you to refresh the remaining members’ configuration and HAProxy endpoints:

./vault.yml -l 'vault,!10.10.10.14' -t vault_config,vault_launch,vault_haproxy,vault_backup

Safeguards

VAULT includes safeguards for deployment and removal:

Regardless of safeguard settings, recheck bao status, Raft membership, and fault tolerance after a real run. Playbook success does not replace runtime validation.

4 - Administration

Common VAULT administration procedures:

• Create a Cluster: initialize VAULT.
• Check Status: inspect cluster, member, and Raft state.
• Administrator Credentials: use the root token safely.
• Add Members: add members or scale from one to three nodes.
• Remove Members: retire existing members.
• Rebuild a Member: recover from damaged data or replace hardware.
• Update Configuration: refresh configuration and certificates.
• Backups: create snapshots manually and keep off-site copies.
• Restore a Snapshot: roll back data or rebuild after total data loss.
• Upgrade OpenBao: update the software package.
• Destroy a Cluster: remove and recreate VAULT.

See the VAULT FAQ for other questions.

Create a Cluster

Define the vault group in the inventory, and resolve its service domain with node_etc_hosts. See Configuration for details:

vault:
  hosts:
    10.10.10.11: { vault_seq: 1 }
    10.10.10.12: { vault_seq: 2 }
    10.10.10.13: { vault_seq: 3 }
  vars: { vault_cluster: vault, node_cluster: vault, vip_enabled: true, vip_vrid: 129, vip_address: 10.10.10.8 }

Provision the nodes through NODE, including the VIP, then run vault.yml:

./node.yml  -l vault               # Provision nodes and configure the L2 VIP
./node.yml  -t node_hosts          # Write vault.pigsty into /etc/hosts on all nodes
./vault.yml -l vault               # Create the cluster; bin/vault-add is also available

The playbook creates the seal key, initialization material, backup token, and initial snapshot in files/pki/vault/ on the admin node. Immediately back up the entire files/pki/ directory to a secure location.

Check Status

/etc/profile.d/bao.sh sets the bao environment: BAO_ADDR points to local https://127.0.0.1:8200, and BAO_CACERT to /etc/openbao/tls/ca.crt. Only root and the openbao group can read the TLS directory, so run bao as root on a member:

sudo -i                            # Switch to root and load /etc/profile.d/bao.sh
bao status                         # No token needed: initialization, seal, HA role, and Raft indexes

A healthy standby member produces output similar to:

Key                      Value
---                      -----
Seal Type                static
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  2.7.2
Storage Type             raft
Cluster Name             vault
HA Enabled               true
HA Mode                  standby
Active Node Address      https://10.10.10.13:8200
Raft Committed Index     1150
Raft Applied Index       1150

To run as a regular user, use the globally readable Pigsty CA: export BAO_CACERT=/etc/pki/ca.crt.

Raft membership and Autopilot health checks require an administrator token; see Administrator Credentials:

bao operator raft list-peers                 # node_id, address, role, and voter status
bao operator raft autopilot state            # Cluster health, fault tolerance, and member health

Health endpoints require no token and can be accessed by service domain from any managed node:

curl -s --cacert /etc/pki/ca.crt https://vault.pigsty:8200/v1/sys/health   # Active returns 200
curl -s --cacert /etc/pki/ca.crt https://vault.pigsty:8200/v1/sys/leader   # Current active member address

Grafana’s Vault Overview shows members, roles, seal status, Raft replication lag, and snapshot times. See Monitoring.

Administrator Credentials

The root token and recovery keys generated during initialization are stored in files/pki/vault/init.json on the admin node:

jq -r .root_token ~/pigsty/files/pki/vault/init.json   # Read the root token on the admin node

On a member, enter the root token interactively into an environment variable to keep it out of command arguments and shell history:

sudo -i
read -rs BAO_TOKEN && export BAO_TOKEN      # Paste the root token and press Enter; input is hidden
bao operator raft list-peers

The root token has full privileges; use it briefly for administrative tasks only. Give applications policies and AppRoles with the minimum required privileges; see Usage.

In auto-unseal mode, the five recovery shares (threshold three) cannot unseal the service. They authorize quorum operations such as bao operator generate-root, for example to replace a lost root token.

Add Members

Recommended: Helper Script

# First add the new member to the inventory's vault group with a unique vault_seq, then run:
bin/vault-add 10.10.10.14

The script runs ./vault.yml -l 10.10.10.14: it issues certificates, distributes the seal key, joins the existing cluster through Raft retry_join, unseals automatically, and registers monitoring. The new member does not have the lowest vault_seq, so it never initializes the cluster.

Then refresh existing members’ configuration, certificates, HAProxy endpoints, and backup scripts:

./vault.yml -l 'vault,!10.10.10.14' -t vault_config,vault_launch,vault_haproxy,vault_backup

If configuration or certificates change, vault_launch restarts existing members one at a time to apply the updates.

Scale from One to Three Nodes

When scaling to three production members, the service domain changes from pointing to one member to pointing to a VIP. Existing certificates must include the VIP:

1. Add two members and configure node_cluster, vip_enabled, vip_vrid, and vip_address on the vault group.

2. Provision nodes and the VIP with ./node.yml -l vault. Already managed nodes need only ./node.yml -l vault -t node_vip.

3. Add members: bin/vault-add 10.10.10.12 10.10.10.13.

4. Refresh the original member’s certificate, configuration, and endpoint. Certificate changes trigger its restart:

   ./vault.yml -l 10.10.10.11 -t vault_config,vault_launch,vault_haproxy,vault_backup

5. Point vault.pigsty at the VIP in node_etc_hosts, then run ./node.yml -t node_hosts to update all nodes.

6. Verify that bao operator raft list-peers shows three voters and /v1/sys/health returns 200 through the VIP.

Check whether a certificate includes the VIP with openssl x509 -in /etc/openbao/tls/server.crt -noout -ext subjectAltName as root on the member. If it does not, delete the corresponding files/pki/vault/vault-<seq>.crt on the admin node and repeat step 4.

Remove Members

Recommended: Helper Script

bin/vault-rm 10.10.10.13          # Remove 10.10.10.13 from the cluster

The script runs ./vault-rm.yml -l 10.10.10.13 -e vault_safeguard=false: it executes bao operator raft remove-peer vault-3 on a survivor, deregisters monitoring, stops services, and deletes local data and configuration.

Keep the member in the inventory’s vault group during execution. The playbook needs the inventory to locate a surviving member for peer removal. If removal fails, for example due to an invalid administrator token, it aborts without deleting data.

After success, remove the member from the inventory and refresh the survivors’ configuration and HAProxy endpoints:

vi pigsty.yml                     # Remove 10.10.10.13 from vault
./vault.yml -l vault -t vault_config,vault_launch,vault_haproxy,vault_backup

Manual Steps

To preserve local data and snapshots on the removed member:

./vault-rm.yml -l 10.10.10.13 -e vault_safeguard=false -e vault_rm_data=false   # Remove peer and stop service; retain data

Rebuild a Member

For damaged data or hardware replacement at the same address, remove the member, then add it again. Keep it in the inventory throughout:

bin/vault-rm  10.10.10.13          # Remove from Raft, stop service, clean local data
bin/vault-add 10.10.10.13          # Rejoin with the same vault_seq

This applies only to multi-member clusters. Running bin/vault-rm against the sole member destroys the entire cluster; use Restore a Snapshot instead.

The rebuilt member synchronizes all data from the active member. Even if it has the lowest vault_seq, the playbook does not initialize again because the other members are already initialized.

If the host is completely unavailable and cannot run vault-rm.yml, remove it manually from a survivor using an administrator token, then add its replacement:

bao operator raft remove-peer vault-3      # Run as root on a survivor with BAO_TOKEN set

Update Configuration

The playbook generates VAULT configuration. Edit the inventory and rerun it instead of manually changing /etc/openbao/openbao.hcl.

./vault.yml -l vault -t vault_conf,vault_launch     # Refresh config; restart changed members sequentially

vault_launch first starts selected stopped services, then restarts previously running members whose certificates, seal key, configuration, or systemd unit changed. Newly started members already loaded the latest configuration and skip a second restart to avoid cold-start and initialization races. With a leader present, it waits for each member to rejoin before proceeding; without one, startup continues to allow quorum recovery. Check cluster health before updating. If an earlier run stopped after writing configuration, manually restart members that have not loaded it, one at a time.

Renew certificates: server certificates default to 20 years. To reissue them, for example after changing the VIP or domain, delete the corresponding certificates on the admin node and run:

rm -f files/pki/vault/vault-*.crt                   # Remove old certificates; retain private keys
./vault.yml -l vault -t vault_cert,vault_launch     # Reissue, distribute, and restart sequentially

Refresh HAProxy after enabling/disabling the VIP, changing membership, or running node.yml with haproxy_clean=true:

./vault.yml -l vault -t vault_haproxy

Backups

Daily snapshots: each member runs vault-backup at 03:00, staggered by vault_seq, retrieving a Raft snapshot from the active member into /data/vault/backup/ and retaining the latest seven. Thus every member of a three-node cluster holds complete snapshot copies.

The backup token can only read sys/storage/raft/snapshot. The script uses flock on ${BACKUP_DIR}/.backup.lock to serialize concurrent local jobs, such as scheduled and manual backups. Successful jobs write completion time and size into node_exporter’s textfile directory. No successful snapshot for 26 hours triggers VaultBackupStale.

vault-backup                       # Run as root; the last output line is the snapshot path
vault-backup 14                    # Take a snapshot and retain the latest 14
journalctl -t vault-backup         # Inspect scheduled backup logs

Initial and final snapshots: immediately after initialization, the playbook takes a snapshot and retrieves it to files/pki/vault/backup/ on the admin node. It also retrieves a final snapshot before destroying the cluster. Operators may copy other snapshots there; these files are not cleaned automatically.

If initial deployment stops before retrieval, a rerun detects the missing admin-node snapshot, takes one, and retrieves it.

Off-site copies: scheduled snapshots stay on members; operators are responsible for off-site replication. Snapshots are encrypted by OpenBao’s barrier key and cannot be opened without unseal.key, so they may be copied to other storage. Store files/pki/vault/ separately from snapshots. For example, retrieve a member’s latest snapshot from the admin node:

(umask 077; ssh 10.10.10.11 "sudo bash -c 'cat \$(ls -t /data/vault/backup/*.snap | head -1)'" \
  > files/pki/vault/backup/vault-$(date +%Y%m%d).snap)

Restore a Snapshot

Recovery requires a snapshot, its matching files/pki/vault/unseal.key, and the original files/pki/vault/init.json. The original root token and recovery keys become valid again after restoration.

Roll Back a Working Cluster

To roll a healthy cluster back to a snapshot, for example after accidental key deletion, run as root on the active member:

read -rs BAO_TOKEN && export BAO_TOKEN                          # Enter the root token
vault-backup                                                    # Snapshot the current state first
bao operator raft snapshot restore /data/vault/backup/vault-<timestamp>.snap

Raft replicates the restored state to the other members. Check bao status and application keys afterward.

Rebuild After Total Data Loss

If all members lose their data, vault.yml refuses to initialize: all members report uninitialized, but init.json on the admin node identifies a previously initialized cluster. Bootstrap a temporary empty cluster, then overwrite it with the snapshot:

1. Prepare the snapshot and verify that files/pki/vault/unseal.key on the admin node is the key used when it was created.

2. Archive the original initialization material; otherwise the playbook refuses initialization:

   mkdir -p files/pki/vault/orig
   mv files/pki/vault/init.json files/pki/vault/orig/

3. Bootstrap the temporary cluster, generating a new init.json. Its old backup.token is invalid, so the playbook reissues it and backs up the old file as backup.token.*~:

   ./vault.yml -l vault

4. Copy the snapshot to the temporary cluster’s active member (HA Mode: active in bao status), for example /tmp/restore.snap. Use the temporary cluster’s root token to force restoration; -force permits a snapshot from another cluster:

   sudo -i
   read -rs BAO_TOKEN && export BAO_TOKEN    # Enter the root token from the new init.json
   bao operator raft snapshot restore -force /tmp/restore.snap

5. Wait for members to become healthy. In OpenBao 2.7.2 testing, a standby sometimes continued reporting decryption failed: cipher: message authentication failed after a forced cross-cluster restore. Verify its unseal.key matches the admin node, then restart only that member and wait for normal bao status before continuing. Do not restart all members together:

   ansible 10.10.10.13 -b -m systemd -a 'name=openbao state=restarted'   # Substitute the affected member's address

6. After restoration, the original root token works again and temporary cluster credentials are invalid. Restore the original init.json, then run token provisioning and backup tasks. The playbook detects the invalid backup token and reissues/distributes it:

   mv -f files/pki/vault/orig/init.json files/pki/vault/
   ./vault.yml -l vault -t vault_provision,vault_backup

7. Use the original root token to check bao operator raft list-peers and application keys. Run vault-backup manually on every member to verify backups work again.

Snapshots also preserve audit device configuration. After unsealing, OpenBao writes to /var/log/openbao/audit.log. An unwritable path causes post-unseal initialization to fail; check directory permissions. Do not disable auditing to bypass this problem.

If VAULT holds other components’ keys, restore in dependency order: VAULT → KES → Silo → pgBackRest → PostgreSQL.

Upgrade OpenBao

The Pigsty Infra repository provides openbao. Upgrades are manual; refresh the local repository and take a snapshot first:

1. Run vault-backup on a member and copy the snapshot to a secure location.
2. Upgrade standby members one at a time: update the package, run systemctl restart openbao, wait for bao status to show unsealed and Autopilot to report the member healthy, then proceed.
3. Upgrade the active member last. Its restart triggers a new active election and the VIP endpoint switches accordingly.

ansible 10.10.10.12 -b -m package -a 'name=openbao state=latest'   # Upgrade one member's package
ansible 10.10.10.12 -b -m systemd -a 'name=openbao state=restarted'

Before a major upgrade, read the OpenBao release notes and upgrade guide.

Destroy a Cluster

Use vault-rm.yml or bin/vault-rm to destroy the whole cluster.

bin/vault-rm                                        # Type vault to confirm
./vault-rm.yml -l vault -e vault_safeguard=false    # Equivalent command

Whole-cluster removal lists tde/* and kes/* consumers, pauses five seconds, takes a final snapshot on a member, and retrieves it to the admin node’s files/pki/vault/backup/ before deregistering monitoring, stopping services, and deleting data. It does not run raft remove-peer or delete the admin node’s files/pki/vault/ or members’ audit logs.

If the final snapshot fails, removal aborts before deleting anything. A broken or never-initialized cluster cannot produce a snapshot, so explicitly skip this step if necessary:

./vault-rm.yml -l vault -e vault_safeguard=false --skip-tags vault_snapshot   # Remove without a final snapshot

Create a new empty cluster: after removal, archive the old initialization material before creating a fresh empty cluster instead of restoring a snapshot; otherwise vault.yml refuses initialization:

mkdir -p files/pki/vault/archive-$(date +%Y%m%d)
mv files/pki/vault/init.json files/pki/vault/archive-$(date +%Y%m%d)/
./vault.yml -l vault

The old backup.token is invalid in the new cluster. The playbook reissues it and backs up the old file as backup.token.*~.

You may retain and reuse unseal.key. Old snapshots still require that key, so do not delete it.

5 - Usage

After configuring and deploying VAULT, use this page to get started. The following engines and authentication methods are enabled by default:

OpenBao’s API is largely compatible with HashiCorp Vault, so common Vault client libraries and tools generally work directly. See the OpenBao documentation for full usage.

Endpoint

All clients should use the service domain over HTTPS and validate the server certificate with Pigsty CA:

In a three-member cluster, vault.pigsty points to the VIP and HAProxy forwards to the active member. Direct standby access also works; the standby forwards requests to the active member.

export BAO_ADDR=https://vault.pigsty:8200
export BAO_CACERT=/etc/pki/ca.crt

bao is installed with openbao on VAULT members. Other nodes can use curl with the HTTP API, or install openbao to obtain bao.

Administrator Login

Run these examples as root on a VAULT member with the root token. See Administrator Credentials:

sudo -i                                     # BAO_ADDR and BAO_CACERT are set by /etc/profile.d/bao.sh
read -rs BAO_TOKEN && export BAO_TOKEN      # Enter the root token; input is hidden
bao secrets list                            # List enabled secrets engines
bao auth list                               # List enabled authentication methods

Use the root token only for administration. Issue credentials with minimal privileges to applications.

KV Secrets

secret/ uses KV v2, which saves a new version on every write:

bao kv put secret/app/db username=dbuser_app password='S3cret!'   # Write a secret
bao kv get secret/app/db                                          # Read the latest version
bao kv get -field=password secret/app/db                          # Read one field
bao kv get -version=1 secret/app/db                               # Read a historical version
bao kv list secret/app                                            # List secrets under a path
bao kv metadata get secret/app/db                                 # Inspect version metadata
bao kv delete secret/app/db                                       # Soft-delete the latest version; recover with undelete

KV v2 policies and HTTP APIs use secret/data/<path> for secret reads/writes and secret/metadata/<path> for metadata.

Transit Encryption

transit/ provides encryption as a service. Applications send data for encryption or decryption without handling keys, suitable for encrypting sensitive database fields in the application layer.

bao write -f transit/keys/app                                    # Create the app encryption key
bao write transit/encrypt/app plaintext=$(echo -n 'hello' | base64)
# ciphertext    vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==
bao write -field=plaintext transit/decrypt/app \
  ciphertext='vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==' | base64 -d
bao write -f transit/keys/app/rotate                             # Rotate the key; old ciphertext remains readable

Base64-encode plaintext first. After rotation, new data uses the new key version, while older ciphertext remains decryptable. Use transit/rewrap/app to rewrap old ciphertext with the new version.

Issue Application Credentials

Create a policy with only the permissions each application needs, then issue machine credentials through AppRole:

# 1. Create a policy: read/write secret/app/ and use the app Transit key only
bao policy write app - <<'EOF'
path "secret/data/app/*"     { capabilities = ["create", "read", "update", "delete", "list"] }
path "secret/metadata/app/*" { capabilities = ["read", "list"] }
path "transit/encrypt/app"   { capabilities = ["update"] }
path "transit/decrypt/app"   { capabilities = ["update"] }
EOF

# 2. Create an AppRole whose login tokens use this policy
bao write auth/approle/role/app token_policies=app token_ttl=1h token_max_ttl=24h

# 3. Read role_id and secret_id and distribute them to the application
bao read -field=role_id auth/approle/role/app/role-id
bao write -f -field=secret_id auth/approle/role/app/secret-id

role_id acts as a username and secret_id as a password. Applications exchange them for a short-lived token, then use that token to access secrets:

bao write auth/approle/login role_id=<role_id> secret_id=<secret_id>

Deliver secret_id through a secure channel. Restrict credentials further with secret_id_ttl, secret_id_num_uses, and token_bound_cidrs as needed.

Through the VIP, HAProxy forwards in TCP mode, so OpenBao sees the VAULT member’s address rather than the original client. Audit logs record that member address, and token_bound_cidrs set to client networks will reject these requests.

HTTP API

Clients without bao can call the HTTP API directly. For example, log in with AppRole and read a secret:

CA=/etc/pki/ca.crt
URL=https://vault.pigsty:8200

# Log in with AppRole to obtain a client token
TOKEN=$(curl -s --cacert $CA -X POST $URL/v1/auth/approle/login \
  -d '{"role_id":"<role_id>","secret_id":"<secret_id>"}' | jq -r .auth.client_token)

# Read a KV v2 secret; note data/ in the path
curl -s --cacert $CA -H "X-Vault-Token: $TOKEN" $URL/v1/secret/data/app/db | jq .data.data

Unauthenticated endpoints support health checks and monitoring:

curl -s --cacert /etc/pki/ca.crt https://vault.pigsty:8200/v1/sys/health     # Health status
curl -s --cacert /etc/pki/ca.crt https://vault.pigsty:8200/v1/sys/leader     # Active member

Reserved Paths

These paths and names are used by Pigsty or reserved for future features; do not use them for applications:

Before whole-cluster removal, vault-rm.yml lists existing tde/* and kes/* mounts to identify consumers that will permanently lose keys.

6 - Monitoring

VAULT reuses Pigsty’s observability stack. OpenBao metrics flow over HTTPS to VictoriaMetrics; service and backup logs flow through Journald / Vector to VictoriaLogs. vmalert loads 14 recording rules and six alerts, and Grafana provides the Vault Overview dashboard.

Dashboard

VAULT provides one dashboard: Vault Overview.

Vault Overview Dashboard

Vault Overview is available at /ui/d/vault-overview. Use the Cluster variable at the top to select a cluster.

The initial Overview section shows:

• Vault Spec: members, unsealed members, quorum, and fault tolerance. Vault Members: each member’s current role (Active / Standby / Sealed / Down).
• Vault Member List: availability, seal state, role, Autopilot health, Raft applied index, replication lag, lease/token counts, and time since the last snapshot.
• Vault Health: Autopilot cluster health. Has Leader: whether exactly one member is active. Backup Age: age of the oldest of the members’ latest successful snapshots.
• Vault Aliveness: the main timeline. Its top band shows whether exactly one member is active; member bands use Active / Standby / Sealed colors, with red for unavailable members.
• Leader Elections, Raft statistics, request rate, average latency, audit failures, lease/secret counts, and active alerts.

Expand other groups for details:

Metric Collection

Monitoring registration creates a file-discovery target on INFRA nodes for each member. VictoriaMetrics’ vault scrape job consumes this directory:

# /infra/targets/vault/vault-1.yml
- labels: { ip: 10.10.10.11 , ins: vault-1 , cls: vault }
  targets: [ 10.10.10.11:8200 ]

The job accesses https://<ip>:8200/v1/sys/metrics?format=prometheus directly and validates member certificates with Pigsty CA. Metrics share the API port and require no separate exporter:

• unauthenticated_metrics_access is enabled only on the node IP listener, so scraping requires no token. The loopback listener does not allow unauthenticated metrics.
• Scraping uses HTTPS. OpenBao’s listeners provide TLS only; a separate plaintext metrics port is unnecessary.
• Sealed members remain scrapeable. vault_up indicates process/network reachability, while vault_core_unsealed indicates seal state.

After each successful snapshot, vault-backup writes two metrics to /var/lib/node_exporter/textfile/vault.prom. node_exporter collects them, and rules associate them with VAULT instances by ip.

VAULT metrics carry job="vault", cls, ins, ip, and instance. See Metrics for details.

Logs

OpenBao runs under systemd and logs to Journald. Vector collects those logs into VictoriaLogs. Scheduled vault-backup output uses logger -t vault-backup and follows the same path. Use these LogsQL queries in Grafana:

job:syslog unit:openbao                  # OpenBao service logs
job:syslog app:="vault-backup"           # Snapshot backup logs

Audit logs remain in /var/log/openbao/audit.log on members, rotated daily with seven files retained. They are not collected into VictoriaLogs. They include request paths and client information; configure centralized collection to meet your requirements.

If an audit device cannot write, for example because the disk is full, OpenBao rejects all requests. VaultAuditFailure covers this condition.

Alert Rules

VAULT provides six alerts in files/victoria/rules/vault.yml:

INFRA’s generic AgentDown alert excludes the vault job to avoid duplicating VaultDown.

The following excerpt reproduces the current rule source:

# metrics scrape failure: process down or unreachable (sealed members are still scraped)
- alert: VaultDown
  expr: vault_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: vault }
  annotations:
    summary: "CRIT VaultDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      vault_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      metrics of the member can not be scraped: check openbao service on {{ $labels.ip }}
      /ui/d/vault-overview?var-cls={{ $labels.cls }}

# member reports itself sealed: a sealed single node also triggers VaultNoLeader
- alert: VaultSealed
  expr: vault_core_unsealed{job="vault"} < 1
  for: 1m
  labels: { level: 1, severity: WARN, category: vault }
  annotations:
    summary: "WARN VaultSealed {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      vault_core_unsealed[ins={{ $labels.ins }}] = {{ $value }} < 1
      the member is sealed and can not serve keys: check `bao status` on {{ $labels.ip }}
      /ui/d/vault-overview?var-cls={{ $labels.cls }}

# not exactly one active member observed: kms outage for cache misses & restarts
- alert: VaultNoLeader
  expr: vault:cls:active != 1
  for: 30s
  labels: { level: 0, severity: CRIT, category: vault }
  annotations:
    summary: "CRIT VaultNoLeader: {{ $labels.cls }} {{ $value }}"
    description: |
      vault:cls:active[cls={{ $labels.cls }}] = {{ $value }} != 1
      /ui/d/vault-overview?from=now-5m&to=now&var-cls={{ $labels.cls }}

# audit device rejects writes: openbao rejects requests, check disk
- alert: VaultAuditFailure
  expr: vault:ins:audit_failure_5m > 0
  for: 1m
  labels: { level: 0, severity: CRIT, category: vault }
  annotations:
    summary: "CRIT VaultAuditFailure: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      vault:ins:audit_failure_5m[ins={{ $labels.ins }}] = {{ $value | printf "%.0f" }} > 0
      requests are rejected while the audit log can not be written, check disk on {{ $labels.ip }}
      /ui/d/vault-overview?var-cls={{ $labels.cls }}

# production cluster (3+ members) can not survive another member failure
- alert: VaultNoFailureTolerance
  expr: >-
    (vault:cls:failure_tolerance < 1)
    and on (job,cls) (count by (job,cls) (vault_up) >= 3)
  for: 2m
  labels: { level: 1, severity: WARN, category: vault }
  annotations:
    summary: "WARN VaultNoFailureTolerance: {{ $labels.cls }} {{ $value }}"
    description: |
      vault:cls:failure_tolerance[cls={{ $labels.cls }}] = {{ $value }} < 1
      losing one more member loses quorum
      /ui/d/vault-overview?var-cls={{ $labels.cls }}

# no successful local snapshot on the member for 26h (or never)
- alert: VaultBackupStale
  expr: time() - (vault:ins:backup_time or vault_up * 0) > 26 * 3600
  for: 5m
  labels: { level: 1, severity: WARN, category: vault }
  annotations:
    summary: "WARN VaultBackupStale {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      last successful snapshot of {{ $labels.ins }} is {{ $value | humanizeDuration }} ago (or never)
      check `journalctl -t vault-backup` and node_exporter on {{ $labels.ip }}
      /ui/d/vault-overview?var-cls={{ $labels.cls }}

7 - Metrics

This page distinguishes raw OpenBao telemetry from recording metrics evaluated by vmalert. files/victoria/rules/vault.yml defines 14 recording rules. Their results are not available from /v1/sys/metrics.

Raw metrics vary with OpenBao version, member role, and runtime state. The metrics below are those used by Vault Overview and its rules, not the full telemetry output. See OpenBao telemetry metrics for the complete list.

Labels

VAULT metrics use these labels:

Recording rules use vault:ins:* for instances and vault:cls:* for clusters.

Raw Metric Behavior

OpenBao retains Vault’s vault_ prefix and exports Prometheus metrics through go-metrics. Note:

• Counters lack a _total suffix, for example vault_raft_apply and vault_token_creation.
• Summaries use milliseconds: quantiles and _sum for vault_core_handle_request are in milliseconds. Recording rules divide average latency by 1,000 to convert it to seconds.
• Some counters appear only after their first event, such as vault_audit_log_request_failure. Rules use increase_pure to count increases from zero.
• Only the leader reports Autopilot and follower metrics. They persist for 24 hours after it steps down (prometheus_retention_time). Rules use only the current active member’s values and map node_id / peer_id back to ins.
• Usage gauges (tokens, KV secrets, identity entities) are computed every ten minutes (usage_gauge_period).

Recording Rules

Expressions preserve source semantics. ins and cls indicate instance and cluster aggregation; actual retained labels follow the by and vector-matching clauses.

With no requests in a time window, average latency is 0/0 = NaN, which is discarded and appears as no data.

vault_up comes from INFRA’s shared aliveness rules, based on up{job="vault"}. It is not counted among the module’s rules. It indicates scrape success, not whether a member is unsealed or serving requests.

Common Raw Metrics

State and requests

Raft and Autopilot

Secrets, tokens, and audit

Runtime

Snapshot backups (from node_exporter’s textfile collector, under the node scrape job)

vault-backup writes these two metrics to /var/lib/node_exporter/textfile/vault.prom only after success. The file is deleted when a member is removed.

8 - FAQ

What Does VAULT Do?

VAULT supplies KMS (key management) and secrets storage for a Pigsty deployment, covering three main uses:

• Transparent data encryption: store PGSQL pg_tde master keys so leaked database disks and backups cannot be decrypted without VAULT keys.
• Object storage encryption: store keys for MINIO (Silo) KES server-side encryption.
• Application secrets: use KV, Transit, and AppRole to store/access passwords and API keys or call encryption services.

The first two integrations currently require manual configuration; automatic integration is still being developed. Applications can use the third immediately after deployment; see Usage.

Why OpenBao Instead of HashiCorp Vault?

HashiCorp changed Vault’s license to BSL in 2023. Later versions are no longer open-source software and are not packaged for distribution by Pigsty. OpenBao is a community fork from before that license change, hosted by the Linux Foundation under MPL-2.0. Its API is largely compatible; Vault libraries, tools, and most documentation remain useful references.

VAULT names the secrets-vault capability rather than a specific engine, following the same convention as the MINIO module using Silo.

How Many Members Should a Cluster Have?

Use one member for development, testing, and demos. Use three with an L2 VIP for production HA; see Configuration.

Raft requires a majority. If half or more of the members are unavailable, the cluster cannot serve requests. Three members tolerate one failure. Two members tolerate no failures and offer lower availability than one. The playbook warns about this topology; use it only temporarily during scaling.

What Happens When VAULT Is Unavailable?

The impact depends on how consumers use keys:

• Running consumers that have cached keys in memory can usually continue.
• Operations that must retrieve keys fail, such as restarting PostgreSQL encrypted with pg_tde or KES fetching a new data key.
• Application secret reads and encryption/decryption requests fail.

In production, VAULT should be at least as available as the databases that depend on it. VaultDown and VaultNoLeader are CRIT alerts.

Is Auto-Unseal Secure?

VAULT uses OpenBao static seal: a 32-byte key in files/pki/vault/unseal.key on the admin node is distributed to /etc/openbao/unseal.key on each member. Members unseal automatically after restart without manual key-share entry.

This balances operations and security: database restarts do not wait for manual unsealing. For stricter requirements, switch to manual unsealing or an external KMS seal after deployment; that is outside the current module’s automation.

Which Files Must Be Backed Up?

Back up the admin node’s entire files/pki/ directory and copy VAULT snapshots off-site:

• files/pki/vault/unseal.key: without it, all snapshots are unreadable.
• files/pki/vault/init.json: root token and recovery keys for administrative access after restoration.
• files/pki/ca/: Pigsty CA, which signs VAULT server certificates.
• A recent Raft snapshot from members’ /data/vault/backup/ or the admin node’s files/pki/vault/backup/.

Snapshots are encrypted and can be copied to other storage, but keep them separate from unseal.key. See Backups and Restore a Snapshot.

Why Does bao status Return Permission Denied?

BAO_CACERT points to /etc/openbao/tls/ca.crt, readable only by root and the openbao group. Regular users see:

failed to read environment: Error loading CA File: open /etc/openbao/tls/ca.crt: permission denied

Run as root (sudo -i), or use the globally readable CA: export BAO_CACERT=/etc/pki/ca.crt.

Why Does vault.yml Refuse to Initialize?

vault.yml initializes only when all required conditions hold. Common reasons for refusal:

• An unreachable member: the playbook cannot determine whether the cluster is initialized and aborts to prevent creating a second cluster. Restore network/service access to every member and rerun.
• All members are uninitialized, but files/pki/vault/init.json exists: the cluster was initialized before and has lost its data. To recover, see Rebuild After Total Data Loss. To create a new empty cluster, archive init.json first; see Destroy a Cluster.

Why Does vault-rm.yml Abort by Default?

vault_safeguard defaults to true, like MYSQL’s mysql_safeguard. Deleting VAULT deletes all its keys, making encrypted data and all backups permanently unreadable. The impact extends beyond any one database cluster.

After verifying the target and snapshot, explicitly override it with ./vault-rm.yml -l <exact-target> -e vault_safeguard=false, or use bin/vault-rm.

Why Are Metrics Token-Free and Served over HTTPS?

OpenBao normally requires a token for /v1/sys/metrics. VAULT enables unauthenticated_metrics_access on the node IP listener so monitoring needs no VAULT credentials. The loopback listener does not allow unauthenticated access. Metrics contain counts and state, not secrets.

OpenBao provides TLS only, with metrics and business APIs sharing port 8200. A separate plaintext port would increase exposure without benefit, so VictoriaMetrics scrapes over HTTPS using Pigsty CA.

Why Does VaultBackupStale Fire?

Each member runs vault-backup daily. The alert fires if a member has no successful snapshot for 26 hours, or has never succeeded. Common causes:

• Scheduled job failure: inspect journalctl -t vault-backup for an invalid token, missing active member, or full disk.
• node_exporter’s textfile collector is disabled: newer NODE versions configure its directory. Refresh older nodes with ./node.yml -l vault -t node_exporter.
• An invalid backup token, for example after snapshot restoration: run ./vault.yml -l vault -t vault_provision,vault_backup. The playbook checks the token and reissues/distributes it if necessary.

After fixing the issue, run vault-backup manually. The alert clears at the next evaluation.

What Causes Frequent Leader Changes?

Raft leaders maintain their term through heartbeats. Timeouts cause elections, commonly due to network jitter, high disk latency, or host load that prevents timely scheduling, especially on oversubscribed VMs.

Inspect Leader Elections, Follower Last Heartbeat, and Raft Commit Time in Vault Overview, then correlate with node CPU, disk, and network metrics. Occasional leader changes do not affect data consistency. HAProxy switches to the new active member automatically.

Where Are Audit Logs?

Each member enables a file audit device at /var/log/openbao/audit.log, rotated daily with seven files retained. OpenBao applies HMAC to sensitive fields.

Standbys forward requests to the active member, which records them. A member that has never been active may have no audit.log; this does not indicate broken audit configuration.

Audit logs are not collected into VictoriaLogs or deleted on VAULT removal. If audit writes fail, OpenBao rejects requests and VaultAuditFailure fires; check disk space first.

Can I Use an Existing External Vault?

Currently no. The module manages only OpenBao clusters deployed by Pigsty, not external or managed HashiCorp Vault, OpenBao, or cloud KMS services.

Applications may still access your existing Vault directly, but Pigsty deployment, monitoring, and future automatic consumer integration do not manage it.