VAULT on PIGSTY

FAQ

What Does VAULT Do?

VAULT supplies KMS (key management) and secrets storage for a Pigsty deployment, covering three main uses:

Transparent data encryption: store PGSQL pg_tde master keys so leaked database disks and backups cannot be decrypted without VAULT keys.
Object storage encryption: store keys for MINIO (Silo) KES server-side encryption.
Application secrets: use KV, Transit, and AppRole to store/access passwords and API keys or call encryption services.

The first two integrations currently require manual configuration; automatic integration is still being developed. Applications can use the third immediately after deployment; see Usage.

Metrics

This page distinguishes raw OpenBao telemetry from recording metrics evaluated by vmalert. files/victoria/rules/vault.yml defines 14 recording rules. Their results are not available from /v1/sys/metrics.

Raw metrics vary with OpenBao version, member role, and runtime state. The metrics below are those used by Vault Overview and its rules, not the full telemetry output. See OpenBao telemetry metrics for the complete list.

Labels

VAULT metrics use these labels:

Label	Meaning	Example
`job`	Scrape job	`vault`
`cls`	Cluster name, `vault_cluster`	`vault`
`ins`	Instance name, `<vault_cluster>-<vault_seq>`	`vault-1`
`ip`	Member address	`10.10.10.11`
`instance`	Scrape address	`10.10.10.11:8200`

Recording rules use vault:ins:* for instances and vault:cls:* for clusters.

Monitoring

VAULT reuses Pigsty’s observability stack. OpenBao metrics flow over HTTPS to VictoriaMetrics; service and backup logs flow through Journald / Vector to VictoriaLogs. vmalert loads 14 recording rules and six alerts, and Grafana provides the Vault Overview dashboard.

Dashboard

VAULT provides one dashboard: Vault Overview.

Vault Overview Dashboard

Vault Overview is available at /ui/d/vault-overview. Use the Cluster variable at the top to select a cluster.

Usage

After configuring and deploying VAULT, use this page to get started. The following engines and authentication methods are enabled by default:

Path	Type	Purpose
`secret/`	KV v2 secrets engine	Versioned static secrets such as passwords, API keys, and certificates
`transit/`	Transit secrets engine	Encryption as a service; applications send data for encryption/decryption while keys remain in VAULT
`auth/approle/`	AppRole authentication	Machine credentials for applications and automation
`auth/token/`	Token authentication	Built-in; maximum lease adjusted to ten years

OpenBao’s API is largely compatible with HashiCorp Vault, so common Vault client libraries and tools generally work directly. See the OpenBao documentation for full usage.

Administration

Common VAULT administration procedures:

Create a Cluster: initialize VAULT.
Check Status: inspect cluster, member, and Raft state.
Administrator Credentials: use the root token safely.
Add Members: add members or scale from one to three nodes.
Remove Members: retire existing members.
Rebuild a Member: recover from damaged data or replace hardware.
Update Configuration: refresh configuration and certificates.
Backups: create snapshots manually and keep off-site copies.
Restore a Snapshot: roll back data or rebuild after total data loss.
Upgrade OpenBao: update the software package.
Destroy a Cluster: remove and recreate VAULT.

See the VAULT FAQ for other questions.

Playbooks

VAULT provides two playbooks: vault.yml creates, updates, and scales clusters, while vault-rm.yml removes members or the whole cluster. The helper scripts bin/vault-add and bin/vault-rm wrap these playbooks.

`vault.yml`

Source: vault.yml

Installs, configures, and starts OpenBao in the hardcoded vault group. It is idempotent: repeated runs reconcile configuration changes without reinitializing the cluster.

vault.yml provides these task subsets:

id: derive node identity and variables through node_id.
vault_assert: validate member IDs, data paths, ports, and VIP configuration; derive instance names and members (always runs).
vault_install: install openbao and ensure the openbao system user exists.
vault_dir: create configuration, data, snapshot, and audit log directories.
vault_config: generate VAULT configuration.
- vault_cert: issue and distribute TLS certificates.
  - vault_cert_issue: issue server certificates with Pigsty CA on the admin node.
  - vault_cert_copy: copy the CA certificate, server certificate, and private key to members.
- vault_seal: generate and distribute the static seal key; abort if an existing member key differs from the admin node’s key.
- vault_conf: render openbao.hcl, the systemd unit, bao client environment, and log rotation configuration.
vault_launch: start stopped members. Members already running before execution restart one at a time only if their certificates, seal key, configuration, or unit changed, waiting for each to rejoin. Newly started members skip a second restart to avoid startup races.
vault_init: initialize once on the member with the lowest vault_seq, then wait for all members to unseal and complete Raft election (leader_address nonempty).
vault_provision: enable secret/, transit/, and approle; issue the backup token, reissuing it automatically if an existing token is invalid, for example from an old cluster.
vault_haproxy: generate the <VIP>:8200 HAProxy endpoint on members when VIP is enabled; otherwise remove it.
vault_backup: install vault-backup, its daily schedule, and take the first snapshot.
vault_register (aliases register, add_metrics): register members with VictoriaMetrics.

vault_init runs bao operator init only when all these conditions hold:

Parameters

VAULT provides eight parameters in two groups:

VAULT: five parameters for deployment and configuration.
VAULT_REMOVE: three parameters controlling cluster removal.

VAULT deliberately exposes few settings. Member count determines cluster mode, HA access reuses NODE’s VIP parameters, and other ports, directories, engines, and backup policies follow fixed conventions.

Parameter Overview

The VAULT group configures instance identity, cluster name, data directory, port, and service domain.

Parameter	Type	Level	Description
`vault_seq`	`int`	`I`	Required VAULT instance identifier
`vault_cluster`	`string`	`C`	Cluster name, default `vault`
`vault_data`	`path`	`C`	Data directory, default `/data/vault`
`vault_port`	`port`	`G`	API port, default `8200`
`vault_domain`	`string`	`G`	Service domain, default `vault.pigsty`

The VAULT_REMOVE group controls removal safeguards, data cleanup, and package removal.

Configuration

Before deploying VAULT, define a group named vault in the inventory. There is no profile or mode parameter: the mode is derived entirely from the number of members in vault.

Single node: no HA; suitable for development, testing, demos, and sandboxes.
Three nodes: tolerates one member failure and provides HA access with an L2 VIP; suitable for production.
Two nodes: functional but has no fault tolerance; treat it as a single-node setup.

Members	Quorum	Tolerated Failures	Mode	Service Endpoint
1	1	0	Lab	`vault.pigsty` points to the member
2	2	0	Warning	Any member failure loses quorum
3	2	1	Production	`vault.pigsty` points to the VIP; HAProxy forwards to the active member

VAULT uses OpenBao’s built-in Raft storage. Only one active member handles writes at a time; the remaining standby members forward requests to it.

Module: VAULT

OpenBao is an open-source key and secrets management system (MPL-2.0), forked by the community after HashiCorp Vault adopted the BSL license and hosted by the Linux Foundation. Its API is largely compatible with Vault. Pigsty’s VAULT module deploys an OpenBao cluster in the fixed vault group as the deployment’s KMS (key management service) and secrets store. Pigsty manages TLS, auto-unseal, the HA endpoint, backups, monitoring, and lifecycle.

Current status: Beta

VAULT is in Beta. Deployment, repeated execution, rolling restarts, standby read forwarding, VIP access, and snapshot backup/recovery have been validated on a three-member Ubuntu 24.04 cluster. Grafana display, alert behavior, and other operating systems still need validation. Automatic integration is not yet implemented for PGSQL pg_tde or MINIO KES; manual configuration is required. Test thoroughly in your own environment before production use, especially snapshot recovery.