Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

Module: PGSQL

Declare, deploy, expose, observe, back up, and manage PostgreSQL clusters with Pigsty v4.5.

PGSQL is the core Pigsty module. An Ansible inventory declares PostgreSQL clusters. Patroni and etcd manage high availability. pgBackRest provides backup and PITR. HAProxy, VIP, DNS, PgBouncer, and the observability stack expose and monitor database services.

This page follows the Pigsty v4.5.0 source layout. For exact defaults, use the parameter reference. Keeping defaults in one reference prevents conflicting or outdated copies.


Modeling and Configuration

  • Cluster model: clusters, instances, identity, and roles.
  • Architecture: relationships among Patroni, etcd, service access, and observability.
  • Cluster configuration: primary, replica, offline instance, synchronous commit, standby cluster, delayed cluster, and Citus.
  • Kernel: PostgreSQL major version, distribution, and package selection.
  • Users, databases, HBA, and ACL: business objects and access control.
  • Service access: read-write/read-only services, HAProxy, VIP, DNS, and pooling.
  • Extension catalog: the 600 currently packaged extensions and their platform coverage.

Deployment and Administration

TaskEntry point
Initialize a cluster or add an instanceCluster administration · pgsql.yml
Create or change usersUser administration · pgsql-user.yml
Create or change databasesDatabase administration · pgsql-db.yml
Change HBA or parametersHBA administration · Component administration
Patroni switchovers, maintenance, and incidentsPatroni administration
Install, create, update, or remove extensionsExtension administration
Monitor an external instancepgsql-monitor.yml
Prepare a migrationMigration · pgsql-migration.yml
Remove an instance or clusterSafe removal workflow · pgsql-rm.yml

pgsql.yml, pgsql-user.yml, pgsql-db.yml, and related playbooks change the target environment when run. By default, pgsql-rm.yml can delete data and backups. First identify the exact cluster or node and verify a recent backup. For removal, the operator must also type and confirm the exact target.


Backup and Recovery

Restore is destructive. Production recovery requires an independent, recent backup that has passed a restore test. Confirm each stage separately: shutdown, restore, data validation, timeline promotion, DCS rebuild, replica rebuild, and a new full backup.


Monitoring

The current source contains 29 PostgreSQL and PGCAT dashboards in files/grafana/pgsql. They cover fleets, clusters, instances, databases, tables, queries, sessions, transactions, replication, services, PgBouncer, PITR, and alerts.


Parameter Groups

The PGSQL parameter reference is the single documentation source for v4.5.0 defaults and semantics:

  • PG_ID: cluster and instance identity.
  • PG_BUSINESS: users, databases, services, and other business objects.
  • PG_INSTALL: kernel, packages, and extensions.
  • PG_BOOTSTRAP: Patroni bootstrap, replication, and database initialization.
  • PG_PROVISION: in-database objects and privileges.
  • PG_BACKUP: pgBackRest and backup repositories.
  • PG_ACCESS: PgBouncer, services, VIP, and DNS.
  • PG_MONITOR: exporters, monitoring registration, and metrics.
  • PG_REMOVE: removal safeguards and cleanup scope.

Further Reading

1 - Configuration

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

Pigsty uses configuration to control PostgreSQL behavior. This configuration combines inventory files in ~/pigsty/conf/*.yml with PGSQL parameters. After you write the configuration, you can reproduce a customized cluster in a few minutes. The configuration includes instances, users, databases, access control, extensions, and tuning policies.


Configuration Entry

  1. Prepare Inventory: Copy a pigsty/conf/*.yml template or write an Ansible inventory. Put cluster groups (all.children.<cls>.hosts) and global variables (all.vars) in the same file.
  2. Define Parameters: Override the required PGSQL parameters in vars. The override order is global → cluster → host. The last applicable setting determines the value.
  3. Apply Configuration: Use ./configure -c <conf>, bin/pgsql-add <cls>, and other playbooks to apply the configuration. Pigsty generates the required Patroni, PgBouncer, and pgBackRest files from these parameters.

The default demonstration inventory, conf/pgsql.yml, provides a minimal example. It defines one pg-meta cluster, global pg_version: 18, and a few application users and databases. You can add more clusters to this configuration.


Focus Areas & Documentation Index

Use the following pages to configure each part of a PostgreSQL deployment:

  • Cluster & Instances: Define instance topology (standalone, primary-replica, standby cluster, delayed cluster, Citus, etc.) through pg_cluster / pg_role / pg_seq / pg_upstream.
  • Kernel Version: Select the core version, flavor, and tuning templates using pg_version, pg_mode, pg_packages, pg_extensions, pg_conf, and other parameters.
  • Users/Roles: Declare system roles, business accounts, password policies, and connection pool attributes in pg_default_roles and pg_users.
  • Database Objects: Create databases as needed using pg_databases, baseline, schemas, extensions, pool_* fields and automatically integrate with pgbouncer/Grafana.
  • Access Control (HBA): Maintain host-based authentication policies using pg_default_hba_rules and pg_hba_rules to ensure access boundaries for different roles/networks.
  • Privilege Model (ACL): Converge object privileges through pg_default_privileges, pg_default_roles, pg_revoke_public parameters, providing an out-of-the-box layered role system.

Use these parameters to declare infrastructure in the inventory for your application requirements. Pigsty performs the operations and ensures idempotency.


A Typical Example

This example defines topology, kernel version, extensions, users, and databases in one configuration file:

all:
  children:
    pg-analytics:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
        10.10.10.12: { pg_seq: 2, pg_role: replica, pg_offline_query: true }
      vars:
        pg_cluster: pg-analytics
        pg_conf: olap.yml
        pg_extensions: [ postgis, timescaledb, pgvector ]
        pg_databases:
          - { name: bi, owner: dbuser_bi, schemas: [mart], extensions: [timescaledb], pool_mode: session }
        pg_users:
          - { name: dbuser_bi, password: DBUser.BI, roles: [dbrole_admin], pgbouncer: true }
  vars:
    pg_version: 18
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_hba_rules:
      - { user: dbuser_bi, db: bi, addr: intra, auth: ssl, title: 'BI only allows intranet SSL access' }
  • The pg-analytics cluster contains one primary and one offline replica.
  • Global settings specify pg_version: 18 with a set of extension examples and load olap.yml tuning.
  • Declare business objects in pg_databases and pg_users, automatically generating schema/extension and connection pool entries.
  • Additional pg_hba_rules restrict access sources and authentication methods.

Adjust and apply this inventory to create a customized PostgreSQL cluster without manual configuration.

1.1 - Cluster / Instance

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

Select instance and cluster types that meet your requirements.

Pigsty supports several PostgreSQL instance and cluster types:

  • Primary: Define a single instance cluster.
  • Replica: Define a basic HA cluster with one primary and one replica.
  • Offline: Define an instance dedicated to OLAP/ETL/interactive queries
  • Sync Standby: Enable synchronous commit to ensure no data loss.
  • Quorum Commit: Use quorum sync commit for a higher consistency level.
  • Standby Cluster: Clone an existing cluster and follow it
  • Delayed Cluster: Clone an existing cluster for emergency data recovery
  • Citus Cluster: Define a Citus distributed database cluster

Primary

The simplest cluster contains one primary instance:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test

This configuration contains only identity parameters. Matching the Ansible group name to pg_cluster makes selectors such as -l pg-test convenient. However, matching group names are not required for membership. Pigsty finds members from each host’s pg_cluster identity. One PostgreSQL cluster can therefore span multiple inventory groups.

Use the following command to create this cluster:

bin/pgsql-add pg-test

One instance can be suitable for demonstrations, development tests, temporary hosting, or non-critical analytical tasks. A single-node cluster has no high availability. After hardware failure, use PITR or another recovery method to meet RTO/RPO requirements. Consider adding read-only replicas for this reason.


Replica

To add a read-only replica, add a node to pg-test and set its pg_role to replica.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }  # <--- newly added replica
  vars:
    pg_cluster: pg-test

If the cluster does not exist, create the complete cluster. If the primary is initialized, add a replica to the existing cluster:

bin/pgsql-add pg-test               # initialize the entire cluster at once
bin/pgsql-add pg-test 10.10.10.12   # add replica to existing cluster

If the primary fails, the high-availability system can promote a replica to take over. Replicas also process read-only queries. Many applications have more reads than writes, so replicas can handle most of their read-only workload.


Offline

Offline instances are read-only replicas for slow queries, ETL, OLAP, and interactive queries. Slow queries and long transactions can reduce online application performance and stability. We therefore recommend that you isolate them from online traffic.

To add an offline instance, add a new instance and set pg_role to offline.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline }  # <--- newly added offline replica
  vars:
    pg_cluster: pg-test

Dedicated offline instances operate like ordinary replicas but act as backup servers for the pg-test-replica service. Offline and primary instances serve read-only traffic only when every replica instance is down.

A separate offline server may be too expensive when resources are limited. Instead, mark an existing replica with pg_offline_query. This replica handles both online read-only requests and offline queries. Use pg_default_hba_rules and pg_hba_rules for additional access control on offline instances.


Sync Standby

With Sync Standby enabled, PostgreSQL selects one replica as the sync standby. The other replicas are candidates. Before it confirms a commit, the primary waits for the sync standby to flush data to disk. The sync standby has the latest data without replication lag. A switchover to that standby has no data loss.

PostgreSQL uses asynchronous streaming replication by default. If the primary fails, WAL that has not replicated may be lost. pg_rpo is Patroni’s sampled lag threshold for failover candidates. It is not a hard limit on actual data loss. The loss window also depends on write rate, replication state, and Patroni sampling time.

Critical workloads, such as financial transactions, may permit no data loss or read replication lag. Use synchronous commit for these requirements. To enable sync standby mode, select crit.yml with pg_conf.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: crit.yml   # <--- use crit template

To enable sync standby on an existing cluster, configure the cluster and enable synchronous_mode:

$ pg edit-config pg-test    # run as admin user on admin node
+++
-synchronous_mode: false    # <--- old value
+synchronous_mode: true     # <--- new value
 synchronous_mode_strict: false

Apply these changes? [y/N]: y

Patroni manages synchronous_standby_names automatically. It selects one replica as the sync standby. It writes and applies that replica’s application_name in the primary configuration.


Quorum Commit

Quorum Commit lets you set commit acknowledgment criteria, particularly for clusters with multiple replicas. You can adjust the consistency level and the associated availability tradeoff.

To require confirmation from at least two replicas, set synchronous_node_count through Patroni cluster configuration. Apply this setting:

synchronous_mode: true          # ensure synchronous commit is enabled
synchronous_node_count: 2       # specify "at least" how many replicas must successfully commit

To require more synchronous replicas, increase synchronous_node_count. After cluster size changes, verify that this setting remains valid to avoid service unavailability.

In this case, the PostgreSQL configuration parameter synchronous_standby_names is automatically managed by Patroni.

synchronous_standby_names = '2 ("pg-test-3","pg-test-2")'
Example: Using multiple sync standbys
$ pg edit-config pg-test
---
+synchronous_node_count: 2

Apply these changes? [y/N]: y

After applying the configuration, two sync standbys appear.

+ Cluster: pg-test (7080814403632534854) +---------+----+-----------+-----------------+
| Member    | Host        | Role         | State   | TL | Lag in MB | Tags            |
+-----------+-------------+--------------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.10 | Leader       | running |  1 |           | clonefrom: true |
| pg-test-2 | 10.10.10.11 | Sync Standby | running |  1 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.12 | Sync Standby | running |  1 |         0 | clonefrom: true |
+-----------+-------------+--------------+---------+----+-----------+-----------------+

You can also require confirmation from any n replicas. For example, this configuration requires any one replica:

synchronous_mode: quorum        # use quorum commit
postgresql:
  parameters:                   # modify PostgreSQL's configuration parameter synchronous_standby_names, using `ANY n ()` syntax
    synchronous_standby_names: 'ANY 1 (*)'  # you can specify a specific replica list or use * to wildcard all replicas.
Example: Enable ANY quorum commit
$ pg edit-config pg-test

+    synchronous_standby_names: 'ANY 1 (*)' # in ANY mode, this parameter is needed
- synchronous_node_count: 2  # in ANY mode, this parameter is not needed

Apply these changes? [y/N]: y

After applying, the configuration takes effect, and all standbys become regular replicas in Patroni. However, in pg_stat_replication, you can see sync_state becomes quorum.


Standby Cluster

You can clone an existing cluster and create a standby cluster for data migration, horizontal splitting, multi-region deployment, or disaster recovery.

Normally, the standby cluster follows the upstream cluster and keeps its data synchronized. Promote it to make it an independent cluster.

Use a normal cluster definition, then add pg_upstream to its primary. This primary is called the Standby Leader.

This inventory defines pg-test and its standby cluster, pg-test2:

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-test2 is the standby cluster of pg-test
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 } # <--- pg_upstream defined here
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-test2 }

pg-test2-1 replicates from pg-test and acts as the Standby Leader for pg-test2.

Set pg_upstream on the standby cluster primary to pull backups automatically from the upstream cluster.

bin/pgsql-add pg-test     # create original cluster
bin/pgsql-add pg-test2    # create standby cluster
Example: Change replication upstream

If necessary (e.g., upstream primary-standby switchover/failover), you can change the standby cluster’s replication upstream through cluster configuration.

To do this, simply change standby_cluster.host to the new upstream IP address and apply.

$ pg edit-config pg-test2

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.13     # <--- old upstream
+  host: 10.10.10.12     # <--- new upstream
   port: 5432

 Apply these changes? [y/N]: y
Example: Promote standby cluster

You can promote the standby cluster at any time. It then accepts writes independently and can diverge from the original cluster.

To do this, you must configure the cluster and completely erase the standby_cluster section, then apply.

$ pg edit-config pg-test2
-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Apply these changes? [y/N]: y
Example: Cascade replication

If you specify pg_upstream on a replica instead of the primary, you can configure cascade replication for the cluster.

For cascade replication, use the IP address of an instance in the same cluster. Otherwise, initialization fails. The replica streams from that instance instead of the primary.

The WAL relay is called a Bridge Instance. It reduces the WAL transmission load on the primary. Consider cascade replication through bridge instances when you have dozens of replicas.

pg-test:
 hosts: # pg-test-1 ---> pg-test-2 ---> pg-test-3
   10.10.10.11: { pg_seq: 1, pg_role: primary }
   10.10.10.12: { pg_seq: 2, pg_role: replica } # <--- bridge instance
   10.10.10.13: { pg_seq: 3, pg_role: replica, pg_upstream: 10.10.10.12 }
   # ^--- replicate from pg-test-2 (bridge) instead of pg-test-1 (primary)
 vars: { pg_cluster: pg-test }

Delayed Cluster

A Delayed Cluster is a special type of standby cluster used to quickly recover “accidentally deleted” data.

For example, define pg-testdelay to retain the state of pg-test from one hour earlier:

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-testdelay is the delayed cluster of pg-test
pg-testdelay:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11, pg_delay: 1d }
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-testdelay }

You can also configure a “replication delay” on an existing standby cluster.

$ pg edit-config pg-testdelay
 standby_cluster:
   create_replica_methods:
   - basebackup
   host: 10.10.10.11
   port: 5432
+  recovery_min_apply_delay: 1h    # <--- add delay duration here, e.g. 1 hour

Apply these changes? [y/N]: y

After accidental row or table deletion, adjust this delay to advance the delayed cluster to the required time. Read the data from that cluster to repair the original cluster.

Delayed clusters need additional resources. They recover data much faster than PITR and have less effect on the system. Consider them for critical clusters.


Citus Cluster

Pigsty natively supports Citus. See conf/ha/citus.yml for a complete example.

To define a Citus cluster, you need to specify the following parameters:

Add HBA rules for SSL access from localhost and other data nodes. For example:

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # global parameters for all Citus clusters
    pg_mode: citus                    # pgsql cluster mode must be set to: citus
    pg_shard: pg-citus                # citus horizontal shard name: pg-citus
    pg_primary_db: meta               # citus database name: meta
    pg_dbsu_password: DBUser.Postgres # if using dbsu, need to configure a password for it
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

On the coordinator, create distributed tables and reference tables. You can query them from any data node. From Citus 11.2, any Citus database node can act as coordinator.

SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table($$public.pgbench_accounts$$);
SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_branches$$);
SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_history$$);
SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_tellers$$);

1.2 - Kernel Version

How to choose the appropriate PostgreSQL kernel and major version.

Choosing a “kernel” in Pigsty means determining the PostgreSQL major version, mode/distribution, packages to install, and tuning templates to load.

The Pigsty v4.5 source supports PostgreSQL 14-18 and uses 18 by default. Use the following configuration settings to select a kernel.


Major Version and Packages

  • pg_version: Select the PostgreSQL major version. The default is 18. Pigsty maps the version to the correct package name prefix.
  • pg_packages: Select core packages with package aliases. The default, pgsql-main pgsql-common, includes the kernel, Patroni, PgBouncer, pgBackRest, and other common tools.
  • pg_extensions: Select additional extension packages. Aliases are supported. The default is empty, which installs only core dependencies.
all:
  vars:
    pg_version: 18
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_extensions: [ postgis, timescaledb, pgvector, pgml ]

During installation, Ansible downloads packages for pg_version=18 and installs extension files on the system. Database initialization scripts can then run CREATE EXTENSION.

Extension availability differs between versions in Pigsty’s offline repository. PostgreSQL 14 has fewer extensions, while 17/18 have the widest coverage. If an extension is not included, add it through repo_extra_packages.


Kernel Mode (pg_mode)

pg_mode selects the kernel distribution. The default, pgsql, selects standard PostgreSQL. Pigsty supports these modes:

ModeScenario
pgsqlStandard PostgreSQL, HA + replication
citusCitus distributed cluster, requires additional pg_shard / pg_group
gpsqlCloudberry / Greenplum / MatrixDB
mssqlBabelfish
mysqlOpenGauss/HaloDB compatible with MySQL protocol
polarAlibaba PolarDB (based on pg polar distribution)
ivoryIvorySQL (Oracle-compatible syntax)
pgtdePercona PostgreSQL with pg_tde under /usr/pgtde-$v
orioleOrioleDB storage engine
agensAgensGraph graph database kernel
pgedgepgEdge distributed replication kernel

pg_mode selects binary paths, Patroni integration, and some kernel-specific behavior. It does not automatically add every required package, extension, and application database. For deployment, use the matching conf/*.yml template. Alternatively, set pg_packages, pg_extensions, pg_libs, and pg_databases explicitly. This is a minimal Citus example:

all:
  children:
    pg-citus1:
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1, pg_group: 0 }
    pg-citus2:
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2, pg_group: 1 }
  vars:
    pg_mode: citus
    pg_shard: pg-citus
    pg_primary_db: citus
    pg_extensions: [ citus ]
    pg_libs: 'citus, pg_stat_statements'
    pg_databases:
      - { name: citus, extensions: [ citus ] }

conf/ha/citus.yml provides the current complete example. The example above explicitly installs Citus packages and creates the extension in the citus database.


Extensions and Pre-installed Objects

Use these parameters to control components loaded automatically after database startup:

  • pg_libs: List to write to shared_preload_libraries. For example: pg_libs: 'timescaledb, pg_stat_statements, auto_explain'.
  • pg_default_extensions / pg_default_schemas: Control schemas and extensions pre-created in template1 and postgres by initialization scripts.
  • pg_parameters: Rendered by Pigsty into postgresql.auto.conf during configuration. Do not also manage the same settings manually with ALTER SYSTEM.

Example: Enable TimescaleDB, pgvector and customize some system parameters.

pg-analytics:
  vars:
    pg_cluster: pg-analytics
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_default_extensions:
      - { name: timescaledb }
      - { name: vector }
    pg_parameters:
      timescaledb.max_background_workers: 8

During initialization, Pigsty creates default extensions in template1 and postgres. New databases based on template1 inherit these objects. Pigsty writes pg_parameters directly to postgresql.auto.conf.


Tuning Template (pg_conf)

pg_conf selects a Patroni template from roles/pgsql/templates/*.yml. Pigsty includes four general templates:

TemplateApplicable Scenario
oltp.ymlDefault template, for 4–128 core TP workload
olap.ymlOptimized for analytical scenarios
crit.ymlEmphasizes sync commit/minimal latency, suitable for zero-loss scenarios like finance
tiny.ymlLightweight machines / edge scenarios / resource-constrained environments

You can replace the template or customize a YAML file in templates/. Then select it in the cluster vars.

pg-ledger:
  hosts: { 10.10.10.21: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-ledger
    pg_conf: crit.yml
    pg_parameters:
      synchronous_commit: 'remote_apply'
      max_wal_senders: 16
      wal_keep_size: '2GB'

Pigsty uses crit.yml as the Patroni configuration and writes pg_parameters overrides to postgresql.auto.conf. The instances then use synchronous commit immediately.


Combined Instance: A Complete Example

pg-rag:
  hosts:
    10.10.10.31: { pg_seq: 1, pg_role: primary }
    10.10.10.32: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-rag
    pg_version: 18
    pg_mode: pgsql
    pg_conf: olap.yml
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_extensions: [ pgvector, pgml, postgis ]
    pg_libs: 'pg_stat_statements, auto_explain'
    pg_parameters:
      max_parallel_workers: 8
      shared_buffers: '32GB'
  • First primary + one replica, using olap.yml tuning.
  • Install PG18 plus common RAG extensions; only libraries that actually require preloading belong in pg_libs.
  • Patroni/pgbouncer/pgbackrest generated by Pigsty, no manual intervention needed.

Adjust these parameters for the application requirements to customize the kernel configuration.

1.3 - Package Alias

Pigsty provides a package alias translation mechanism that shields the differences in binary package details across operating systems, making installation easier.

PostgreSQL package names differ between operating systems:

  • EL systems (RHEL/Rocky/Alma/…) use formats like pgvector_18, postgis36_18*
  • Debian/Ubuntu systems use formats like postgresql-18-pgvector, postgresql-18-postgis-3

Without aliases, you need different naming rules for each system. You must also include the correct PostgreSQL version in the names.

Package Alias

Use a Package Alias to specify the same component on different platforms. Pigsty converts it to the correct package names:

# Using aliases - simple, unified, cross-platform
pg_extensions: [ postgis, pgvector, timescaledb ]

# Equivalent to actual package names on EL9 + PG18
pg_extensions: [ postgis36_18*, pgvector_18*, timescaledb-tsl_18* ]

# Equivalent to actual package names on Ubuntu 24 + PG18
pg_extensions: [ postgresql-18-postgis-3, postgresql-18-pgvector, postgresql-18-timescaledb-tsl ]

Alias Translation

An alias can also identify a group of packages. For example, the default pg_packages value is:

pg_packages:                      # pg packages to be installed, alias can be used
  - pgsql-main pgsql-common

Pigsty reads the alias list for the current operating system. For el10.x86_64, it expands the aliases to PostgreSQL kernel, extension, and utility packages:

pgsql-main:    "postgresql$v postgresql$v-server postgresql$v-libs postgresql$v-contrib postgresql$v-plperl postgresql$v-plpython3 postgresql$v-pltcl postgresql$v-llvmjit pg_repack_$v* wal2json_$v* pgvector_$v*"
pgsql-common:  "patroni patroni-etcd pgbouncer pgbackrest pg_exporter pgbackrest_exporter vip-manager"

Then Pigsty substitutes the selected PostgreSQL major version into pgsql-main. For example, with pg_version = 18:

pg18-main:   "postgresql18 postgresql18-server postgresql18-libs postgresql18-contrib postgresql18-plperl postgresql18-plpython3 postgresql18-pltcl postgresql18-llvmjit pg_repack_18* wal2json_18* pgvector_18*"

This process lets you select functional components without specifying platform-specific package details.


Which Variables Can Use Aliases?

These four parameters accept package aliases. Pigsty converts the aliases to package names through the same process:


Alias List

The Pigsty source directory roles/node_id/vars/ contains alias mappings for each operating system and architecture:


How It Works

Alias Translation Process

User config alias --> Detect OS -->  Find alias mapping table ---> Replace $v placeholder ---> Install actual packages
     ↓                 ↓                   ↓                                   ↓
  postgis          el9.x86_64         postgis36_$v*                   postgis36_18*
  postgis          u24.x86_64         postgresql-$v-postgis-3         postgresql-18-postgis-3

Version Placeholder

The alias system uses $v as a PostgreSQL version placeholder. Pigsty replaces each $v in an alias with the value of pg_version.

For example, when pg_version: 18:

Alias Definition (EL)Expanded Result
postgresql$v*postgresql18*
pgvector_$v*pgvector_18*
timescaledb-tsl_$v*timescaledb-tsl_18*
Alias Definition (Debian/Ubuntu)Expanded Result
postgresql-$vpostgresql-18
postgresql-$v-pgvectorpostgresql-18-pgvector
postgresql-$v-timescaledb-tslpostgresql-18-timescaledb-tsl

Wildcard Matching

On EL systems, many aliases use the * wildcard to match related subpackages. For example:

  • postgis36_18* will match postgis36_18, postgis36_18-client, postgis36_18-utils, etc.
  • postgresql18* will match postgresql18, postgresql18-server, postgresql18-libs, postgresql18-contrib, etc.

One alias can install a complete extension. You do not need to list each subpackage.

1.4 - User/Role

How to define and customize PostgreSQL users and roles through configuration?

In this document, “user” refers to a logical object within a database cluster created with CREATE USER/ROLE.

In PostgreSQL, users belong to the database cluster, rather than a specific database. Create business users before you create business databases.

Pigsty defines roles and users through two config parameters:

The first parameter defines roles and users shared across the environment. The second defines business roles and users for one cluster. Both parameters use arrays of user definition objects. Pigsty creates users and roles in array order. Thus, a later user can belong to an earlier role.

By default, all users marked with pgbouncer: true are added to the Pgbouncer connection pool user list.


Define Users

Example from Pigsty demo pg-meta cluster:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
      - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }
      - {name: dbuser_remove   ,state: absent }  # use state: absent to delete user

Each user or role definition is an object. Only name is required:

- name: dbuser_meta               # REQUIRED, `name` is the only mandatory field
  state: create                   # Optional, user state: create (default), absent
  password: DBUser.Meta           # Optional, password, can be scram-sha-256 hash or plaintext
  login: true                     # Optional, can login, default true
  superuser: false                # Optional, is superuser, default false
  createdb: false                 # Optional, can create databases, default false
  createrole: false               # Optional, can create roles, default false
  inherit: true                   # Optional, inherit role privileges, default true
  replication: false              # Optional, can replicate, default false
  bypassrls: false                # Optional, bypass row-level security, default false
  connlimit: -1                   # Optional, connection limit, default -1 (unlimited)
  expire_in: 3650                 # Optional, expire N days from creation (priority over expire_at)
  expire_at: '2030-12-31'         # Optional, expiration date in YYYY-MM-DD format
  comment: pigsty admin user      # Optional, user comment
  roles: [dbrole_admin]           # Optional, roles array
  parameters:                     # Optional, role-level config params
    search_path: public
  pgbouncer: true                 # Optional, add to connection pool user list, default false
  pool_mode: transaction          # Optional, pgbouncer pool mode, default transaction
  pool_connlimit: 100             # Optional, user-level max pool connections; omitted inherits global default 100

User-level pool quota is consistently defined by pool_connlimit (mapped to Pgbouncer max_user_connections).


Parameter Overview

The only required field is name. It must be a valid username that is unique within the cluster. All other parameters have defaults.

FieldCategoryTypeAttrDescription
nameBasicstringRequiredUsername, must be valid and unique
stateBasicenumOptionalState: create (default), absent
passwordBasicstringMutableUser password, plaintext or hash
commentBasicstringMutableUser comment
loginPrivilegeboolMutableCan login, default true
superuserPrivilegeboolMutableIs superuser, default false
createdbPrivilegeboolMutableCan create databases, default false
createrolePrivilegeboolMutableCan create roles, default false
inheritPrivilegeboolMutableInherit role privileges, default true
replicationPrivilegeboolMutableCan replicate, default false
bypassrlsPrivilegeboolMutableBypass RLS, default false
connlimitPrivilegeintMutableConnection limit, -1 unlimited
expire_inValidityintMutableExpire N days from now (priority)
expire_atValiditystringMutableExpiration date, YYYY-MM-DD format
rolesRolearrayAdditiveRoles array, string or object format
parametersParamsobjectMutableRole-level parameters
pgbouncerPoolboolMutableAdd to connection pool, default false
pool_modePoolenumMutablePool mode: transaction (default)
pool_connlimitPoolintMutablePool user max connections

Parameter Details

name

This required string specifies the username. The username must be unique within the cluster.

Use a valid PostgreSQL identifier that matches ^[a-z_][a-z0-9_]{0,62}$. Start with a lowercase letter or underscore. Use only lowercase letters, digits, and underscores. The maximum length is 63 characters.

- name: dbuser_app         # Standard naming
- name: app_readonly       # Underscore separated
- name: _internal          # Underscore prefix (for internal roles)

state

Enum for user operation: create or absent. Default create.

StateDescription
createDefault, create user, update if exists
absentDelete user with DROP ROLE
- name: dbuser_app             # state defaults to create
- name: dbuser_old
  state: absent                # Delete user

Pigsty prevents deletion of these system users with state: absent to prevent cluster failure:

password

This mutable string specifies the user password. Users without a password cannot log in with password authentication.

Password can be:

FormatExampleDescription
PlaintextDBUser.MetaNot recommended, logged to config
SCRAM-SHA-256SCRAM-SHA-256$4096:xxx$yyy:zzzRecommended, PG10+ default
MD5 hashmd5...Legacy compatibility
# Plaintext (not recommended, logged to config)
- name: dbuser_app
  password: MySecretPassword

# SCRAM-SHA-256 hash (recommended)
- name: dbuser_app
  password: 'SCRAM-SHA-256$4096:xxx$yyy:zzz'

When Pigsty sets a password, it temporarily disables logging to prevent password disclosure:

SET log_statement TO 'none';
ALTER USER "dbuser_app" PASSWORD 'xxx';
SET log_statement TO DEFAULT;

To generate SCRAM-SHA-256 hash:

# Using PostgreSQL (requires pgcrypto extension)
psql -c "SELECT encode(digest('password' || 'username', 'sha256'), 'hex')"

comment

String, mutable. User comment, defaults to business user {name}.

Pigsty sets the comment with COMMENT ON ROLE. Special characters are supported, and Pigsty automatically escapes quotes.

- name: dbuser_app
  comment: 'Main business application account'
COMMENT ON ROLE "dbuser_app" IS 'Main business application account';

login

This mutable Boolean controls login access. The default is true.

Set false to create a Role rather than a user. Roles usually group permissions.

In PostgreSQL, CREATE USER equals CREATE ROLE ... LOGIN.

# Create login-able user
- name: dbuser_app
  login: true

# Create role (no login, for permission grouping)
- name: dbrole_custom
  login: false
  comment: custom permission role
CREATE USER "dbuser_app" LOGIN;
CREATE USER "dbrole_custom" NOLOGIN;

superuser

This mutable Boolean controls superuser privileges. The default is false.

Superusers have full database privileges. They bypass all permission checks.

- name: dbuser_admin
  superuser: true            # Dangerous: full privileges
ALTER USER "dbuser_admin" SUPERUSER;

Pigsty provides a default superuser through pg_admin_username (dbuser_dba). Do not create additional superusers unless necessary.

createdb

This mutable Boolean controls permission to create databases. The default is false.

- name: dbuser_dev
  createdb: true             # Allow create database
ALTER USER "dbuser_dev" CREATEDB;

Some applications, such as Gitea and Odoo, may require the CREATEDB privilege for their administration users.

createrole

This mutable Boolean controls permission to create other roles. The default is false.

Users with CREATEROLE can create, modify, and delete other roles that are not superusers.

- name: dbuser_admin
  createrole: true           # Allow manage other roles
ALTER USER "dbuser_admin" CREATEROLE;

inherit

This mutable Boolean controls automatic inheritance of privileges from roles to which the user belongs. The default is true.

If you set false, use an explicit SET ROLE command to use the role privileges.

# Auto-inherit role privileges (default)
- name: dbuser_app
  inherit: true
  roles: [dbrole_readwrite]

# Requires explicit SET ROLE
- name: dbuser_special
  inherit: false
  roles: [dbrole_admin]
ALTER USER "dbuser_special" NOINHERIT;
-- User must execute SET ROLE dbrole_admin to get privileges

replication

This mutable Boolean controls permission to start streaming replication. The default is false.

Usually, only replication users (replicator) need this permission. Regular users should not have it unless they need logical decoding subscriptions.

- name: replicator
  replication: true          # Allow streaming replication
  roles: [pg_monitor, dbrole_readonly]
ALTER USER "replicator" REPLICATION;

bypassrls

This mutable Boolean controls permission to bypass row-level security (RLS) policies. The default is false.

When enabled, the user can access all rows, even when RLS policies apply. Usually, only administrators need this permission.

- name: dbuser_myappadmin
  bypassrls: true            # Bypass RLS policies
ALTER USER "dbuser_myappadmin" BYPASSRLS;

connlimit

This mutable integer sets the maximum number of concurrent connections. The default is -1 (unlimited).

A positive integer limits the number of simultaneous sessions for this user. The limit does not apply to superusers.

- name: dbuser_app
  connlimit: 100             # Max 100 concurrent connections

- name: dbuser_batch
  connlimit: 10              # Limit batch user connections
ALTER USER "dbuser_app" CONNECTION LIMIT 100;

expire_in

This mutable integer sets expiration to N days from the current date.

This parameter takes priority over expire_at. Each playbook run recalculates the expiration date. Use this parameter for temporary users that need periodic renewal.

- name: temp_user
  expire_in: 30              # Expire in 30 days

- name: contractor_user
  expire_in: 90              # Expire in 90 days

Generates SQL:

-- expire_in: 30, assuming current date is 2025-01-01
ALTER USER "temp_user" VALID UNTIL '2025-01-31';

expire_at

String, mutable. Expiration date in YYYY-MM-DD format, or special value infinity.

This parameter has lower priority than expire_in. Use infinity for users that must not expire.

- name: contractor_user
  expire_at: '2024-12-31'    # Expire on specific date

- name: permanent_user
  expire_at: 'infinity'      # Never expires
ALTER USER "contractor_user" VALID UNTIL '2024-12-31';
ALTER USER "permanent_user" VALID UNTIL 'infinity';

roles

This additive array lists the roles to which the user belongs. Array elements can be strings or objects.

Simple format - strings for role names:

- name: dbuser_app
  roles:
    - dbrole_readwrite
    - pg_read_all_data
GRANT "dbrole_readwrite" TO "dbuser_app";
GRANT "pg_read_all_data" TO "dbuser_app";

Use objects for more detailed control:

- name: dbuser_app
  roles:
    - dbrole_readwrite                            # Simple string: GRANT role
    - { name: dbrole_admin, admin: true }         # WITH ADMIN OPTION
    - { name: pg_monitor, set: false }            # PG16+: disallow SET ROLE
    - { name: pg_signal_backend, inherit: false } # PG16+: don't auto-inherit
    - { name: old_role, state: absent }           # Revoke role membership

Object Format Parameters:

ParamTypeDescription
namestringRole name (required)
stateenumgrant (default) or absent/revoke: control membership
adminbooltrue: WITH ADMIN OPTION, false: REVOKE ADMIN
setboolPG16+: true: WITH SET TRUE, false: REVOKE SET
inheritboolPG16+: true: WITH INHERIT TRUE, false: REVOKE INHERIT

PostgreSQL 16+ New Features:

PostgreSQL 16 introduced finer-grained role membership control:

  • ADMIN OPTION: Allow granting role to other users
  • SET OPTION: Allow using SET ROLE to switch to this role
  • INHERIT OPTION: Auto-inherit this role’s privileges
# PostgreSQL 16+ complete example
- name: dbuser_app
  roles:
    # Normal membership
    - dbrole_readwrite

    # Can grant dbrole_admin to other users
    - { name: dbrole_admin, admin: true }

    # Cannot SET ROLE to pg_monitor (only inherit privileges)
    - { name: pg_monitor, set: false }

    # Don't auto-inherit pg_execute_server_program (need explicit SET ROLE)
    - { name: pg_execute_server_program, inherit: false }

    # Revoke old_role membership
    - { name: old_role, state: absent }

The set and inherit options work only in PG16+. Earlier versions ignore them and add warning comments.

parameters

This mutable object sets role-level configuration parameters with ALTER ROLE ... SET. The parameters apply to all sessions for this user.

- name: dbuser_analyst
  parameters:
    work_mem: '256MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'
    log_statement: 'all'
ALTER USER "dbuser_analyst" SET "work_mem" = '256MB';
ALTER USER "dbuser_analyst" SET "statement_timeout" = '5min';
ALTER USER "dbuser_analyst" SET "search_path" = 'analytics,public';
ALTER USER "dbuser_analyst" SET "log_statement" = 'all';

Use the special value DEFAULT to reset a parameter to its PostgreSQL default. This value is case-insensitive:

- name: dbuser_app
  parameters:
    work_mem: DEFAULT          # Reset to default
    statement_timeout: '30s'   # Set new value
ALTER USER "dbuser_app" SET "work_mem" = DEFAULT;
ALTER USER "dbuser_app" SET "statement_timeout" = '30s';

Common role-level params:

ParameterDescriptionExample
work_memQuery work memory'64MB'
statement_timeoutStatement timeout'30s'
lock_timeoutLock wait timeout'10s'
idle_in_transaction_session_timeoutIdle transaction timeout'10min'
search_pathSchema search path'app,public'
log_statementLog level'ddl'
temp_file_limitTemp file size limit'10GB'

Query user-level parameters through the pg_db_role_setting system view.

pgbouncer

This mutable Boolean adds the user to the Pgbouncer user list. The default is false.

For production users that need connection pool access, explicitly set pgbouncer: true. The default, false, prevents accidental pool access for internal users.

# Prod user: needs connection pool
- name: dbuser_app
  password: DBUser.App
  pgbouncer: true

# Internal user: no connection pool needed
- name: dbuser_internal
  password: DBUser.Internal
  pgbouncer: false           # Default, can be omitted

Users with pgbouncer: true are added to /etc/pgbouncer/userlist.txt.

pool_mode

Enum, mutable. User-level pool mode: transaction, session, or statement. Default transaction.

ModeDescriptionUse Case
transactionReturn connection after txnMost OLTP apps, default
sessionReturn connection after sessionApps needing session state
statementReturn after each statementSimple stateless queries
# DBA user: session mode (may need SET commands etc.)
- name: dbuser_dba
  pgbouncer: true
  pool_mode: session

# Normal business user: transaction mode
- name: dbuser_app
  pgbouncer: true
  pool_mode: transaction

User-level pool params are configured via /etc/pgbouncer/useropts.txt:

dbuser_dba      = pool_mode=session max_user_connections=16
dbuser_monitor  = pool_mode=session max_user_connections=8

pool_connlimit

This mutable integer sets the maximum pool connections for the user. If you omit it, Pigsty does not generate a user-level override. The global pgbouncer.ini default of 100 then applies. PgBouncer uses 0 for unlimited connections.

- name: dbuser_app
  pgbouncer: true
  pool_connlimit: 50         # Max 50 pool connections for this user

ACL System

Pigsty provides a built-in access control / ACL model. Assign these default business roles to users as required:

RolePrivilegesTypical Use Case
dbrole_readwriteGlobal read-writePrimary application accounts
dbrole_readonlyGlobal read-onlyRead-only application access
dbrole_adminDDL privilegesApplication administrators and table creation
dbrole_offlineIndependent read-only; instance scope controlled by HBAAd hoc users, ETL, and analytics
# Typical business user configuration
pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true
    roles: [dbrole_readwrite]    # Prod account, read-write

  - name: dbuser_readonly
    password: DBUser.Readonly
    pgbouncer: true
    roles: [dbrole_readonly]     # Read-only account

  - name: dbuser_admin
    password: DBUser.Admin
    pgbouncer: true
    roles: [dbrole_admin]        # Admin, can execute DDL

  - name: dbuser_etl
    password: DBUser.ETL
    roles: [dbrole_offline]      # Analytics account; restrict instance scope through HBA

dbrole_offline does not restrict a user to offline instances by itself. To apply this restriction, set role: offline on the corresponding HBA rule. See Offline Role and Instance Isolation.

To define your own ACL system, customize these parameters and templates:


Pgbouncer Users

Pigsty enables Pgbouncer as connection pool middleware by default. Pigsty adds users from pg_users to the Pgbouncer user list only when they have pgbouncer: true.

Users in connection pool are listed in /etc/pgbouncer/userlist.txt:

"postgres" ""
"dbuser_wiki" "SCRAM-SHA-256$4096:+77dyhrPeFDT/TptHs7/7Q==$KeatuohpKIYzHPCt/tqBu85vI11o9mar/by0hHYM2W8=:X9gig4JtjoS8Y/o1vQsIX/gY1Fns8ynTXkbWOjUfbRQ="
"dbuser_view" "SCRAM-SHA-256$4096:DFoZHU/DXsHL8MJ8regdEw==$gx9sUGgpVpdSM4o6A2R9PKAUkAsRPLhLoBDLBUYtKS0=:MujSgKe6rxcIUMv4GnyXJmV0YNbf39uFRZv724+X1FE="
"dbuser_monitor" "SCRAM-SHA-256$4096:fwU97ZMO/KR0ScHO5+UuBg==$CrNsmGrx1DkIGrtrD1Wjexb/aygzqQdirTO1oBZROPY=:L8+dJ+fqlMQh7y4PmVR/gbAOvYWOr+KINjeMZ8LlFww="
"dbuser_meta" "SCRAM-SHA-256$4096:leB2RQPcw1OIiRnPnOMUEg==$eyC+NIMKeoTxshJu314+BmbMFpCcspzI3UFZ1RYfNyU=:fJgXcykVPvOfro2MWNkl5q38oz21nSl1dTtM65uYR1Q="

User-level pool params are maintained in /etc/pgbouncer/useropts.txt:

dbuser_dba      = pool_mode=session max_user_connections=16
dbuser_monitor  = pool_mode=session max_user_connections=8

When you create users, Pigsty reloads the Pgbouncer user list online. The reload does not affect existing connections.

Pgbouncer runs as the same dbsu as PostgreSQL. The default is the postgres operating system user. Use the pgb alias to access Pgbouncer administration functions.

Use pgbouncer_auth_query to query authentication data dynamically for pool users. This parameter removes the need to manage pool users manually.


For user management operations, see User Management.

For user access privileges, see Access Control: Role System.

1.5 - Database

How to define and customize PostgreSQL databases through configuration?

In this document, “database” refers to a logical object within a database cluster created with CREATE DATABASE.

A PostgreSQL cluster can serve multiple databases at the same time. Define the required databases in the Pigsty cluster configuration.

Pigsty customizes the template1 database with default schemas, extensions, and privileges. New databases inherit these settings from template1. Use template to select another template for instant database cloning.

By default, Pigsty adds each application database to the PgBouncer connection pool with a 1:1 mapping. pg_exporter discovers application databases automatically to monitor their objects. Pigsty also registers all databases as PostgreSQL data sources in Grafana on every INFRA node for PGCAT dashboards.


Define Database

Define application databases in pg_databases, an array of database definition objects. During cluster initialization, Pigsty creates databases in definition order. A later database can therefore use an earlier database as a template.

Example from Pigsty demo pg-meta cluster:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
      - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }

Each database definition has the fields below. Only name is required:

- name: meta                      # REQUIRED, `name` is the only mandatory field
  state: create                   # Optional, database state: create (default), absent, recreate
  baseline: cmdb.sql              # Optional, SQL baseline file path (relative to Ansible search path, e.g., files/)
  pgbouncer: true                 # Optional, add to pgbouncer database list? default true
  schemas: [pigsty]               # Optional, additional schemas to create, array of schema names
  extensions:                     # Optional, extensions to install: array of extension objects
    - { name: postgis , schema: public }  # Can specify schema, or omit (installs to first schema in search_path)
    - { name: timescaledb }               # Some extensions create and use fixed schemas
  comment: pigsty meta database   # Optional, database comment/description
  owner: postgres                 # Optional, database owner, defaults to current user
  template: template1             # Optional, template to use, default template1
  strategy: FILE_COPY             # Optional, clone strategy: FILE_COPY or WAL_LOG (PG15+)
  encoding: UTF8                  # Optional, inherits from template/cluster config (UTF8)
  locale: C                       # Optional, inherits from template/cluster config (C)
  lc_collate: C                   # Optional, inherits from template/cluster config (C)
  lc_ctype: C                     # Optional, inherits from template/cluster config (C)
  locale_provider: libc           # Optional, locale provider: libc, icu, builtin (PG15+)
  icu_locale: en-US               # Optional, ICU locale rules (PG15+)
  icu_rules: ''                   # Optional, ICU collation rules (PG16+)
  builtin_locale: C.UTF-8         # Optional, builtin locale provider rules (PG17+)
  tablespace: pg_default          # Optional, default tablespace
  is_template: false              # Optional, mark as template database
  allowconn: true                 # Optional, allow connections, default true
  revokeconn: false               # Optional; when true, retain CONNECT only for owner, admin, monitor, and replication users
  register_datasource: true       # Optional, register to grafana datasource? default true
  connlimit: -1                   # Optional, connection limit, -1 means unlimited
  parameters:                     # Optional, database-level params via ALTER DATABASE SET
    work_mem: '64MB'
    statement_timeout: '30s'
  pool_auth_user: dbuser_meta     # Optional, auth user for pgbouncer auth_query
  pool_mode: transaction          # Optional, database-level pgbouncer pool mode
  pool_size: 50                   # Optional, database-level pgbouncer default pool size
  pool_reserve: 30                # Optional, database-level pgbouncer reserve pool
  pool_size_min: 0                # Optional, database-level pgbouncer min pool size
  pool_connlimit: 100             # Optional, database-level max database connections

From Pigsty v4.1.0, use pool_reserve and pool_connlimit for database pools. These unify the legacy aliases pool_size_reserve / pool_max_db_conn.


Parameter Overview

Only name is required. It must be a valid database name that is unique within the cluster. All other parameters have defaults. Immutable parameters take effect only at creation. To change them, recreate the database.

FieldCategoryTypeAttrDescription
nameBasicstringRequiredDatabase name, must be valid and unique
stateBasicenumOptionalState: create (default), absent, recreate
ownerBasicstringMutableDatabase owner, defaults to postgres
commentBasicstringMutableDatabase comment
templateTemplatestringImmutableTemplate database, default template1
strategyTemplateenumImmutableClone strategy: FILE_COPY or WAL_LOG (PG15+)
encodingEncodingstringImmutableCharacter encoding, default inherited (UTF8)
localeEncodingstringImmutableLocale setting, default inherited (C)
lc_collateEncodingstringImmutableCollation rule, default inherited (C)
lc_ctypeEncodingstringImmutableCharacter classification, default inherited (C)
locale_providerEncodingenumImmutableLocale provider: libc, icu, builtin (PG15+)
icu_localeEncodingstringImmutableICU locale rules (PG15+)
icu_rulesEncodingstringImmutableICU collation customization (PG16+)
builtin_localeEncodingstringImmutableBuiltin locale rules (PG17+)
tablespaceStoragestringMutableDefault tablespace, change triggers data migration
is_templatePrivilegeboolMutableMark as template database
allowconnPrivilegeboolMutableAllow connections, default true
revokeconnPrivilegeboolMutableRevoke PUBLIC CONNECT privilege
connlimitPrivilegeintMutableConnection limit, -1 for unlimited
baselineInitstringMutableSQL baseline file path, runs on every provisioning
schemasInit(string|object)[]MutableSchema definitions to create
extensionsInit(string|object)[]MutableExtension definitions to install
parametersInitobjectMutableDatabase-level parameters
pgbouncerPoolboolMutableAdd to connection pool, default true
pool_modePoolenumMutablePool mode: transaction (default)
pool_sizePoolintMutableDefault pool size, default 50
pool_size_minPoolintMutableMin pool size, default 0
pool_reservePoolintMutableReserve pool size, default 30
pool_connlimitPoolintMutableMax database connections, default 100
pool_auth_userPoolstringMutableAuth query user
register_datasourceMonitorboolMutableRegister to Grafana datasource, default true

Parameter Details

name

String, required. Database name - must be unique within the cluster.

The current role does not enforce this regular expression. It double-quotes SQL identifiers, but also uses the name in temporary paths and shell/SQL commands. For safe operation throughout the automation, keep the name within 63 bytes and follow ^[A-Za-z_][A-Za-z0-9_$]{0,62}$. Avoid spaces, quotes, slashes, and other special characters.

- name: myapp              # Simple naming
- name: my_application     # Underscore separated
- name: app_v2             # Version included

state

Enum for database operation: create, absent, or recreate. Default create.

StateDescription
createDefault, create or modify database, adjust mutable params if exists
absentDelete database with DROP DATABASE WITH (FORCE)
recreateDrop then create, for database reset
- name: myapp                # state defaults to create
- name: olddb
  state: absent              # Delete database
- name: testdb
  state: recreate            # Rebuild database

owner

String. Database owner, defaults to pg_dbsu (postgres) if not specified.

The target user must exist. To change the owner, Pigsty runs the following commands. The old owner retains existing privileges.

The database owner has full control, including schema, table, and extension creation. This is useful for multi-tenant deployments.

ALTER DATABASE "myapp" OWNER TO "new_owner";
GRANT ALL PRIVILEGES ON DATABASE "myapp" TO "new_owner";

comment

String. Database comment, defaults to business database {name}.

Pigsty uses COMMENT ON DATABASE to set the comment. Comments support Chinese and special characters; Pigsty escapes quotes automatically. PostgreSQL stores them in pg_shdescription, the shared-object comment catalog. Use \l+ to show them.

COMMENT ON DATABASE "myapp" IS 'my main application database';
- name: myapp
  comment: my main application database

template

String, immutable. Template database for creation, default template1.

CREATE DATABASE clones the template, including objects, extensions, schemas, and permissions. Pigsty customizes template1 during cluster initialization. New databases inherit these settings.

TemplateDescription
template1Default, includes Pigsty pre-configured extensions/schemas/perms
template0Clean template, required for non-default locale providers
Custom databaseUse existing database as template for cloning

For the icu or builtin locale provider, specify template: template0. You cannot override the locale settings inherited from template1.

- name: myapp_icu
  template: template0        # Required for ICU
  locale_provider: icu
  icu_locale: zh-Hans

template0 omits the monitoring extensions, schemas, and default privileges. Use it for a fully custom database.

strategy

Enum, immutable. Clone strategy: FILE_COPY or WAL_LOG. Available PG15+.

StrategyDescriptionUse Case
FILE_COPYDirect file copy with checkpoints before and afterLarge templates, lower WAL volume
WAL_LOGBlock-by-block copy written to WAL; PG15+ defaultSmall templates, non-blocking

WAL_LOG permits template connections during cloning but is less efficient for large templates. PG14 and earlier ignore this setting.

- name: cloned_db
  template: source_db
  strategy: WAL_LOG          # WAL-based cloning

encoding

String, immutable. Character encoding, inherits from template if unspecified (usually UTF8).

We strongly recommend UTF8 unless you have special requirements. You cannot change the encoding after creation.

- name: legacy_db
  template: template0        # Use template0 for non-default encoding
  encoding: LATIN1

locale

String, immutable. Locale setting - sets both lc_collate and lc_ctype. Inherits from template (usually C).

This setting controls string sort order and character classification. Use C or POSIX for the best performance and cross-platform consistency. For language-specific sorting, use a locale such as zh_CN.UTF-8.

- name: chinese_db
  template: template0
  locale: zh_CN.UTF-8        # Chinese locale
  encoding: UTF8

lc_collate

String, immutable. String collation rule. Inherits from template (usually C).

This setting controls ORDER BY and comparison results. Common values are C (byte order, fastest), C.UTF-8, en_US.UTF-8, and zh_CN.UTF-8. You cannot change it after creation.

- name: myapp
  template: template0
  lc_collate: en_US.UTF-8    # English collation
  lc_ctype: en_US.UTF-8

lc_ctype

String, immutable. Character classification rule for upper/lower case, digits, letters. Inherits from template (usually C).

This setting affects upper(), lower(), regex \w, and similar operations. You cannot change it after creation.

locale_provider

Enum, immutable. Locale implementation provider: libc, icu, or builtin. Available PG15+, default libc.

ProviderVersionDescription
libc-OS C library, traditional default, varies by system
icuPG15+ICU library, cross-platform consistent, more langs
builtinPG17+PostgreSQL builtin, most efficient, C/C.UTF-8 only

For icu or builtin, use template: template0. Also set the corresponding icu_locale or builtin_locale.

- name: fast_db
  template: template0
  locale_provider: builtin   # Builtin provider, most efficient
  builtin_locale: C.UTF-8

icu_locale

String, immutable. ICU locale identifier. Available PG15+ when locale_provider: icu.

ICU identifiers follow BCP 47. Common values:

ValueDescription
en-USUS English
en-GBBritish English
zh-HansSimplified Chinese
zh-HantTraditional Chinese
ja-JPJapanese
ko-KRKorean
- name: chinese_app
  template: template0
  locale_provider: icu
  icu_locale: zh-Hans        # Simplified Chinese ICU collation
  encoding: UTF8

icu_rules

String, immutable. Custom ICU collation rules. Available PG16+.

Use ICU Collation Customization to adjust the default sort behavior.

- name: custom_sort_db
  template: template0
  locale_provider: icu
  icu_locale: en-US
  icu_rules: '&V << w <<< W'  # Custom V/W sort order

builtin_locale

String, immutable. Builtin locale provider rules. Available PG17+ when locale_provider: builtin. Values: C or C.UTF-8.

PG17 introduced the builtin provider. It is faster than libc and provides consistent behavior across platforms. It supports only C/C.UTF-8 collation.

- name: fast_db
  template: template0
  locale_provider: builtin
  builtin_locale: C.UTF-8    # Builtin UTF-8 support
  encoding: UTF8

tablespace

String, mutable. Default tablespace, default pg_default.

Changing the tablespace moves all database objects to the new tablespace. For large databases, this physical migration can take a long time. Use it carefully.

- name: archive_db
  tablespace: slow_hdd       # Archive data on slow storage
ALTER DATABASE "archive_db" SET TABLESPACE "slow_hdd";

is_template

Boolean, mutable. Mark database as template, default false.

If true, any user with CREATEDB can clone this database as a template. Template databases usually contain predefined schemas, extensions, and data.

- name: app_template
  is_template: true          # Mark as template, allow user cloning
  schemas: [core, api]
  extensions: [postgis, pg_trgm]

Before it deletes an is_template: true database, Pigsty runs ALTER DATABASE ... IS_TEMPLATE false.

allowconn

Boolean, mutable. Allow connections, default true.

If false, the database rejects all connections, including superuser connections. Use this setting for maintenance or archival.

- name: archive_db
  allowconn: false           # Disallow all connections
ALTER DATABASE "archive_db" ALLOW_CONNECTIONS false;

revokeconn

Boolean, mutable. Revoke PUBLIC CONNECT privilege, default false.

When true, Pigsty executes:

  • Revoke PUBLIC CONNECT, regular users can’t connect
  • Grant connect to replication user (replicator) and monitor user (dbuser_monitor)
  • Grant connect to admin user (dbuser_dba) and owner with WITH GRANT OPTION

Setting false restores PUBLIC CONNECT privilege.

- name: secure_db
  owner: dbuser_secure
  revokeconn: true           # Revoke public connect, only specified users

connlimit

Integer, mutable. Max concurrent connections, default -1 (unlimited).

A positive integer limits simultaneous sessions. The limit does not apply to superusers.

- name: limited_db
  connlimit: 50              # Max 50 concurrent connections
ALTER DATABASE "limited_db" CONNECTION LIMIT 50;

baseline

String. SQL baseline file path executed while provisioning the database.

Baseline files usually contain schema definitions, initial data, and stored procedures. The path is relative to the Ansible search path, usually files/.

If baseline is defined, the role runs that file on every provisioning pass for the database. It runs even if the database already exists, and after state: recreate. Make the SQL idempotent, or do not rerun it against an existing database.

- name: myapp
  baseline: myapp_schema.sql  # Looks for files/myapp_schema.sql

schemas

Array, mutable (add/remove). Schema definitions to create or drop. Elements can be strings or objects.

Simple format - strings for schema names (create only):

schemas:
  - app
  - api
  - core

Full format - objects for owner and drop operations:

schemas:
  - name: app                # Schema name (required)
    owner: dbuser_app        # Schema owner (optional), generates AUTHORIZATION clause
  - name: deprecated
    state: absent            # Drop schema (CASCADE)

Creation uses IF NOT EXISTS. Deletion uses CASCADE, which deletes all objects in the schema.

CREATE SCHEMA IF NOT EXISTS "app" AUTHORIZATION "dbuser_app";
DROP SCHEMA IF EXISTS "deprecated" CASCADE;

extensions

Array, mutable (add/remove). Extension definitions to install or uninstall. Elements can be strings or objects.

Simple format - strings for extension names (install only):

extensions:
  - postgis
  - pg_trgm
  - vector

Full format - objects for schema, version, and uninstall:

extensions:
  - name: vector             # Extension name (required)
    schema: public           # Install to schema (optional)
    version: '0.5.1'         # Specific version (optional)
  - name: old_extension
    state: absent            # Uninstall extension (CASCADE)

Installation uses IF NOT EXISTS ... CASCADE. If an extension exists, PostgreSQL reports a NOTICE and skips it. When possible, PostgreSQL installs dependencies automatically. Uninstallation uses CASCADE and deletes dependent objects.

CREATE EXTENSION IF NOT EXISTS "vector" WITH SCHEMA "public" VERSION '0.5.1' CASCADE;
DROP EXTENSION IF EXISTS "old_extension" CASCADE;

parameters

Object, mutable. Database-level config params via ALTER DATABASE ... SET. Applies to all sessions connecting to this database.

- name: analytics
  parameters:
    work_mem: '256MB'
    maintenance_work_mem: '512MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'

Use special value DEFAULT (case-insensitive) to reset to PostgreSQL default:

parameters:
  work_mem: DEFAULT          # Reset to default
  statement_timeout: '30s'   # Set new value
ALTER DATABASE "myapp" SET "work_mem" = DEFAULT;
ALTER DATABASE "myapp" SET "statement_timeout" = '30s';

pgbouncer

Boolean, mutable. Add database to Pgbouncer pool list, default true.

If false, PgBouncer excludes the database and clients cannot access it through the connection pool. Use this for internal management databases or direct connections.

- name: internal_db
  pgbouncer: false           # No connection pool access

pool_mode

Enum, mutable. Pgbouncer pool mode: transaction, session, or statement. Default transaction.

ModeDescriptionUse Case
transactionReturn connection after txnMost OLTP apps, default
sessionReturn connection after sessionApps needing session state
statementReturn after each statementSimple stateless queries
- name: session_app
  pool_mode: session         # Session-level pooling

pool_size

Integer, mutable. Pgbouncer default pool size, default 50.

The pool size limits regular backend connections for this database pool. pool_size_min controls prewarmed connections. Adjust these values for the workload.

- name: high_load_db
  pool_size: 128             # Larger pool for high load

pool_size_min

Integer, mutable. Pgbouncer minimum pool size, default 0.

A value > 0 creates that number of backend connections in advance. This reduces latency for the first request.

- name: latency_sensitive
  pool_size_min: 10          # Pre-warm 10 connections

pool_reserve

Integer, mutable. Pgbouncer reserve pool size, default 30.

If the default pool is full, PgBouncer can add up to pool_reserve connections for increased traffic.

- name: bursty_db
  pool_size: 50
  pool_reserve: 30           # Allow up to 30 more connections after the regular pool is exhausted

pool_connlimit

Integer, mutable. Max connections via Pgbouncer pool, default 100.

This PgBouncer limit is independent of the database connlimit parameter.

- name: limited_pool_db
  pool_connlimit: 50         # Pool max 50 connections

pool_auth_user

String, mutable. User for Pgbouncer auth query.

Enable pgbouncer_auth_query to use this field. If set, all PgBouncer connections to this database use the specified user for auth-query password verification.

- name: myapp
  pool_auth_user: dbuser_monitor  # Use monitor user for auth query

register_datasource

Boolean, mutable. Register database to Grafana as PostgreSQL datasource, default true.

Set false to skip Grafana registration. Use this for temporary, test, or internal databases that do not need monitoring.

- name: temp_db
  register_datasource: false  # Don't register to Grafana

Template Inheritance

If you do not set them explicitly, many parameters inherit from the template database. The default template is template1. Cluster initialization parameters determine its encoding settings:

Cluster ParamDefaultDescription
pg_encodingUTF8Cluster encoding
pg_localeC / C-UTF-8 (if supported)Cluster locale
pg_lc_collateC / C-UTF-8 (if supported)Cluster collation
pg_lc_ctypeC / C-UTF-8 (if supported)Cluster ctype

Unless you select another template, new databases clone template1. PG_PROVISION customizes it with extensions, schemas, and default privileges.


Deep Customization

Use these parameters to customize the template database:

If these parameters are insufficient, use pg_init to specify custom cluster initialization scripts:


Locale Providers

PostgreSQL 15+ introduced locale_provider to select the locale implementation. You cannot change it after database creation.

The Pigsty configure wizard selects the builtin C.UTF-8/C locale provider from the PostgreSQL and OS versions. Databases inherit the cluster locale by default. To select a different provider, use template0.

Using ICU provider (PG15+):

- name: myapp_icu
  template: template0        # ICU requires template0
  locale_provider: icu
  icu_locale: en-US          # ICU locale rules
  encoding: UTF8

Using builtin provider (PG17+):

- name: myapp_builtin
  template: template0
  locale_provider: builtin
  builtin_locale: C.UTF-8    # Builtin locale rules
  encoding: UTF8

Provider comparison: libc (traditional, OS-dependent), icu (PG15+, cross-platform, feature-rich), builtin (PG17+, most efficient C/C.UTF-8).


Connection Pool

PgBouncer improves short-connection performance and reduces contention. It prevents excessive connections from overwhelming the database and provides flexibility during migrations.

Pigsty configures one connection pool for each PostgreSQL instance. It runs as pg_dbsu, which defaults to the postgres OS user. The pool connects to PostgreSQL through the /var/run/postgresql Unix socket.

Pigsty adds all databases in pg_databases to pgbouncer by default. Set pgbouncer: false to exclude specific databases. Pgbouncer database list and config params are defined in /etc/pgbouncer/database.txt:

meta                        = host=/var/run/postgresql mode=session
grafana                     = host=/var/run/postgresql mode=transaction
bytebase                    = host=/var/run/postgresql auth_user=dbuser_meta
kong                        = host=/var/run/postgresql pool_size=32 reserve_pool=64
gitea                       = host=/var/run/postgresql min_pool_size=10
wiki                        = host=/var/run/postgresql
noco                        = host=/var/run/postgresql
mongo                       = host=/var/run/postgresql

When you create databases, Pigsty reloads the PgBouncer database list online. Existing connections remain active.

1.6 - HBA Rules

Configuration reference for PostgreSQL and PgBouncer Host-Based Authentication (HBA) rules in Pigsty.

Overview

HBA (Host-Based Authentication) controls who can connect to a database, from which source, and with which authentication method. For the authentication model and default rules, see Authentication. Pigsty manages HBA rules through pg_default_hba_rules and pg_hba_rules.

Pigsty generates these files during cluster initialization or HBA refresh:

Config FilePathDescription
PostgreSQL HBA/pg/data/pg_hba.confPostgreSQL server HBA rules
PgBouncer HBA/etc/pgbouncer/pgb_hba.confConnection pool HBA rules

HBA rules are controlled by these parameters:

ParameterLevelDescription
pg_default_hba_rulesGPostgreSQL global default HBA
pg_hba_rulesG/C/IPostgreSQL cluster/instance add
pgb_default_hba_rulesGPgBouncer global default HBA
pgb_hba_rulesG/C/IPgBouncer cluster/instance add

Rule features:

  • Role filtering: Rules support role field, auto-filter based on instance’s pg_role
  • Order sorting: Rules support order field, controls position in final config file
  • Two syntaxes: Supports alias form (simplified) and raw form (direct HBA text)

Refresh HBA

After you change the inventory, generate the configuration files again and reload the services:

bin/pgsql-hba <cls>                   # Refresh entire cluster HBA (recommended)
bin/pgsql-hba <cls> <ip>...           # Refresh specific instances in cluster

Script executes the following playbook:

./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true

PostgreSQL only: ./pgsql.yml -l <cls> -t pg_hba,pg_reload -e pg_reload=true

PgBouncer only: ./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload

Don’t edit config files directly

Do not edit /pg/data/pg_hba.conf or /etc/pgbouncer/pgb_hba.conf directly. The next playbook run will overwrite these files. Make changes in pigsty.yml, then run bin/pgsql-hba to refresh them.


Parameter Details

pg_default_hba_rules

This global PostgreSQL HBA rule list provides base access control for all clusters. It is usually defined in all.vars.

  • Type: rule[], Level: Global (G)
pg_default_hba_rules:
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

pg_hba_rules

This list adds PostgreSQL HBA rules. You can override it at cluster or instance level. Pigsty merges it with the default rules and sorts the result by order.

  • Type: rule[], Level: Global/Cluster/Instance (G/C/I), Default: []
pg_hba_rules:
  - {user: app_user, db: app_db, addr: intra, auth: pwd, title: 'app user access'}

pgb_default_hba_rules

This is the global default PgBouncer HBA rule list. It is usually defined in all.vars.

  • Type: rule[], Level: Global (G)
pgb_default_hba_rules:
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

pgb_hba_rules

PgBouncer cluster/instance-level additional HBA rules.

  • Type: rule[], Level: Global/Cluster/Instance (G/C/I), Default: []

Note: PgBouncer HBA does not support db: replication.


Rule Fields

Each HBA rule is a YAML dict supporting these fields:

FieldTypeRequiredDefaultDescription
userstringNoallUsername, supports all, placeholders, +rolename
dbstringNoallDatabase name, supports all, replication, db name
addrstringYes*-Address alias or CIDR, see Address Aliases
authstringNopwdAuth method alias, see Auth Methods
titlestringNo-Rule description, rendered as comment in config
rolestringNocommonInstance role filter, see Role Filtering
orderintNo1000Sort weight, lower first, see Order Sorting
ruleslistYes*-Raw HBA text lines, mutually exclusive with addr

Either addr or rules must be specified. Use rules to write raw HBA format directly.


Address Aliases

Pigsty provides address aliases to simplify HBA rule writing:

AliasExpands ToDescription
localUnix socketLocal Unix socket
localhostUnix socket + 127.0.0.1/32 + ::1/128Loopback addresses
admin${admin_ip}/32Admin IP address
infraAll infra group node IPsInfrastructure nodes
clusterAll current cluster member IPsSame cluster instances
intra / intranet10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16Intranet CIDRs
world / all0.0.0.0/0 + ::/0Any address (IPv4 + IPv6)
<CIDR>Direct usee.g., 192.168.1.0/24

Intranet CIDRs can be customized via node_firewall_intranet:

node_firewall_intranet:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

Auth Methods

Pigsty provides auth method aliases for simplified config:

AliasActual MethodConnection TypeDescription
pwdscram-sha-256 or md5hostAuto-select based on pg_pwd_enc
sslscram-sha-256 or md5hostsslForce SSL + password
ssl-shascram-sha-256hostsslForce SSL + SCRAM-SHA-256
ssl-md5md5hostsslForce SSL + MD5
certcerthostsslClient certificate auth
trusttrusthostUnconditional trust (dangerous)
deny / rejectrejecthostReject connection
identidenthostOS user mapping (PostgreSQL)
peerpeerlocalOS user mapping (PgBouncer/local)

pg_pwd_enc defaults to scram-sha-256. For legacy client compatibility, you can set it to md5.


User Variables

During configuration generation, Pigsty replaces these user placeholders with the actual usernames:

PlaceholderDefaultCorresponding Param
${dbsu}postgrespg_dbsu
${repl}replicatorpg_replication_username
${monitor}dbuser_monitorpg_monitor_username
${admin}dbuser_dbapg_admin_username

Role Filtering

The role field in HBA rules controls which instances the rule applies to:

RoleDescription
commonDefault, applies to all instances
primaryPrimary instance only
replicaReplica instance only
offlineOffline instance only (pg_role: offline or pg_offline_query: true)
standbyStandby instance
delayedDelayed replica instance

Role filtering uses the instance pg_role variable. Pigsty comments out non-matching rules with the prefix #.

pg_hba_rules:
  # Only applies on primary: writer can only connect to primary
  - {user: writer, db: all, addr: intra, auth: pwd, role: primary, title: 'writer only on primary'}

  # Only applies on offline instances: ETL dedicated network
  - {user: '+dbrole_offline', db: all, addr: '172.20.0.0/16', auth: ssl, role: offline, title: 'offline dedicated'}

Order Sorting

PostgreSQL uses the first matching HBA rule. Rule order therefore affects authentication. Pigsty uses order to sort the generated rules.

Order Interval Convention

IntervalUsage
0 - 99User high-priority rules (before all defaults)
100 - 650Default rule zone (spaced by 50 for insertion)
1000+User rule default (rules without order)

PostgreSQL Default Rules Order

OrderRule Description
100dbsu local ident
150dbsu replication local
200replicator localhost
250replicator intra replication
300replicator intra postgres
350monitor localhost
400monitor infra
450admin infra ssl
500admin world ssl
550dbrole_readonly localhost
600dbrole_readonly intra
650dbrole_offline intra

PgBouncer Default Rules Order

OrderRule Description
100dbsu local peer
150all localhost pwd
200monitor pgbouncer intra
250monitor world deny
300admin intra pwd
350admin world deny
400all intra pwd

Syntax Examples

Alias Form: Using Pigsty simplified syntax

pg_hba_rules:
  - title: allow grafana view access
    role: primary
    user: dbuser_view
    db: meta
    addr: infra
    auth: ssl

Rendered result:

# allow grafana view access [primary]
hostssl  meta               dbuser_view        10.10.10.10/32     scram-sha-256

Raw Form: Using PostgreSQL HBA syntax directly

pg_hba_rules:
  - title: allow intranet password access
    role: common
    rules:
      - host all all 10.0.0.0/8 scram-sha-256
      - host all all 172.16.0.0/12 scram-sha-256
      - host all all 192.168.0.0/16 scram-sha-256

Rendered result:

# allow intranet password access [common]
host all all 10.0.0.0/8 scram-sha-256
host all all 172.16.0.0/12 scram-sha-256
host all all 192.168.0.0/16 scram-sha-256

Common Scenarios

Blacklist IP: Use order: 0 to ensure first match

pg_hba_rules:
  - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'block bad ip'}

Whitelist App Server: High priority for specific IP

pg_hba_rules:
  - {user: app_user, db: app_db, addr: '192.168.1.10/32', auth: ssl, order: 50, title: 'app server'}

Admin Force Certificate: Override default SSL password auth

pg_hba_rules:
  - {user: '${admin}', db: all, addr: world, auth: cert, order: 10, title: 'admin cert only'}

Offline Instance Dedicated Network: Only on offline instances

pg_hba_rules:
  - {user: '+dbrole_offline', db: all, addr: '172.20.0.0/16', auth: ssl-sha, role: offline, title: 'etl network'}

Restrict Access by Database: Sensitive databases limited to specific networks

pg_hba_rules:
  - {user: fin_user, db: finance_db, addr: '10.20.0.0/16', auth: ssl, title: 'finance only'}
  - {user: hr_user, db: hr_db, addr: '10.30.0.0/16', auth: ssl, title: 'hr only'}

PgBouncer Dedicated Rules: Note no db: replication support

pgb_hba_rules:
  - {user: '+dbrole_readwrite', db: all, addr: world, auth: ssl, title: 'app via pgbouncer'}

Complete Cluster Example

pg-prod:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary}
    10.10.10.12: {pg_seq: 2, pg_role: replica}
    10.10.10.13: {pg_seq: 3, pg_role: offline}
  vars:
    pg_cluster: pg-prod

    pg_hba_rules:
      # Blacklist: known malicious IP (highest priority)
      - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'blacklist'}

      # App server whitelist (high priority)
      - {user: app_user, db: app_db, addr: '192.168.1.0/24', auth: ssl, order: 50, title: 'app servers'}

      # ETL tasks: offline instances only
      - {user: etl_user, db: all, addr: '172.20.0.0/16', auth: pwd, role: offline, title: 'etl tasks'}

      # Cluster internal monitoring
      - {user: '${monitor}', db: all, addr: cluster, auth: pwd, order: 380, title: 'cluster monitor'}

    pgb_hba_rules:
      # App via connection pool
      - {user: '+dbrole_readwrite', db: all, addr: '192.168.1.0/24', auth: ssl, title: 'app via pgbouncer'}

Verification & Troubleshooting

View Current HBA Rules

psql -c "TABLE pg_hba_file_rules"         # View via SQL (recommended)
cat /pg/data/pg_hba.conf                  # View PostgreSQL HBA file
cat /etc/pgbouncer/pgb_hba.conf           # View PgBouncer HBA file
grep '^#' /pg/data/pg_hba.conf | head -20 # View rule titles (verify order)

Test Connection Auth

psql -h <host> -p 5432 -U <user> -d <db> -c "SELECT 1"

Common Issues

Error MessagePossible CauseSolution
no pg_hba.conf entry for host...No matching HBA ruleAdd corresponding rule and refresh
password authentication failedWrong password or encCheck password and pg_pwd_enc
Rule not taking effectNot refreshed or orderRun bin/pgsql-hba, check order

Important Notes

  1. Order sensitive: PostgreSQL HBA is first-match-wins, use order wisely
  2. Role matching: Ensure role field matches target instance’s pg_role
  3. Address format: CIDR must be correct, e.g., 10.0.0.0/8 not 10.0.0.0/255.0.0.0
  4. PgBouncer limitation: Does not support db: replication
  5. TLS prerequisite: ssl and cert require server-side TLS; clients must still use verify-full to authenticate the server
  6. Test first: Validate in test environment before modifying HBA
  7. Refresh on scale: Rules using addr: cluster need refresh after cluster membership changes

1.7 - Access Control

Configuration reference for Pigsty built-in roles, users, default privileges, and database ACLs.

Access control combines roles, object privileges, database ACLs, and HBA. This page describes configuration parameters. For design and boundaries, see Access Control Concepts.

Pigsty provides a compact ACL model described by these parameters:

  • pg_default_roles: system roles and system users.
  • pg_users: application users and roles.
  • pg_default_privileges: default privileges on objects created by managed administrators and owners.
  • pg_revoke_public, pg_default_schemas, and pg_default_extensions: default behavior for template1.

Manage these parameters together with HBA and database definitions to produce reproducible access-control configuration.


Default Role System (pg_default_roles)

The defaults contain four business roles and four system users:

NameTypeDescription
dbrole_readonlyNOLOGINShared read-only role with SELECT and USAGE
dbrole_readwriteNOLOGINInherits read-only and adds INSERT, UPDATE, and DELETE
dbrole_adminNOLOGINInherits pg_monitor and read-write; can create objects and triggers
dbrole_offlineNOLOGINIndependent read-only role; instance scope must be restricted explicitly through HBA
postgresUserSystem superuser; same name as pg_dbsu
replicatorUserStreaming replication and backup; inherits monitoring and read-only privileges
dbuser_dbaUserPrimary administration account, also synchronized to PgBouncer
dbuser_monitorUserMonitoring account with pg_monitor; records slow SQL by default

pg_default_roles contains these definitions as a complete list. To customize it, copy and retain the required default roles and system users. Then add new roles in dependency order. If a role name changes, update its references in HBA, default privileges, and scripts.


Default Users and Credential Parameters

These parameters control system-user names and passwords:

ParameterDefaultPurpose
pg_dbsupostgresDatabase and OS superuser
pg_dbsu_passwordEmpty stringdbsu password, disabled by default
pg_replication_usernamereplicatorReplication user name
pg_replication_passwordDBUser.ReplicatorReplication password
pg_admin_usernamedbuser_dbaAdministrator user name
pg_admin_passwordDBUser.DBAAdministrator password
pg_monitor_usernamedbuser_monitorMonitoring user
pg_monitor_passwordDBUser.MonitorMonitoring password

After changing these parameters, update the corresponding user definitions in pg_default_roles so user names and role attributes remain consistent.


Application Roles and Grants (pg_users)

Declare application users with pg_users. The roles field grants application roles. For field details, see User Configuration.

Example read-only and read-write users:

pg_users:
  - { name: app_reader,  password: DBUser.Reader,  roles: [dbrole_readonly],  pgbouncer: true }
  - { name: app_writer,  password: DBUser.Writer,  roles: [dbrole_readwrite], pgbouncer: true }

Application users inherit default object privileges through dbrole_*. Database CONNECT privileges and pg_hba_rules also control which databases and sources can connect.

For more specific ACLs, use standard GRANT and REVOKE in baseline SQL or a later playbook. Include these additional grants in reviews.


Default Privilege Template (pg_default_privileges)

pg_default_privileges applies to pg_dbsu, pg_admin_username, dbrole_admin, and every declared database owner. The default template is:

pg_default_privileges:
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

Objects created by these identities receive the corresponding privileges automatically. Other object creators need their own ALTER DEFAULT PRIVILEGES configuration.

Additional notes:

  • pg_revoke_public defaults to true, revoking CREATE from PUBLIC on databases and the public schema.
  • pg_default_schemas and pg_default_extensions control schemas and extensions created in template1/postgres, usually for monitoring objects such as the monitor schema and pg_stat_statements.

Common Scenarios

Read-only Account for a Partner

pg_users:
  - name: partner_ro
    password: Partner.Read
    roles: [dbrole_readonly]
pg_hba_rules:
  - { user: partner_ro, db: analytics, addr: 203.0.113.0/24, auth: ssl }

This adds an HBA rule for partner access to analytics over TLS from the specified CIDR. pg_hba_rules does not remove broader default rules. If the account must access only this database, narrow the default HBA policy and configure database CONNECT privileges.

DDL for an Application Administrator

pg_users:
  - name: app_admin
    password: DBUser.AppAdmin
    roles: [dbrole_admin]

app_admin inherits DDL privileges from dbrole_admin. To apply the dbrole_admin default privileges to new objects, first run SET ROLE dbrole_admin. If app_admin is a declared database owner, it can create objects directly as that owner.

Custom Default Privileges

pg_default_privileges:
  - GRANT INSERT,UPDATE,DELETE ON TABLES TO dbrole_admin
  - GRANT SELECT,UPDATE ON SEQUENCES TO dbrole_admin
  - GRANT SELECT ON TABLES TO reporting_group

This parameter replaces the complete default privilege list. Referenced roles must already exist. Changes affect only objects created afterward. Grant privileges separately on existing objects.


Integration with Other Components

  • HBA rules: use pg_hba_rules to bind roles, databases, and sources. To restrict dbrole_offline, set role: offline on its rule.
  • PgBouncer: users with pgbouncer: true are written to userlist.txt; pool_mode and pool_connlimit control pool-level quotas.
  • Database monitoring: dbuser_monitor receives privileges from pg_default_roles. When adding another monitoring user, grant pg_monitor and check access to the monitor schema.

You can version these parameters with the inventory. Continue to review effective privileges through PostgreSQL catalogs.


1.8 - Parameters

Configure PostgreSQL parameters at cluster, instance, database, and user levels

You can set PostgreSQL parameters at several levels with different scopes and priorities. Pigsty supports four levels, from global to local:

LevelScopeConfiguration MethodStorage Location
ClusterAll instances in clusterPatroni DCS / Tuning Templatesetcd + postgresql.conf
InstanceSingle PG instancepg_parameters / ALTER SYSTEMpostgresql.auto.conf
DatabaseAll sessions in a DBpg_databases[].parameterspg_db_role_setting
UserAll sessions of a userpg_users[].parameterspg_db_role_setting

Priority from low to high: Cluster < Instance < Database < User < Session (SET command). Higher priority settings override lower ones.

For complete PostgreSQL parameter documentation, see PostgreSQL Docs: Server Configuration.


Cluster Level

Cluster-level parameters apply to every primary and replica in a PostgreSQL cluster. Pigsty manages them through Patroni and stores them in DCS, which is etcd by default.

Pigsty provides four Patroni tuning templates for different workloads. Select one with pg_conf:

TemplateUse CaseCharacteristics
oltp.ymlOLTP transactionsLow latency, high concurrency (default)
olap.ymlOLAP analyticsLarge queries, high throughput
crit.ymlCritical/FinancialMax durability, safety over perf
tiny.ymlTiny instancesResource-constrained, dev/test

Template files are in roles/pgsql/templates/. They calculate parameter values from the hardware specifications. During cluster initialization, Pigsty generates /etc/patroni/patroni.yml from the template. See Tuning Templates.

Before cluster creation, adjust the templates to change initial parameters. After initialization, change parameters through Patroni configuration management.

Patroni DCS Config

Patroni stores cluster configuration in DCS, which is etcd by default. This keeps all members configured consistently.

Storage Structure:

/pigsty/                          # namespace (patroni_namespace)
  └── pg-meta/                    # cluster name (pg_cluster)
      ├── config                  # cluster config (shared)
      ├── leader                  # current primary info
      ├── members/                # member registration
      │   ├── pg-meta-1
      │   └── pg-meta-2
      └── ...

Rendering Flow:

  1. Init: Template (e.g., oltp.yml) rendered via Jinja2 to /etc/patroni/patroni.yml
  2. Start: Patroni reads local config, writes PostgreSQL parameters to DCS
  3. Runtime: Patroni periodically syncs DCS config to local PostgreSQL

Local Cache:

Each Patroni instance caches DCS config locally at /pg/conf/<instance>.yml:

  • On start: Load from DCS, cache locally
  • Runtime: Periodically sync DCS to local cache
  • DCS unavailable: Continue with local cache (no failover possible)

Config File Hierarchy

Patroni renders DCS config to local PostgreSQL config files:

/pg/data/
├── postgresql.conf          # Main config (managed by Patroni)
├── postgresql.base.conf     # Base config (via include directive)
├── postgresql.auto.conf     # Instance overrides (ALTER SYSTEM)
├── pg_hba.conf              # Client auth config
└── pg_ident.conf            # User mapping config

Load Order (priority low to high):

  1. postgresql.conf: Dynamically generated by Patroni with DCS cluster params
  2. postgresql.base.conf: Loaded via include, static base config
  3. postgresql.auto.conf: Auto-loaded by PostgreSQL, instance overrides

Since postgresql.auto.conf loads last, its parameters override earlier files.


Instance Level

Instance-level parameters apply to one PostgreSQL instance and override cluster-level configuration. Pigsty writes them to postgresql.auto.conf. This file loads last and can override any cluster parameter.

Use instance parameters for settings such as:

  • Set hot_standby_feedback = on on replicas
  • Adjust work_mem or maintenance_work_mem for specific instances
  • Set recovery_min_apply_delay for delayed replicas

Using pg_parameters

In Pigsty config, use pg_parameters to define instance-level parameters:

pg-meta:
  hosts:
    10.10.10.10:
      pg_seq: 1
      pg_role: primary
      pg_parameters:                              # instance-level params
        log_statement: all                        # log all SQL for this instance only
  vars:
    pg_cluster: pg-meta
    pg_parameters:                                # cluster default instance params
      log_timezone: Asia/Shanghai
      log_min_duration_statement: 1000

Run ./pgsql.yml -l <cls> -t pg_param to write these parameters to postgresql.auto.conf.

Override Hierarchy

pg_parameters can be defined at different Ansible config levels, priority low to high:

all:
  vars:
    pg_parameters:                    # global default
      log_statement: none

  children:
    pg-meta:
      vars:
        pg_parameters:                # cluster override
          log_statement: ddl
      hosts:
        10.10.10.10:
          pg_parameters:              # instance override (highest)
            log_statement: all

Using ALTER SYSTEM

You can also modify instance parameters at runtime via ALTER SYSTEM:

-- Set parameters
ALTER SYSTEM SET work_mem = '256MB';
ALTER SYSTEM SET log_min_duration_statement = 1000;

-- Reset to default
ALTER SYSTEM RESET work_mem;
ALTER SYSTEM RESET ALL;  -- Reset all ALTER SYSTEM settings

-- Reload config to take effect
SELECT pg_reload_conf();

ALTER SYSTEM writes to postgresql.auto.conf.

Note: In Pigsty-managed clusters, postgresql.auto.conf is managed by Ansible via pg_parameters. Manual ALTER SYSTEM changes may be overwritten on next playbook run. Use pg_parameters in pigsty.yml for persistent instance-level params.

List-Type Parameters

Some PostgreSQL parameters accept comma-separated lists. In YAML, quote the entire value. Otherwise, YAML parses it as an array:

# Correct: quote the entire value
pg_parameters:
  shared_preload_libraries: 'timescaledb, pg_stat_statements'
  search_path: '"$user", public, app'

# Wrong: unquoted causes YAML parse error
pg_parameters:
  shared_preload_libraries: timescaledb, pg_stat_statements   # YAML parses as array!

Pigsty auto-detects these list parameters and renders them without outer quotes:

ParameterDescriptionExample Value
shared_preload_librariesPreload shared libs'timescaledb, pg_stat_statements'
search_pathSchema search path'"$user", public, app'
local_preload_librariesLocal preload libs'auto_explain'
session_preload_librariesSession preload libs'pg_hint_plan'
log_destinationLog output targets'csvlog, stderr'
unix_socket_directoriesUnix socket dirs'/var/run/postgresql, /tmp'
temp_tablespacesTemp tablespaces'ssd_space, hdd_space'
debug_io_directDirect I/O mode (PG16+)'data, wal'

Rendering Example:

# pigsty.yml config (quotes required in YAML)
pg_parameters:
  shared_preload_libraries: 'timescaledb, pg_stat_statements'
  search_path: '"$user", public, app'
  work_mem: 64MB
# Rendered postgresql.auto.conf (list params unquoted)
shared_preload_libraries = timescaledb, pg_stat_statements
search_path = "$user", public, app
work_mem = '64MB'

Database Level

Database-level parameters apply to all sessions connected to a specific database. ALTER DATABASE ... SET sets them in pg_db_role_setting.

Configuration

Use the parameters field in pg_databases:

pg_databases:
  - name: analytics
    owner: dbuser_analyst
    parameters:
      work_mem: 256MB                              # analytics needs more memory
      maintenance_work_mem: 1GB                    # large table maintenance
      statement_timeout: 10min                     # allow long queries
      search_path: '"$user", public, mart'         # list param needs quotes

Like instance-level params, list-type values must be quoted in YAML.

Rendering Rules

Pigsty sets database parameters with ALTER DATABASE ... SET. It selects the required syntax automatically:

List-type params (search_path, temp_tablespaces, local_preload_libraries, session_preload_libraries, log_destination) without outer quotes:

ALTER DATABASE "analytics" SET "search_path" = "$user", public, mart;

Scalar params with quoted values:

ALTER DATABASE "analytics" SET "work_mem" = '256MB';
ALTER DATABASE "analytics" SET "statement_timeout" = '10min';

Note: log_destination is in the database whitelist, but its context is sighup. It cannot take effect at database level. Set it at instance level with pg_parameters.

View Database Params

-- View params for a specific database
SELECT datname, unnest(setconfig) AS setting
FROM pg_db_role_setting drs
JOIN pg_database d ON d.oid = drs.setdatabase
WHERE drs.setrole = 0 AND datname = 'analytics';

Manual Management

-- Set params
ALTER DATABASE analytics SET work_mem = '256MB';
ALTER DATABASE analytics SET search_path = "$user", public, myschema;

-- Reset params
ALTER DATABASE analytics RESET work_mem;
ALTER DATABASE analytics RESET ALL;

User Level

User-level parameters apply to all sessions for a specific database user. ALTER USER ... SET sets them in pg_db_role_setting.

Configuration

Use the parameters field in pg_users or pg_default_roles:

pg_users:
  - name: dbuser_analyst
    password: DBUser.Analyst
    parameters:
      work_mem: 256MB                              # more memory for analytics
      statement_timeout: 5min                      # allow longer queries
      search_path: '"$user", public, analytics'    # list param needs quotes
      log_statement: all                           # log all SQL

Rendering Rules

Same as database-level:

List-type params (search_path, temp_tablespaces, local_preload_libraries, session_preload_libraries) without outer quotes:

ALTER USER "dbuser_analyst" SET "search_path" = "$user", public, analytics;

Scalar params with quoted values:

ALTER USER "dbuser_analyst" SET "work_mem" = '256MB';
ALTER USER "dbuser_analyst" SET "statement_timeout" = '5min';

DEFAULT Value

Use DEFAULT (case-insensitive) to reset a parameter to PostgreSQL default:

parameters:
  work_mem: DEFAULT          # reset to default
  statement_timeout: 30s     # set specific value
ALTER USER "dbuser_app" SET "work_mem" = DEFAULT;
ALTER USER "dbuser_app" SET "statement_timeout" = '30s';

View User Params

-- View params for a specific user
SELECT rolname, unnest(setconfig) AS setting
FROM pg_db_role_setting drs
JOIN pg_roles r ON r.oid = drs.setrole
WHERE rolname = 'dbuser_analyst';

Manual Management

-- Set params
ALTER USER dbuser_app SET work_mem = '128MB';
ALTER USER dbuser_app SET search_path = "$user", public, myschema;

-- Reset params
ALTER USER dbuser_app RESET work_mem;
ALTER USER dbuser_app RESET ALL;

Priority

When the same parameter is set at multiple levels, PostgreSQL applies this priority (low to high):

postgresql.conf           ← Cluster params (Patroni DCS)
       ↓
postgresql.auto.conf      ← Instance params (pg_parameters / ALTER SYSTEM)
       ↓
Database level            ← ALTER DATABASE SET
       ↓
User level                ← ALTER USER SET
       ↓
Session level             ← SET command

Database vs User Priority:

If the same parameter is set for both the database and the connecting user, PostgreSQL uses the user-level parameter. The user setting has higher priority.

Example:

# Database: analytics has work_mem = 256MB
pg_databases:
  - name: analytics
    parameters:
      work_mem: 256MB

# User: analyst has work_mem = 512MB
pg_users:
  - name: analyst
    parameters:
      work_mem: 512MB
  • analyst connecting to analytics: work_mem = 512MB (user takes precedence)
  • Other users connecting to analytics: work_mem = 256MB (database applies)
  • analyst connecting to other DBs: work_mem = 512MB (user applies)

2 - Service/Access

Split read and write operations, route traffic correctly, and reliably deliver PostgreSQL cluster capabilities.

Split read and write operations, route traffic correctly, and reliably deliver PostgreSQL cluster capabilities.

A service exposes database cluster capabilities to clients. It hides the details of the underlying cluster.

Services provide stable access in production, including after automatic failover in high-availability clusters. Single-node users usually do not need services.


Personal User

Services are intended for production environments. Personal users and single-node clusters can connect directly through an instance name or IP address.

For example, Pigsty’s default single-node pg-meta.meta database can be directly connected using three different users:

psql postgres://dbuser_dba:[email protected]/meta     # Direct connection with DBA superuser
psql postgres://dbuser_meta:[email protected]/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

Production environments use primary-replica clusters with replication. Exactly one instance, the primary, leads the cluster and accepts writes. Replicas continuously fetch change logs from the primary to keep their data consistent. Replicas also handle read-only requests, which substantially reduces primary load in workloads with many reads. Thus, separating write requests from read-only requests is common.

For frequent short connections, Pgbouncer pools requests to reduce the cost of connection and backend process creation. Operations such as ETL and database changes need direct connections that bypass the pool. Failures in high-availability clusters can trigger failover and change the primary. Write traffic must follow the new primary automatically. Services represent these access requirements: separate reads and writes, pooled or direct connections, and automatic routing after failover.

Typically, database clusters must provide this most basic service:

  • Read-write service (primary): Can read and write to the database

For production database clusters, at least these two services should be provided:

  • Read-write service (primary): Write data: Only carried by the primary.
  • Read-only service (replica): Read data: Can be carried by replicas, but can also be carried by the primary if no replicas are available

Additionally, depending on specific business scenarios, there might be other services, such as:

  • Default direct access service (default): Service that allows (admin) users to bypass the connection pool and directly access the database
  • Offline replica service (offline): Dedicated replica that doesn’t handle online read-only traffic, used for ETL and analytical queries
  • Synchronous replica service (standby): Read-only service with no replication delay, handled by synchronous standby/primary for read-only queries
  • Delayed replica service (delayed): Access older data from the same cluster from a certain time ago, handled by delayed replicas

Default Service

By default, Pigsty provides four services for each PostgreSQL cluster:

ServicePortDescription
primary5433Production read-write, connect to primary pool (6432)
replica5434Production read-only, connect to replica pool (6432)
default5436Admin, ETL writes, direct access to primary (5432)
offline5438OLAP, ETL, personal users, interactive queries

Taking the default pg-meta cluster as an example, it provides four default services:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # pg-meta-primary : production read-write via primary pgbouncer(6432)
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # pg-meta-replica : production read-only via replica pgbouncer(6432)
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # pg-meta-default : direct connection via primary postgres(5432)
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : direct connection via offline postgres(5432)

From the sample cluster architecture diagram, you can see how these four services work:

pigsty-ha.png

pg_dns_target controls the DNS target of pg-meta. The default, auto, selects the L2 VIP when enabled. Otherwise, it selects the IP address of the inventory primary. VIP is disabled by default. See Accessing Services.


Service Implementation

Pigsty implements services with HAProxy on nodes. Different ports identify different services.

Pigsty enables HAProxy on each managed node, including database nodes. Database nodes have primary or replica roles, but each node exposes the same services. With the correct service port, you can access the primary read-write service through a replica node. Thus, access to any cluster instance gives access to all cluster services.

This design is similar to Kubernetes NodePort services. Each Pigsty service has two core elements:

  1. Access endpoints exposed via NodePort (port number, from where to access?)
  2. Target instances chosen through Selectors (list of instances, who will handle it?)

Pigsty provides services up to the cluster HAProxy endpoint. Clients can reach these load balancers in several ways. See Accessing Services.

Declare all services in configuration files. For example, pg_default_services defines the default PostgreSQL services:

pg_default_services:
- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

Define additional services in pg_services. Both pg_default_services and pg_services are arrays of service definition objects.


Define Service

Pigsty allows you to define your own services:

  • pg_default_services: Services uniformly exposed by all PostgreSQL clusters, with four by default.
  • pg_services: Additional PostgreSQL services, can be defined at global or cluster level as needed.
  • haproxy_services: Directly customize HAProxy service content, can be used for other component access

For PostgreSQL clusters, you usually need only the first two parameters. Each service generates a configuration file on every related HAProxy instance: /etc/haproxy/conf.d/<pg_cluster>-<service>.cfg. The following custom standby service provides read-only access without replication lag. Add its definition to pg_services:

- name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
  port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
  ip: "*"                         # optional, service bind ip address, `*` for all ip by default
  selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
  backup: "[? pg_role == `primary`]"  # optional, backup server selector, these instances will only be used when default selector instances are all down
  dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default, meaning pg_default_service_dest decides
  check: /sync                    # optional, health check url path, / by default, here using Patroni API: /sync, only sync standby and primary will return 200 healthy status
  maxconn: 5000                   # optional, max allowed front-end connection, default 5000
  balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other options: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

For the three-node pg-test example cluster, Pigsty generates /etc/haproxy/conf.d/pg-test-standby.cfg from this definition:

#---------------------------------------------------------------------
# service: pg-test-standby @ 10.10.10.11:5435
#---------------------------------------------------------------------
# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12
# service backups   10.10.10.11
listen pg-test-standby
    bind *:5435            # <--- Binds to port 5435 on all IP addresses
    mode tcp               # <--- Load balancer works on TCP protocol
    maxconn 5000           # <--- Max connections 5000, can be increased as needed
    balance roundrobin     # <--- Load balance algorithm is rr round-robin, can also use leastconn
    option httpchk         # <--- Enable HTTP health check
    option http-keep-alive # <--- Keep HTTP connections
    http-check send meth OPTIONS uri /sync   # <---- Using /sync here, Patroni health check API, only sync standby and primary will return 200 healthy status
    http-check expect status 200             # <---- Health check return code 200 means healthy
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers: selector "[]" includes all three pg-test instances as pg-test-standby backends; /sync admits only the primary and synchronous standby.
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup  # <----- Only primary satisfies condition pg_role == `primary`, selected by backup selector.
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100         #        Therefore acts as fallback instance: normally doesn't serve requests, only serves read-only requests after all other replicas are down, maximizing avoidance of read-write service being affected by read-only service
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100         #

The selector: "[]" expression includes all three pg-test instances in the pg-test-standby backend list. However, the Patroni /sync health check returns HTTP 200 only for the primary and synchronous standby. Only those instances can handle requests. The primary also matches pg_role == primary, so the backup selector marks it as a backup server. It handles requests only when no synchronous standby can do so.


Primary Service

The Primary service provides cluster read-write access on port 5433. It is a critical production service. Its definition is:

- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  • selector: "[]" includes all cluster members in the Primary service.
  • Only the primary passes check: /primary and receives traffic.
  • dest: default makes pg_default_service_dest control the destination.
  • Pigsty replaces the dest value default with pg_default_service_dest, whose default is pgbouncer.
  • Thus, the default destination is the primary connection pool on pgbouncer_port, normally 6432.

Set pg_default_service_dest to postgres to bypass the connection pool. The Primary service then connects directly to pg_port, normally 5432. Use this setting when you do not need pooling.

Example: pg-test-primary haproxy configuration
listen pg-test-primary
    bind *:5433         # <--- primary service defaults to port 5433
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary # <--- primary service defaults to using Patroni RestAPI /primary health check
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

Patroni high availability ensures that no more than one instance passes the /primary health check at a time. Thus, the Primary service always routes traffic to the primary.

The Primary service also protects traffic routing during a split-brain condition. For example, kill -9 can stop primary Patroni without a watchdog. HAProxy sends traffic only when Patroni is alive and reports primary status. This protection is unavailable with a direct database connection.


Replica Service

The Replica service provides read-only access on port 5434. It is the next most important production service after Primary. Its definition is:

- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  • selector: "[]" includes all cluster members in the Replica service.
  • All instances can pass check: /read-only and receive traffic.
  • The backup selector [? pg_role == 'primary' || pg_role == 'offline' ] marks the primary and offline replicas as backups.
  • The primary or offline replicas receive traffic only when all normal replicas are down.
  • dest: default makes pg_default_service_dest control the destination.
  • Pigsty replaces the dest value default with pg_default_service_dest, whose default is pgbouncer, as for the Primary service.
  • Thus, the default destination is the replica connection pool on pgbouncer_port, normally 6432.
Example: pg-test-replica haproxy configuration
listen pg-test-replica
    bind *:5434
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /read-only
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

The Replica service first sends read-only requests to available dedicated replicas. It sends them to the primary only when all replicas are down. For a two-node primary-replica cluster, this means it uses the replica while available and the primary when the replica fails.

The Replica service uses dedicated Offline instances only when all dedicated read-only instances are down. This separates fast online queries from slow offline queries to reduce interference.


Default Service

The Default service provides service on port 5436, and it’s a variant of the Primary service.

The Default service always bypasses the connection pool and connects directly to primary PostgreSQL. Use it for administration connections, ETL writes, and change data capture (CDC).

- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }

If pg_default_service_dest is postgres, Default and Primary have the same behavior. Only their names and ports differ. In this case, you can remove Default from the default services.

Example: pg-test-default haproxy configuration
listen pg-test-default
    bind *:5436         # <--- Except for listening port/target port and service name, other configurations are the same as primary service
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:5432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100

Offline Service

The Offline service listens on port 5438 and connects directly to PostgreSQL without the pool. Use it for slow queries, analytical queries, ETL reads, and personal interactive queries:

- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

Offline routes traffic directly to dedicated offline replicas. It can also use normal read-only instances with the pg_offline_query flag.

  • The selector includes replicas with pg_role = offline and normal read-only instances with pg_offline_query = true.
  • Dedicated offline replicas do not handle Replica service requests by default. This separates fast and slow requests. Flagged normal replicas do handle those requests.
  • The backup selector includes normal replicas without the offline flag. They handle Offline traffic when dedicated or flagged offline instances fail.
  • /replica returns HTTP 200 only for replicas. It rejects the primary, even when the primary is the only remaining cluster instance.
  • Neither selector includes the primary. Thus, Offline traffic never reaches or affects the primary.
Example: pg-test-offline haproxy configuration
listen pg-test-offline
    bind *:5438
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /replica
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup

Offline provides restricted read-only access for personal interactive queries, slow analytical or ETL queries, and long transactions.

The Offline service needs special handling after a switchover. The HAProxy /replica health check automatically rejects the new primary. However, selector uses static pg_role and pg_offline_query inventory labels. In a two-node primary-replica cluster, only the replica may serve Offline queries. A switchover can then leave no eligible backend temporarily. Reloading an unchanged inventory does not add the old primary to the backend list.

First update the inventory labels or pg_offline_query for the new plan, then reload services. Alternatively, switch the primary back.

For a simple business model, you can remove Default and Offline. Use Primary and Replica to connect directly to the database.


Reload Service

Reload services after changes to cluster membership, service definitions, static selector labels, or relative weights. Patroni health checks handle normal Primary and Replica switchovers automatically. These switchovers need no separate reload.

bin/pgsql-svc <cls> [ip...]         # reload service for lb cluster or lb instance
# ./pgsql.yml -t pg_service         # the actual ansible task to reload service

Access Service

Pigsty provides services up to the cluster HAProxy endpoint. Clients can reach these load balancers in several ways.

The typical approach is to use DNS or VIP access, binding to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL services in different ways.

Host

TypeExampleDescription
Cluster Domain Namepg-testAccess via cluster domain name (resolved by dnsmasq @ infra nodes)
Cluster VIP Address10.10.10.3Access via L2 VIP address managed by vip-manager, bound to primary
Instance Hostnamepg-test-1Access via any instance hostname (resolved by dnsmasq @ infra nodes)
Instance IP Address10.10.10.11Access any instance IP address

Port

Pigsty uses different ports to distinguish pg services

PortServiceTypeDescription
5432postgresdatabaseDirect access to postgres server
6432pgbouncermiddlewareGo through connection pool middleware before postgres
5433primaryserviceAccess primary pgbouncer (or postgres)
5434replicaserviceAccess replica pgbouncer (or postgres)
5436defaultserviceAccess primary postgres
5438offlineserviceAccess offline postgres

Combinations

# Access via cluster domain (these examples assume L2 VIP is enabled; otherwise the domain points to the inventory primary by default)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> primary connection pool -> primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://[email protected]:5432/test # L2 VIP -> Primary direct access
postgres://[email protected]:6432/test # L2 VIP -> Primary Connection Pool -> Primary
postgres://[email protected]:5433/test # L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://[email protected]:5434/test # L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://[email protected]:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://[email protected]:5438/test # L2 VIP -> HAProxy -> offline direct connect (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database Instance Direct Connect (singleton access)
postgres://test@pg-test-1:6432/test # DNS -> connection pool -> database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> connection pool -> database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> connection pool -> database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> database direct connect
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> database offline read/write

# Directly specify any cluster instance IP access
postgres://[email protected]:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection Pool -> Database
postgres://[email protected]:5433/test # HAProxy -> connection pool -> database read/write
postgres://[email protected]:5434/test # HAProxy -> connection pool -> database read-only
postgres://[email protected]:5436/test # HAProxy -> Database Direct Connections
postgres://[email protected]:5438/test # HAProxy -> database offline read/write

# Smart client automatic read/write separation
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

Override Service

You can override service defaults in several ways. A common requirement is for Primary and Replica to bypass Pgbouncer and connect directly to PostgreSQL.

Set pg_default_service_dest to postgres. Services with svc.dest='default' then use postgres instead of the default pgbouncer destination.

If Primary already connects to PostgreSQL, Default is redundant. You can remove it.

If you do not need to distinguish personal interactive queries from slow analytical or ETL queries, you can remove Offline. Remove it from pg_default_services.

If replicas do not need to handle online read-only traffic, you can remove Replica from the default list.


Delegate Service

Pigsty exposes PostgreSQL services through HAProxy on nodes. All HAProxy instances in a cluster use the same service definitions.

You can instead delegate PostgreSQL services to a specific node group, such as a dedicated HAProxy load balancer cluster.

Override pg_default_services and set pg_service_provider to the proxy group name.

This example exposes the PostgreSQL Primary service on port 10013 through the proxy HAProxy node group.

pg_service_provider: proxy       # use load balancer on group `proxy` with port 10013
pg_default_services:  [{ name: primary ,port: 10013 ,dest: postgres  ,check: /primary   ,selector: "[]" }]

Use a unique port for each delegated service in the proxy cluster.

A dedicated load balancer cluster example is provided in the 20-node production environment simulation sandbox: conf/ha/simu.yml

3 - PostgreSQL Security

Entry points for PostgreSQL authentication, access control, encrypted communication, data protection, and secure operations.

PostgreSQL security combines authentication, authorization, network boundaries, encrypted communication, data protection, and operating procedures. Pigsty provides configuration parameters for these mechanisms. Operators must still harden, verify, and audit the deployment for its environment.


Concepts and Boundaries

TopicContent
Security and ComplianceDefault state, capability boundaries, and hardening path
AuthenticationHBA, SCRAM, certificate authentication, and credential management
Access ControlBuilt-in roles, default privileges, database ACLs, and instance-access boundaries
Encrypted CommunicationCA, TLS, server authentication, and certificate rotation
Data SecurityPage checksums, replication, backup, PITR, audit, and logging
ComplianceLaunch checks, control mappings, and evidence requirements

Configuration Reference


Administration and Verification

The inventory describes the required state. Acceptance checks should inspect HBA, certificates, listening ports, and sensitive files on running nodes. They should also verify effective roles and privileges through PostgreSQL catalogs.

4 - Administration

Standard Operating Procedures (SOP) for database administration tasks

4.1 - Managing PostgreSQL Clusters

Create/destroy PostgreSQL clusters, scale existing clusters, and clone clusters.

Quick Reference

ActionCommandDescription
Create Clusterbin/pgsql-add <cls>Create a new PostgreSQL cluster
Expand Clusterbin/pgsql-add <cls> <ip...>Add replica to existing cluster
Shrink Clusterbin/pgsql-rm <cls> <ip...>Remove instance from cluster
Remove Clusterbin/pgsql-rm <cls>Destroy entire PostgreSQL cluster
Reload Servicebin/pgsql-svc <cls> [ip...]Reload cluster load balancer config
Reload HBAbin/pgsql-hba <cls> [ip...]Reload cluster HBA access rules
Clone Cluster-Clone via standby cluster or PITR

For other management tasks, see: HA Management, Manage Users, Manage Databases.


Create Cluster

To create a PostgreSQL cluster, first define the cluster in the inventory. Then add nodes and initialize them:

Script
bin/node-add  <cls>     # Add nodes in group <cls>
Playbook
./node.yml  -l <cls>    # Use Ansible playbook to add nodes in group <cls>
Example
bin/node-add pg-test    # Add nodes in pg-test group, runs ./node.yml -l pg-test

Create the cluster on the managed nodes. Run the pgsql.yml playbook on the <cls> group:

Script
bin/pgsql-add <cls>     # Create PostgreSQL cluster <cls>
Playbook
./pgsql.yml -l <cls>    # Use Ansible playbook to create PostgreSQL cluster <cls>
Example
bin/pgsql-add pg-test   # Create pg-test cluster

Example: Create 3-node PG cluster pg-test

demo/pgsql.cast — /demo/pgsql.cast

Risk: Re-running create on existing cluster

If you repeat cluster creation, Pigsty keeps existing data files. However, it overwrites service configurations and restarts the cluster. It also runs any baseline SQL from the database definition again. If that SQL deletes or overwrites data, data loss can occur.


Expand Cluster

To add a replica to an existing PostgreSQL cluster, add its instance definition to the inventory. Use all.children.<cls>.hosts:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary } # existing member
    10.10.10.12: { pg_seq: 2, pg_role: replica } # existing member
    10.10.10.13: { pg_seq: 3, pg_role: replica } # <--- new member
  vars: { pg_cluster: pg-test }

Scale out with a procedure similar to cluster creation. First add the new node to Pigsty:

Script
bin/node-add <ip>       # Add node with IP <ip>
Playbook
./node.yml -l <ip>      # Use Ansible playbook to add node <ip>
Example
bin/node-add 10.10.10.13    # Add node 10.10.10.13, runs ./node.yml -l 10.10.10.13

Then install the PGSQL module on the new node. Use the same pg_cluster value as the existing cluster:

Script
bin/pgsql-add <cls> <ip>  # Add node <ip> to cluster
Playbook
./pgsql.yml -l <ip>       # Core: Use Ansible playbook to install PGSQL module on <ip>
Example
bin/pgsql-add pg-test 10.10.10.13   # Scale out pg-test with node 10.10.10.13

After you scale out, reload the service to add the new member to the load balancer.

Example: Add replica 10.10.10.13 to 2-node cluster pg-test

demo/pgsql-append.cast — /demo/pgsql-append.cast


Shrink Cluster

To remove a replica from an existing PostgreSQL cluster, remove its instance definition. Update all.children.<cls>.hosts in the inventory.

By default, scaling in stops the instance and deletes its data directory. First run pig pg list <cls> and pig pb info. Verify that the target is not the primary and that a recent restorable backup exists. Then have the operator enter the exact <ip>. Run the removal only after confirmation.

First uninstall the PGSQL module from the target node. Run pgsql-rm.yml on <ip>:

Script
bin/pgsql-rm <cls> <ip>   # Remove the PostgreSQL instance on <ip> from cluster <cls>
Playbook
./pgsql-rm.yml -l <ip>    # Directly remove the PostgreSQL instance on <ip> with the Ansible playbook
Example
bin/pgsql-rm pg-test 10.10.10.13  # Remove node 10.10.10.13 from pg-test

After you remove the PGSQL module, you can remove the node from Pigsty:

Script
bin/node-rm <ip>          # Remove <ip> from Pigsty management
Playbook
./node-rm.yml -l <ip>     # Directly remove <ip> from Pigsty management with the Ansible playbook
Example
bin/node-rm 10.10.10.13   # Remove 10.10.10.13 from Pigsty management

After you scale in, remove the instance from the inventory. Then reload the service to remove the instance from the load balancer.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica } # <--- remove after execution
  vars: { pg_cluster: pg-test }

Example: Remove replica 10.10.10.13 from 3-node cluster pg-test

demo/pgsql-shrink.cast — /demo/pgsql-shrink.cast


Remove Cluster

To destroy a cluster, uninstall the PGSQL module from all its nodes. Run pgsql-rm.yml on <cls>:

This operation permanently deletes data. Inspect pig pg list <cls> and pig pb info. Verify a recent backup and any independent copy that you must keep. Have the operator enter the exact cluster name. The following commands destroy that cluster directly.

Script
bin/pgsql-rm <cls>        # Destroy the entire PostgreSQL cluster <cls>
Playbook
./pgsql-rm.yml -l <cls>   # Directly destroy the entire PostgreSQL cluster <cls> with the Ansible playbook
Example
bin/pgsql-rm pg-test      # Destroy cluster pg-test

After you destroy PGSQL, you can remove all its nodes from Pigsty. You can keep nodes that run other services:

Script
bin/node-rm <cls>         # Remove every node in group <cls> from Pigsty management
Playbook
./node-rm.yml -l <cls>    # Directly remove the nodes in group <cls> from Pigsty management with the Ansible playbook
Example
bin/node-rm pg-test       # Remove every node in group pg-test from Pigsty management

After removal, delete the entire cluster definition from inventory.

pg-test: # remove this cluster definition group
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars: { pg_cluster: pg-test }

Example: Destroy 3-node PG cluster pg-test

demo/pgsql-rm.cast — /demo/pgsql-rm.cast

If pg_safeguard is true for the cluster or globally, pgsql-rm.yml stops to prevent accidental removal. To force removal, override this setting on the playbook command line. By default, removal also deletes the cluster backup repository. To keep backups, such as a centralized repository, set pg_rm_backup=false:

./pgsql-rm.yml -l pg-meta -e pg_safeguard=false    # Force removal of protected pg-meta
./pgsql-rm.yml -l pg-meta -e pg_rm_backup=false    # Preserve its backup repository while removing the cluster

Reload Service

PostgreSQL clusters expose services through HAProxy on host nodes. Reload services when service definitions, instance weights, or cluster membership change, including scale-out and scale-in. The reload updates the static HAProxy member configuration. The default Primary and Replica services detect current roles through Patroni REST API health checks. Thus, ordinary switchover or failover redirects traffic automatically. These role changes do not require a new HAProxy configuration.

To reload service configuration, run the pg_service subtask of pgsql.yml. Target <cls> for the whole cluster or <ip> for specific instances:

Script
bin/pgsql-svc <cls>           # Reload service config for entire cluster <cls>
bin/pgsql-svc <cls> <ip...>   # Reload service config for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_service -e pg_reload=true        # Reload entire cluster
./pgsql.yml -l <ip>  -t pg_service -e pg_reload=true        # Reload specific instance
Example
bin/pgsql-svc pg-test                 # Reload pg-test cluster service config
bin/pgsql-svc pg-test 10.10.10.13     # Reload pg-test 10.10.10.13 instance service config

If you use a dedicated load balancer cluster, set pg_service_provider. In this configuration, only a reload on the cluster primary updates the load balancer configuration.

Example: Reload pg-test cluster service config

demo/pgsql-svc.cast — /demo/pgsql-svc.cast

Example: Reload PG Service to Remove Instance

asciicast


Reload HBA

When HBA configuration changes, reload the pg_hba_rules and pgb_hba_rules rules. Some HBA rules depend on inventory roles or address ranges that refer to cluster member aliases. For these rules, reload HBA after you change pg_role labels or scale the cluster. Role selectors use static inventory variables. They do not change automatically after a Patroni switchover.

To reload PostgreSQL and Pgbouncer HBA rules, run the HBA subtasks of pgsql.yml. Target <cls> for the whole cluster or <ip> for specific instances:

Script
bin/pgsql-hba <cls>           # Reload HBA rules for entire cluster <cls>
bin/pgsql-hba <cls> <ip...>   # Reload HBA rules for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true   # Reload entire cluster
./pgsql.yml -l <ip>  -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true   # Reload specific instance
Example
bin/pgsql-hba pg-test                 # Reload pg-test cluster HBA rules
bin/pgsql-hba pg-test 10.10.10.13     # Reload pg-test 10.10.10.13 instance HBA rules

Example: Reload pg-test cluster HBA rules

demo/pgsql-hba.cast — /demo/pgsql-hba.cast


Config Cluster

Patroni manages PostgreSQL configuration parameters. The Patroni configuration template supplies the initial values. After cluster initialization, Patroni dynamically manages and synchronizes the configuration stored in Etcd. Use patronictl to change most Patroni configuration parameters. Use subtasks for other parameters, such as etcd DCS, logging, and REST API configuration. For example, refresh Patroni configuration after etcd cluster membership changes:

./pgsql.yml -l pg-test -t pg_conf                   # Update Patroni config file
ansible pg-test -b -a 'systemctl reload patroni'    # Reload Patroni service

You can override Patroni-managed defaults at different levels: specify params per instance, specify params per user, or specify params per database.


Clone Cluster

You can clone a cluster with a Standby Cluster or Point-in-Time Recovery. A standby cluster is simple and does not require a backup repository. It requires a reachable replication upstream and can copy only the latest state. PITR requires a centralized backup repository, such as Silo. It can clone the cluster to any point within the retention period.

MethodProsConsUse Cases
Standby ClusterNo backup repository neededRequires reachable upstream; latest state onlyDR, read-write separation, migration
PITRRecover to any pointRequires centralized backupUndo mistakes, data audit

Clone via Standby Cluster

A standby cluster continuously synchronizes from the upstream cluster through streaming replication. This is the simplest cloning method. Set pg_upstream on the new cluster primary to pull data automatically from the upstream.

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-test2 is standby cluster (clone) of pg-test
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary, pg_upstream: 10.10.10.11 }  # specify upstream
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-test2 }

Create standby cluster with:

Script
bin/pgsql-add pg-test2    # Create standby cluster, auto-clone from upstream pg-test
Playbook
./pgsql.yml -l pg-test2   # Use Ansible playbook to create standby cluster

The standby cluster follows the upstream and keeps its data synchronized. You can promote it to an independent cluster at any time:

Example: Promote Standby to Independent Cluster

Via Config Cluster, remove standby_cluster config to promote:

$ pg edit-config pg-test2
-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Apply these changes? [y/N]: y

After promotion, pg-test2 becomes independent cluster accepting writes, forked from pg-test.

Example: Change Replication Upstream

If upstream cluster switchover occurs, change standby cluster upstream via Config Cluster:

$ pg edit-config pg-test2

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.11     # <--- old upstream
+  host: 10.10.10.14     # <--- new upstream
   port: 5432

Apply these changes? [y/N]: y

Clone via PITR

Point-in-Time Recovery (PITR) can recover to any point within the backup retention period. It requires a centralized backup repository, such as Silo or S3. It gives more recovery options than a standby cluster.

To clone with PITR, add pg_pitr and specify the recovery target:

# Clone new cluster pg-meta2 from pg-meta backup
pg-meta2:
  hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta2
    pg_pitr:
      cluster: pg-meta                    # Recover from pg-meta backup
      time: '2025-01-10 10:00:00+00'      # Recover to specific time
      archive: false                       # Disable archiving during the independent restore
      action: promote                      # Promote after replay in this one-shot example

Execute clone with pgsql-pitr.yml playbook:

Playbook
./pgsql-pitr.yml -l pg-meta2    # Use the explicitly declared action: promote above
CLI
# Specify PITR options via command line
./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": {"cluster": "pg-meta", "time": "2025-01-10 10:00:00+00", "archive": false, "action": "promote"}}'

PITR supports multiple recovery target types:

Target TypeExampleDescription
Timetime: "2025-01-10 10:00:00+00"Recover to specific timestamp
XIDxid: "250000"Recover to before/after txn
Namename: "before_migration"Recover to named restore point
LSNlsn: "0/4001C80"Recover to specific WAL pos
Latestpg_pitr: {}Recover to end of WAL archive
Post-PITR Processing

Pigsty v4.5 PITR keeps archiving enabled by default (archive: true). If you explicitly set archive: false for exploratory recovery, reset archive_mode, restart the cluster, and perform a new full backup after confirming the recovered data is correct:

psql -c 'ALTER SYSTEM RESET archive_mode;'
pg restart <cls>
pg-backup full    # Execute new full backup

For detailed PITR usage, see Restore Operations documentation.

4.2 - Managing PostgreSQL Users

User management - create, modify, delete users, manage role membership, connection pool config

Quick Start

Pigsty manages users declaratively. First define users in the inventory. Then run bin/pgsql-user <cls> <username> to create or modify them.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users: [{ name: dbuser_app, password: 'DBUser.App', pgbouncer: true }]  # <--- Define user list here!
Script
bin/pgsql-user <cls> <username>    # Create/modify <username> user on <cls> cluster
Playbook
./pgsql-user.yml -l pg-meta -e username=dbuser_app    # Use playbook to create/modify user
Example
bin/pgsql-user pg-meta dbuser_app    # Create/modify dbuser_app user on pg-meta cluster

For the complete user definition reference, see User Configuration. See Access Control for roles and privileges, and Authentication for credential management.

pgsql-user.yml uses name to find a user definition. The playbook does not rename roles. To rename a role, create a replacement role. Migrate ownership, memberships, and client credentials, then validate the change before you remove the old role. Deletion followed by creation is not a lossless rename.

ActionCommandDescription
Create Userbin/pgsql-user <cls> <user>Create new business user or role
Modify Userbin/pgsql-user <cls> <user>Modify existing user properties
Delete Userbin/pgsql-user <cls> <user>Dependency-aware destructive deletion (state: absent)

demo/pgsql-user.cast — /demo/pgsql-user.cast


Create User

During PostgreSQL cluster creation, the pg_user task automatically creates users defined in pg_users.

To create a user on an existing cluster, add its definition to all.children.<cls>.pg_users. Then run:

Script
bin/pgsql-user <cls> <username>   # Create user <username>
Playbook
./pgsql-user.yml -l <cls> -e username=<username>   # Use Ansible playbook
Example
bin/pgsql-user pg-meta dbuser_app    # Create dbuser_app user in pg-meta cluster

Example: Create business user dbuser_app

#all.children.pg-meta.vars.pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true
    roles: [dbrole_readwrite]
    comment: application user for myapp

Result: Pigsty creates dbuser_app on the primary, sets its password, and grants it the dbrole_readwrite role. It adds the user to the Pgbouncer pool and reloads Pgbouncer configuration on all instances.

Recommendation: Use playbook

For manual user creation, you must ensure Pgbouncer user list sync yourself.


Modify User

Use the same command as for creation. The playbook is idempotent. If the target user exists, Pigsty changes its properties to match the configuration.

Script
bin/pgsql-user <cls> <user>   # Modify user <user> properties
Playbook
./pgsql-user.yml -l <cls> -e username=<user>   # Idempotent, can repeat
Example
bin/pgsql-user pg-meta dbuser_app    # Modify dbuser_app to match config

Not directly mutable: name is the identity key in the declarative definition. The playbook does not rename an existing role. Create a replacement role, then migrate ownership, privileges, and clients. Validate the change before you remove the old role.

All other properties can be modified. Common examples:

Modify password: Change the password field. Pigsty temporarily disables logging during the change to prevent password disclosure.

- name: dbuser_app
  password: NewSecretPassword     # New password

Modify privilege attributes: Configure boolean flags for user privileges.

- name: dbuser_app
  superuser: false           # Superuser (use carefully!)
  createdb: true             # Allow CREATE DATABASE
  createrole: false          # Allow CREATE ROLE
  inherit: true              # Auto-inherit role privileges
  replication: false         # Allow streaming replication
  bypassrls: false           # Bypass row-level security
  connlimit: 50              # Connection limit, -1 unlimited

Modify expiration: Use expire_in for expiration after N days, or expire_at for a fixed date. expire_in takes priority. Each playbook run recalculates its expiration date. Use it for temporary users that need periodic renewal.

- name: temp_user
  expire_in: 30                   # Expires in 30 days (relative)

- name: contractor_user
  expire_at: '2024-12-31'         # Expires on date (absolute)

- name: permanent_user
  expire_at: 'infinity'           # Never expires

Modify role membership: Use the roles array with simple or extended definitions. Membership changes are additive: Pigsty keeps existing memberships that the array does not declare. To revoke a membership explicitly, use state: absent.

- name: dbuser_app
  roles:
    - dbrole_readwrite                      # Simple form: grant role
    - { name: dbrole_admin, admin: true }   # With ADMIN OPTION
    - { name: pg_monitor, set: false }      # PG16+: disallow SET ROLE
    - { name: old_role, state: absent }     # Revoke role membership

Manage user parameters: Use the parameters dictionary for user-level parameters. Pigsty generates ALTER USER ... SET commands. Use DEFAULT to reset a parameter.

- name: dbuser_analyst
  parameters:
    work_mem: '256MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'
    log_statement: DEFAULT        # Reset to default

Connection pool config: Set pgbouncer: true to add the user to the pool. You can also set pool_mode and pool_connlimit.

- name: dbuser_app
  pgbouncer: true                 # Add to pool
  pool_mode: transaction          # Pool mode
  pool_connlimit: 50              # Max user connections

Delete User

User deletion terminates sessions, transfers object ownership, revokes grants, and runs DROP ROLE. This operation is irreversible. Confirm the exact cluster, role, successor owner, and a recent backup. Then set the user to state: absent and apply the change.

Script
bin/pgsql-user <cls> <user>   # Actual deletion after confirmation; config must say state: absent
Playbook
./pgsql-user.yml -l <cls> -e username=<user>   # Apply the user definition with state: absent
Example
bin/pgsql-user pg-meta dbuser_old    # Delete dbuser_old (config has state: absent)

Config example:

pg_users:
  - name: dbuser_old
    state: absent

Deletion process: On the primary, the task runs pg-drop-role <user> postgres --force. It disables login and terminates active sessions. It transfers database and tablespace ownership, and objects in each connectable database, to postgres. It runs DROP OWNED to remove grants, revokes role memberships, then runs DROP ROLE. Before changes, it writes an audit snapshot to /tmp/pg_drop_role_<user>_<timestamp>.log.

Protection: The Ansible task skips postgres and the replication, administration, and monitoring usernames configured in the inventory. Direct calls to pg-drop-role protect only the fixed default names: postgres, replicator, dbuser_dba, and dbuser_monitor. The script does not recognize renamed system accounts automatically.

Dependency-aware, not transactional

If REASSIGN OWNED fails in a database, pg-drop-role skips DROP OWNED in that database. However, the procedure across databases is not one transaction. A failure partway through can leave the role NOLOGIN, transfer only some ownership, or leave dependencies. The v4.5 Ansible task also uses ignore_errors. Thus, a successful playbook result does not prove complete deletion. Afterward, verify role absence, successor ownership, application migration, and the audit log.

In v4.5, pgsql-user.yml reloads Pgbouncer. However, it does not reliably remove a deleted role from /etc/pgbouncer/userlist.txt. Check every cluster instance after deletion:

sudo -iu postgres psql -AXtwc "SELECT 1 FROM pg_roles WHERE rolname = 'dbuser_old';"
grep -n '^"dbuser_old"[[:space:]]' /etc/pgbouncer/userlist.txt

If the exact Pgbouncer entry remains, remove only that line through the change control procedure. Reload Pgbouncer and validate application connections. Do not delete entries with a broad pattern.


Manual Deletion

For manual user deletion, use pg-drop-role script directly:

# Check dependencies (read-only)
pg-drop-role dbuser_old --check

# Preview deletion (don't execute)
pg-drop-role dbuser_old --dry-run -v

# Only after confirming a recent backup, exact role, and successor owner
pg-drop-role dbuser_old dbuser_new

# Use --force only after explicitly approving session termination
pg-drop-role dbuser_old dbuser_new --force

Common Use Cases

Common user configuration examples:

Basic business user

- name: dbuser_app
  password: DBUser.App
  pgbouncer: true
  roles: [dbrole_readwrite]
  comment: application user

Read-only user

- name: dbuser_readonly
  password: DBUser.Readonly
  pgbouncer: true
  roles: [dbrole_readonly]

Admin user (can execute DDL)

- name: dbuser_admin
  password: DBUser.Admin
  pgbouncer: true
  pool_mode: session
  roles: [dbrole_admin]
  parameters:
    log_statement: 'all'

Temp user (expires in 30 days)

- name: temp_contractor
  password: TempPassword
  expire_in: 30
  roles: [dbrole_readonly]

Role (no login, for permission grouping)

- name: custom_role
  login: false
  comment: custom role for special permissions

User with advanced role options (PG16+)

- name: dbuser_special
  password: DBUser.Special
  pgbouncer: true
  roles:
    - dbrole_readwrite
    - { name: dbrole_admin, admin: true }
    - { name: pg_monitor, set: false }
    - { name: pg_execute_server_program, inherit: false }

Query Users

Common SQL queries for user info:

List all users

SELECT rolname, rolsuper, rolinherit, rolcreaterole, rolcreatedb,
       rolcanlogin, rolreplication, rolbypassrls, rolconnlimit, rolvaliduntil
FROM pg_roles WHERE rolname NOT LIKE 'pg_%' ORDER BY rolname;

View user role membership

SELECT r.rolname AS member, g.rolname AS role, m.admin_option, m.set_option, m.inherit_option
FROM pg_auth_members m
JOIN pg_roles r ON r.oid = m.member
JOIN pg_roles g ON g.oid = m.roleid
WHERE r.rolname = 'dbuser_app';

View user-level parameters

SELECT rolname, setconfig FROM pg_db_role_setting s
JOIN pg_roles r ON r.oid = s.setrole WHERE s.setdatabase = 0;

View expiring users

SELECT rolname, rolvaliduntil, rolvaliduntil - CURRENT_TIMESTAMP AS time_remaining
FROM pg_roles WHERE rolvaliduntil IS NOT NULL
  AND rolvaliduntil < CURRENT_TIMESTAMP + INTERVAL '30 days'
ORDER BY rolvaliduntil;

Connection Pool Management

Connection pool params in user definitions are applied to Pgbouncer when creating/modifying users.

Pigsty adds users with pgbouncer: true to /etc/pgbouncer/userlist.txt. It sets user-level pool_mode and pool_connlimit in /etc/pgbouncer/useropts.txt.

As the postgres operating system user, use the pgb alias to access the Pgbouncer administration database. For more operations, see Pgbouncer Management.


Manage Default-User Passwords

For a business user, follow Modify User. Save the new password in its pg_users definition, preview the playbook for that user, then apply it. The three default users need extra coordination because other services use their credentials.

ParameterDefaultRoleConsumers
pg_admin_passwordDBUser.DBAdbuser_dbaAdmin clients, Pgbouncer, Infra service files, pgAdmin
pg_monitor_passwordDBUser.Monitordbuser_monitorExporters, Pgbouncer, Grafana data sources
pg_replication_passwordDBUser.ReplicatorreplicatorPatroni replication and .pgpass files

These accounts belong to pg_default_roles, not pg_users. pgsql-user.yml searches only pg_users. Do not change a default password by overriding pg_users on the command line. This changes the business-user list for that run and exposes plaintext passwords in shell history.

Rotate one account at a time:

  1. Save the new parameter in pigsty.yml or the active inventory. Never put the plaintext password on the command line.
  2. On the current primary, open interactive psql as a superuser. Run \password <username> to enter the secret interactively.
  3. Verify the -l cluster or node scope, then run the corresponding refresh playbooks below.
  4. Keep the current administration session open. Verify direct PostgreSQL, Pgbouncer, replication, exporters, and Grafana data sources before you change another account.
# On the current primary of the target cluster
sudo -iu postgres psql -d postgres
\password dbuser_dba       # or dbuser_monitor / replicator

Refresh every consumer for the account. Replace <cls> and constrain infra to the actual targets:

# dbuser_dba: PG-node .pgpass, Pgbouncer, Infra admin files, and pgAdmin files
./pgsql.yml -l <cls> -t pg_pass,pgbouncer_user,pgbouncer_reload -e pg_reload=true
./infra.yml -l infra -t env_pgpass,env_pgscv,env_pgadmin

# dbuser_monitor: PG-node .pgpass, Pgbouncer, exporters, and Grafana data sources
./pgsql.yml -l <cls> -t pg_pass,pgbouncer_user,pgbouncer_reload,pg_exporter,pgbouncer_exporter,add_ds -e pg_reload=true
./infra.yml -l infra -t env_pgpass

# replicator: Patroni configuration plus PostgreSQL-node and Infra .pgpass files
./pgsql.yml -l <cls> -t pg_conf,pg_pass,patroni_reload -e pg_reload=true
./infra.yml -l infra -t env_pgpass

If replication credentials differ between the role and Patroni nodes, new replication connections fail. Change this credential during a maintenance window and validate it promptly. VIBE or another module may have generated an administration connection string in its workspace context. If so, generate that module’s files again.

Check for duplicate Infra .pgpass entries

In v4.5, env_pgpass adds the new line with lineinfile; it does not remove older lines by username. Because libpq uses the first matching line, inspect every target Infra node after the refresh and remove obsolete entries through controlled editing without printing secrets:

awk -F: '$4=="dbuser_dba" || $4=="dbuser_monitor" || $4=="replicator" {print NR, $4}' ~/.pgpass

patroni_password protects the Patroni REST API. It is not a PostgreSQL role password. After you change it in the inventory, refresh the target PostgreSQL cluster and the Infra administration configuration separately:

./pgsql.yml -l <cls> -t pg_conf,patroni_reload -e pg_reload=true
./infra.yml -l infra -t env_patroni

Then validate authentication and cluster state with patronictl or pig pg list <cls>.

4.3 - Managing PostgreSQL Databases

Database management - create, modify, delete, rebuild, and clone databases using templates

Quick Start

Pigsty manages databases declaratively. First define databases in the inventory. Then run bin/pgsql-db <cls> <dbname> to create or modify them.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases: [{ name: some_db }]  # <--- Define database list here!
Script
bin/pgsql-db <cls> <dbname>    # Create/modify <dbname> database on <cls> cluster
Playbook
./pgsql-db.yml -l pg-meta -e dbname=some_db    # Use playbook to create/modify database
Example
bin/pgsql-db pg-meta some_db    # Create/modify some_db database on pg-meta cluster

For the complete database definition reference, see Database Configuration. For database access permissions, see Access Control: Database Isolation.

You can set some parameters only at creation time. To change these parameters, recreate the database with state: recreate.

ActionCommandDescription
Create Databasebin/pgsql-db <cls> <db>Create new business database
Modify Databasebin/pgsql-db <cls> <db>Modify existing database properties
Delete Databasebin/pgsql-db <cls> <db>Delete database (requires state: absent)
Rebuild Databasebin/pgsql-db <cls> <db>Drop and recreate (requires state: recreate)
Clone Databasebin/pgsql-db <cls> <db>Clone database using template

demo/pgsql-db.cast — /demo/pgsql-db.cast


Create Database

During PostgreSQL cluster creation, the pg_db task automatically creates databases defined in pg_databases.

To create a database on an existing cluster, add its definition to all.children.<cls>.pg_databases. Then run:

Script
bin/pgsql-db <cls> <dbname>   # Create database <dbname>
Playbook
./pgsql-db.yml -l <cls> -e dbname=<dbname>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta myapp    # Create myapp database in pg-meta cluster

Example: Create business database myapp

#all.children.pg-meta.vars.pg_databases:
  - name: myapp
    owner: dbuser_myapp
    schemas: [app]
    extensions:
      - { name: pg_trgm }
      - { name: btree_gin }
    comment: my application database

Result: Pigsty creates myapp on the primary with dbuser_myapp as its owner. It creates the app schema and enables the pg_trgm and btree_gin extensions. It also adds the database to the Pgbouncer pool and registers a Grafana data source.

Recommendation: Use playbook

For manual database creation, you must ensure Pgbouncer pool and Grafana datasource sync yourself.


Modify Database

Use the same command as for creation. The playbook is idempotent when no baseline SQL is defined.

If the target database exists, Pigsty changes its properties to match the configuration. However, you can set some properties only at creation.

Script
bin/pgsql-db <cls> <db>   # Modify database <db> properties
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Idempotent, can repeat
Example
bin/pgsql-db pg-meta myapp    # Modify myapp database to match config

Immutable properties: To change these properties after creation, use state: recreate:

  • name (database name), template, strategy (clone strategy)
  • encoding, locale/lc_collate/lc_ctype, locale_provider/icu_locale/icu_rules/builtin_locale

All other properties can be modified. Common examples:

Modify owner: Change the owner field. Pigsty runs ALTER DATABASE ... OWNER TO and grants permissions.

- name: myapp
  owner: dbuser_new_owner     # New owner

Modify connection limit: Use connlimit to limit max connections.

- name: myapp
  connlimit: 100              # Max 100 connections

Revoke public connect: Set revokeconn: true to revoke the PUBLIC CONNECT privilege. Only the owner, DBA, monitor, and replication users can then connect.

- name: myapp
  owner: dbuser_myapp
  revokeconn: true            # Revoke PUBLIC CONNECT

Manage parameters: Use the parameters dictionary to set database-level parameters. Pigsty generates ALTER DATABASE ... SET commands. Use the special value DEFAULT to reset a parameter.

- name: myapp
  parameters:
    work_mem: '256MB'
    maintenance_work_mem: '512MB'
    statement_timeout: '30s'
    search_path: DEFAULT      # Reset to default

Manage schemas: Use the schemas array with simple or extended definitions. To drop a schema with CASCADE, set state: absent.

- name: myapp
  schemas:
    - app                                   # Simple form
    - { name: core, owner: dbuser_myapp }   # Specify owner
    - { name: deprecated, state: absent }   # Drop schema

Manage extensions: Use the extensions array with simple or extended definitions. To uninstall an extension with CASCADE, set state: absent.

- name: myapp
  extensions:
    - postgis                                 # Simple form
    - { name: vector, schema: public }        # Specify schema
    - { name: pg_trgm, state: absent }        # Uninstall extension
CASCADE Warning

Schema deletion and extension removal use CASCADE. This deletes all dependent objects. Understand the effect before you run these operations.

Connection pool config: By default, Pigsty adds all databases to Pgbouncer. Configure pooling with pgbouncer, pool_mode, pool_size, pool_reserve, pool_size_min, pool_connlimit, and pool_auth_user.

- name: myapp
  pgbouncer: true              # Add to pool (default true)
  pool_mode: transaction       # Pool mode: transaction/session/statement
  pool_size: 50                # Default pool size
  pool_reserve: 30             # Reserve pool size
  pool_size_min: 0             # Minimum pool size
  pool_connlimit: 100          # Max database connections
  pool_auth_user: dbuser_meta  # Auth query user (with pgbouncer_auth_query)

Since Pigsty v4.1.0, use pool_reserve and pool_connlimit for database pool fields. These names replace the legacy aliases pool_size_reserve and pool_max_db_conn.


Delete Database

To delete a database, set state to absent and execute:

Script
bin/pgsql-db <cls> <db>   # Delete <db> (config must have state: absent)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta olddb    # Delete olddb (config has state: absent)

Config example:

pg_databases:
  - name: olddb
    state: absent

Deletion process: If is_template: true, Pigsty first runs ALTER DATABASE ... IS_TEMPLATE false. It then uses DROP DATABASE ... WITH (FORCE) (PG13+) to delete the database and terminate all its connections. It removes the database from the Pgbouncer pool and unregisters its Grafana data source.

Protection: Pigsty cannot delete the system databases postgres, template0, and template1. Deletion runs only on the primary. Streaming replication sends the change to replicas.

Danger Warning

Database deletion is irreversible and permanently deletes all data. Before deletion, verify that a recent backup exists. Confirm that no business uses the database, and notify stakeholders. Pigsty is not responsible for any data loss from database deletion. Use at your own risk.


Rebuild Database

recreate state rebuilds database (drop then create):

Script
bin/pgsql-db <cls> <db>   # Rebuild <db> (config must have state: recreate)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta testdb    # Rebuild testdb (config has state: recreate)

Config example:

pg_databases:
  - name: testdb
    state: recreate
    owner: dbuser_test
    baseline: test_init.sql    # Execute after rebuild

Use cases: Test environment reset, clear dev database, modify immutable properties (encoding, locale), restore to initial state.

Difference from manual DROP + CREATE: One command performs the operation. It preserves Pgbouncer and Grafana configuration automatically. It also loads the baseline initialization script automatically.


Clone Database

Use the PostgreSQL template mechanism to clone a database. The template database must have no active connections during cloning.

Script
bin/pgsql-db <cls> <db>   # Clone <db> (config must specify template)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta meta_dev    # Clone meta_dev (config has template: meta)

Config example:

pg_databases:
  - name: meta                   # Source database

  - name: meta_dev
    template: meta               # Use meta as template
    strategy: FILE_COPY          # PG15+ clone strategy, instant on PG18

Instant Clone (PG18+): For PostgreSQL 18+, Pigsty configures file_copy_method by default. With strategy: FILE_COPY, cloning takes about 200ms and does not copy data files. For example, a 30GB database takes 18s with normal cloning and 200ms with instant cloning.

Manual clone: Ensure all connections to template are terminated:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'meta';
CREATE DATABASE meta_dev TEMPLATE meta STRATEGY FILE_COPY;

Limitations: Instant cloning requires a supported filesystem: xfs, btrfs, zfs, or apfs. Do not use the postgres database as a template. In high-concurrency environments, clear all template connections within the cloning window of about 200ms.


Connection Pool Management

Connection pool params in database definitions are applied to Pgbouncer when creating/modifying databases.

By default, Pigsty adds all databases to the Pgbouncer pool (pgbouncer: true). It adds their definitions to /etc/pgbouncer/database.txt. This file also holds database-level pool parameters: pool_auth_user, pool_mode, pool_size, pool_reserve, pool_size_min, and pool_connlimit.

As the postgres operating system user, use the pgb alias to access the Pgbouncer administration database. For more operations, see Pgbouncer Management.

4.4 - Patroni HA Management

Manage PostgreSQL cluster HA with Patroni, including config changes, status check, switchover, restart, and reinit replica.

Overview

Pigsty uses Patroni to manage PostgreSQL clusters. It handles configuration changes, status checks, switchovers, restarts, replica reinitialization, and other operations.

To manage clusters with Patroni, use one of these identities:

Patroni provides the patronictl administration CLI. Pigsty provides the pg wrapper alias to simplify its use.

Using patronictl via pg alias
pg ()
{
    local patroni_conf="/infra/conf/patronictl.yml";
    if [ ! -r ${patroni_conf} ]; then
        patroni_conf="/etc/patroni/patroni.yml";
        if [ ! -r ${patroni_conf} ]; then
            echo "error: patronictl config not found";
            return 1;
        fi;
    fi;
    patronictl -c ${patroni_conf} "$@"
}

Available Commands

CommandFunctionDescription
edit-configEdit ConfigInteractively edit cluster Patroni/PostgreSQL config
listList StatusList cluster members and their status
switchoverSwitchoverSwitch primary role to specified replica (planned)
failoverFailoverForce failover to specified replica (emergency)
restartRestartRestart PostgreSQL instance to apply restart-required params
reloadReloadReload Patroni config (no restart needed)
reinitReinit ReplicaReinitialize replica (wipe data and re-clone)
pausePause Auto-FailoverPause Patroni automatic failover
resumeResume Auto-FailoverResume Patroni automatic failover
historyView HistoryShow cluster failover history
show-configShow ConfigDisplay current cluster config (read-only)
queryExecute QueryExecute SQL query on cluster members
topologyView TopologyDisplay cluster replication topology
versionView VersionDisplay Patroni version info
removeRemove MemberRemove cluster member from DCS (dangerous)

Edit Config

Use edit-config to edit cluster Patroni and PostgreSQL configuration interactively. The command opens an editor for configuration stored in DCS. It applies changes automatically to all members. You can change Patroni parameters, such as ttl, loop_wait, and synchronous_mode. You can also change PostgreSQL parameters in postgresql.parameters.

pg edit-config <cls>                  # Interactive edit cluster config
pg edit-config <cls> --force          # Skip confirmation and apply directly
pg edit-config <cls> -p <k>=<v>       # Modify PostgreSQL param (--pg shorthand)
pg edit-config <cls> -s <k>=<v>       # Modify Patroni param (--set shorthand)

demo/pgsql-config.cast — /demo/pgsql-config.cast

Common config modification examples:

# Modify PostgreSQL param: slow query threshold (prompts for confirmation)
pg edit-config pg-test -p log_min_duration_statement=1000

# Modify PostgreSQL param, skip confirmation
pg edit-config pg-test -p log_min_duration_statement=1000 --force

# Modify multiple PostgreSQL params
pg edit-config pg-test -p work_mem=256MB -p maintenance_work_mem=1GB --force

# Modify Patroni params: increase failure detection window (increase RTO)
pg edit-config pg-test -s loop_wait=15 -s ttl=60 --force

# Modify Patroni param: enable synchronous replication mode
pg edit-config pg-test -s synchronous_mode=true --force

# Modify Patroni param: enable strict synchronous mode (require at least one sync replica for writes)
pg edit-config pg-test -s synchronous_mode_strict=true --force

# Modify restart-required params (need pg restart after)
pg edit-config pg-test -p shared_buffers=4GB --force
pg edit-config pg-test -p shared_preload_libraries='timescaledb, pg_stat_statements' --force
pg edit-config pg-test -p max_connections=200 --force

Some parameter changes require a PostgreSQL restart. Run pg list to find instances marked with *. These instances need a restart. Run pg restart to apply the changes. You can also call the Patroni REST API with curl or a program:

# View current config
curl -s 10.10.10.11:8008/config | jq .

# Modify params via API (requires auth)
curl -u 'postgres:Patroni.API' \
     -d '{"postgresql":{"parameters": {"log_min_duration_statement":200}}}' \
     -s -X PATCH http://10.10.10.11:8008/config | jq .

List Status

Use list to view cluster members and their status. The output shows each instance name, host, role, state, timeline, and replication lag. This is the most common command for cluster health checks.

pg list <cls>                         # List specified cluster status
pg list                               # List all clusters (on admin node)
pg list <cls> -e                      # Show extended info (--extended)
pg list <cls> -t                      # Show timestamp (--timestamp)
pg list <cls> -f json                 # Output as JSON (--format)
pg list <cls> -W 5                    # Refresh every 5 seconds (--watch)

Example output:

+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+

The output columns have these meanings:

  • Member: The instance name, formed from pg_cluster-pg_seq.
  • Host: The instance IP address.
  • Role: Leader (primary), Replica, Sync Standby, or Standby Leader (cascade primary).
  • State: The running state, such as running, streaming, in archive recovery, starting, or stopped.
  • TL: The timeline number, which increases after each switchover.
  • Lag in MB: Replication lag in MB. This value is not shown for the primary.

Instances requiring restart show * after the name:

+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1 * | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 * | 10.10.10.12 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

Switchover

Use switchover for a planned primary-replica role change. Patroni first verifies that the replica is fully synchronized. It then demotes the primary and promotes the target replica. The operation takes seconds, with a short interruption to writes. Use it for primary host maintenance, upgrades, or migration of the primary to a better node.

pg switchover <cls>                   # Interactive switchover, prompts for target replica
pg switchover <cls> --leader <old>    # Specify current primary name
pg switchover <cls> --candidate <new> # Specify target replica name
pg switchover <cls> --scheduled <time> # Scheduled switchover, format: 2024-12-01T03:00
pg switchover <cls> --force           # Skip confirmation

Before a switchover, verify that all replicas are healthy (running or streaming). Verify that replication lag is acceptable, and notify stakeholders.

# Interactive switchover (recommended, shows topology and prompts for selection)
$ pg switchover pg-test
Current cluster topology
+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+
Primary [pg-test-1]:
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-2
When should the switchover take place (e.g. 2024-01-01T12:00) [now]:
Are you sure you want to switchover cluster pg-test, demoting current leader pg-test-1? [y/N]: y

# Non-interactive switchover (specify primary and candidate)
pg switchover pg-test --leader pg-test-1 --candidate pg-test-2 --force

# Scheduled switchover (at 3 AM, for maintenance window)
pg switchover pg-test --leader pg-test-1 --candidate pg-test-2 --scheduled "2024-12-01T03:00"

After switchover, use pg list to confirm new cluster topology.


Failover

Use failover for emergency recovery when the primary is unavailable. It promotes a replica directly, without confirmation from the original primary. Replicas may not be fully synchronized, so failover may cause minor data loss. For operations that are not emergencies, use switchover.

pg failover <cls>                     # Interactive failover
pg failover <cls> --candidate <new>   # Specify replica to promote
pg failover <cls> --force             # Skip confirmation

Failover examples:

# Interactive failover
$ pg failover pg-test
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-2
Are you sure you want to failover cluster pg-test? [y/N]: y
Successfully failed over to "pg-test-2"

# Non-interactive failover (for emergencies)
pg failover pg-test --candidate pg-test-2 --force

Switchover vs Failover: Switchover supports planned maintenance and requires the original primary to be online. It waits for full synchronization and prevents data loss. Failover supports emergency recovery when the original primary can be offline. It promotes a replica directly and may lose data that has not replicated. Use Switchover for routine maintenance and upgrades. Use Failover only when the primary is completely down and cannot recover.

The built-in Patroni failover subcommand currently has no --leader option. Use switchover --leader ... to validate or name the old primary during a planned switchover. Failover accepts only the candidate replica.


Restart

Use restart to restart PostgreSQL instances, usually after changes to parameters that require a restart. For a whole cluster, patronictl submits each selected member in turn. It does not guarantee that replicas restart before the leader. If this order matters, restart replicas by role first. Then restart the leader separately.

pg restart <cls>                      # Restart all instances in cluster
pg restart <cls> <member>             # Restart specific instance
pg restart <cls> --role leader        # Restart primary only
pg restart <cls> --role replica       # Restart all replicas
pg restart <cls> --pending            # Restart only instances marked for restart
pg restart <cls> --scheduled <time>   # Scheduled restart
pg restart <cls> --timeout <sec>      # Set restart timeout (seconds)
pg restart <cls> --force              # Skip confirmation

After modifying restart-required params (shared_buffers, shared_preload_libraries, max_connections, max_worker_processes, etc.), use this command.

# Check which instances need restart (marked with *)
$ pg list pg-test
+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1 * | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 * | 10.10.10.12 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

# Restart single replica
pg restart pg-test pg-test-2

# Restart all cluster members (leader-last order is not guaranteed)
pg restart pg-test --force

# Restart only pending instances
pg restart pg-test --pending --force

# Explicitly restart replicas first, then the leader
pg restart pg-test --role replica --force
pg restart pg-test --role leader --force

# Scheduled restart (for maintenance window)
pg restart pg-test --scheduled "2024-12-01T03:00"

# Set restart timeout to 300 seconds
pg restart pg-test --timeout 300 --force

Reload

Use reload to reload Patroni configuration without a PostgreSQL restart. It reads configuration files again and applies parameters through pg_reload_conf(). It causes less disruption than restart: connections and running queries continue.

pg reload <cls>                       # Reload entire cluster config
pg reload <cls> <member>              # Reload specific instance config
pg reload <cls> --role leader         # Reload primary only
pg reload <cls> --role replica        # Reload all replicas
pg reload <cls> --force               # Skip confirmation

Most PostgreSQL parameter changes apply through reload. Only parameters with postmaster context require a restart, such as shared_buffers, max_connections, shared_preload_libraries, and archive_mode.

# Reload entire cluster
pg reload pg-test

# Reload single instance
pg reload pg-test pg-test-1

# Force reload, skip confirmation
pg reload pg-test --force

Reinit Replica

Use reinit to reinitialize a replica. This deletes all replica data and rebuilds it in the order specified by Patroni create_replica_methods. By default, Pigsty tries basebackup (pg_basebackup) first. If a remote pgBackRest repository is enabled, Pigsty also configures pgbackrest as a fallback. Use reinitialization for corrupted replica data, missing WAL after excessive lag, or a required replica configuration reset.

pg reinit <cls> <member>              # Reinitialize specified replica
pg reinit <cls> <member> --force      # Skip confirmation
pg reinit <cls> <member> --wait       # Wait for rebuild to complete

Warning: This operation deletes all data on the target instance. You can run it only on replicas, not on the primary.

# Reinitialize replica (prompts for confirmation)
$ pg reinit pg-test pg-test-2
Are you sure you want to reinitialize members pg-test-2? [y/N]: y
Success: reinitialize for member pg-test-2

# Force reinitialize, skip confirmation
pg reinit pg-test pg-test-2 --force

# Reinitialize and wait for completion
pg reinit pg-test pg-test-2 --force --wait

During rebuild, use pg list to check progress. Replica state shows creating replica:

+ Cluster: pg-test (7322261897169354773) --------------+----+------+
| Member    | Host        | Role    | State            | TL | Lag  |
+-----------+-------------+---------+------------------+----+------+
| pg-test-1 | 10.10.10.11 | Leader  | running          |  2 |      |
| pg-test-2 | 10.10.10.12 | Replica | creating replica |    |    ? |
+-----------+-------------+---------+------------------+----+------+

Pause

Use pause to pause Patroni automatic failover. While paused, Patroni does not promote replicas automatically, even if the primary fails. Pause during planned maintenance to prevent accidental failover. You can also pause for debugging to prevent cluster state changes, or to control manual switchover timing.

pg pause <cls>                        # Pause automatic failover
pg pause <cls> --wait                 # Pause and wait for all members to confirm

Warning: While paused, the cluster will not recover automatically if the primary fails. Run resume after maintenance.

# Pause automatic failover
$ pg pause pg-test
Success: cluster management is paused

# Check cluster status (shows Maintenance mode: on)
$ pg list pg-test
+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+
 Maintenance mode: on

Resume

Use resume to resume Patroni automatic failover. Run it immediately after maintenance so the cluster can recover automatically from a primary failure.

pg resume <cls>                       # Resume automatic failover
pg resume <cls> --wait                # Resume and wait for all members to confirm
# Resume automatic failover
$ pg resume pg-test
Success: cluster management is resumed

# Confirm resumed (Maintenance mode prompt disappears)
$ pg list pg-test

History

Use history to view cluster failover history. Each automatic or manual switchover creates a timeline record.

pg history <cls>                      # Show failover history
pg history <cls> -f json              # Output as JSON
pg history <cls> -f yaml              # Output as YAML
$ pg history pg-test
+----+-----------+------------------------------+---------------------------+
| TL |       LSN | Reason                       | Timestamp                 |
+----+-----------+------------------------------+---------------------------+
|  1 | 0/5000060 | no recovery target specified | 2024-01-15T10:30:00+08:00 |
|  2 | 0/6000000 | switchover to pg-test-2      | 2024-01-20T14:00:00+08:00 |
|  3 | 0/7000028 | failover to pg-test-1        | 2024-01-25T09:15:00+08:00 |
+----+-----------+------------------------------+---------------------------+

The history columns have these meanings:

  • TL: The timeline number. It increases after each switchover to distinguish primary histories.
  • LSN: The Log Sequence Number at switchover, which identifies the WAL position.
  • Reason: switchover to xxx for a manual operation, failover to xxx for a failure, or no recovery target specified for initialization.
  • Timestamp: The time of the switchover.

Show Config

Use show-config to view the current cluster configuration in DCS. This command only reads configuration. To change it, use edit-config.

pg show-config <cls>                  # Show cluster config
$ pg show-config pg-test
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=pg-test archive-push %p
    max_connections: 100
    shared_buffers: 256MB
    log_min_duration_statement: 1000
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
ttl: 30
synchronous_mode: false

Query

Use query to run SQL quickly on cluster members during debugging. For complex production queries, use psql or an application.

pg query <cls> -c "<sql>"             # Execute on primary
pg query <cls> -c "<sql>" -m <member> # Execute on specific instance (--member)
pg query <cls> -c "<sql>" -r leader   # Execute on primary (--role)
pg query <cls> -c "<sql>" -r replica  # Execute on all replicas
pg query <cls> -f <file>              # Execute SQL from file
pg query <cls> -c "<sql>" -U <user>   # Specify username (--username)
pg query <cls> -c "<sql>" -d <db>     # Specify database (--dbname)
pg query <cls> -c "<sql>" --format json  # Output as JSON
# Check primary connection count
pg query pg-test -c "SELECT count(*) FROM pg_stat_activity"

# Check PostgreSQL version
pg query pg-test -c "SELECT version()"

# Check replication status on all replicas
pg query pg-test -c "SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn()" -r replica

# Execute on specific instance
pg query pg-test -c "SELECT pg_is_in_recovery()" -m pg-test-2

# Use specific user and database
pg query pg-test -c "SELECT current_user, current_database()" -U postgres -d postgres

# Output as JSON
pg query pg-test -c "SELECT * FROM pg_stat_replication" --format json

Topology

Use topology to view cluster replication topology as a tree. It shows primary-replica relationships more clearly than list, especially for cascading replication.

pg topology <cls>                     # Show replication topology
$ pg topology pg-test
+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1   | 10.10.10.11 | Leader  | running |  1 |              |
| + pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| + pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

For cascading replication, the topology shows the replication hierarchy. For example, pg-test-3 replicates from pg-test-2, which replicates from the primary pg-test-1.


Version

Use version to view patronictl version.

pg version                            # Show patronictl version
$ pg version
patronictl version 4.1.0

Remove

Use remove to remove cluster or member metadata from DCS. This operation is dangerous. It removes only DCS metadata; it does not stop PostgreSQL or delete data files. Incorrect use can make cluster state inconsistent.

pg remove <cls>                       # Remove entire cluster metadata from DCS

Normally, you do not need this command. To remove a cluster or instance, use bin/pgsql-rm or pgsql-rm.yml. Consider remove only for orphaned DCS metadata after physical node removal, or metadata left after cluster destruction by other means.

# Remove entire cluster metadata (requires multiple confirmations)
$ pg remove pg-test
Please confirm the cluster name to remove: pg-test
You are about to remove all information in DCS for pg-test, please type: "Yes I am aware": Yes I am aware

4.5 - Managing PostgreSQL HBA Rules

HBA management - refresh rules, verify config, troubleshoot, Pgbouncer HBA

Quick Start

Pigsty manages HBA rules declaratively. First define HBA rules in the inventory. Then run bin/pgsql-hba <cls> to refresh them.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:                            # <--- Define HBA rules here!
      - {user: dbuser_app, db: app, addr: intra, auth: pwd, title: 'app access'}
      - {user: dbuser_api, db: all, addr: world, auth: ssl, title: 'api ssl access'}
Script
bin/pgsql-hba <cls>              # Refresh PostgreSQL and Pgbouncer HBA rules for cluster
bin/pgsql-hba <cls> <ip>...      # Refresh HBA rules for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both
Example
bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster HBA rules
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh specific instance only
bin/pgsql-hba pg-meta 10.10.10.11 10.10.10.12  # Refresh multiple instances

For rule syntax, see HBA Configuration. For authentication methods, default boundaries, and credential management, see Authentication.

ActionDescriptionRisk
Refresh HBA RulesRe-render config files and reload serviceLow
Verify HBA RulesView current rules, test connection authRead
Common ScenariosAdd rules, block IP, role-based, post-expansionLow
TroubleshootingConnection rejected, auth failed, rules not applied-
Pgbouncer HBAPgbouncer connection pool HBA managementLow

demo/pgsql-hba.cast — /demo/pgsql-hba.cast


Refresh HBA Rules

After you change HBA rules in pigsty.yml, generate the configuration files again and reload the services.

Script
bin/pgsql-hba <cls>              # Refresh entire cluster HBA (PostgreSQL + Pgbouncer)
bin/pgsql-hba <cls> <ip>...      # Refresh specific instances (multiple IPs space-separated)
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both
Example
bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh 10.10.10.10 instance only

Result: Pigsty generates PostgreSQL and Pgbouncer HBA configuration files from inventory definitions. It then reloads the services to apply the rules.

Config file locations

ServiceConfig File PathTemplate File
PostgreSQL/pg/data/pg_hba.confroles/pgsql/templates/pg_hba.conf
Pgbouncer/etc/pgbouncer/pgb_hba.confroles/pgsql/templates/pgbouncer.hba
Don’t edit config files directly

Direct changes to /pg/data/pg_hba.conf or /etc/pgbouncer/pgb_hba.conf work temporarily. The next Ansible playbook run overwrites them. Make all HBA rule changes in pigsty.yml, then run bin/pgsql-hba to refresh the rules.

Related Tags

TagDescription
pg_hbaRender PostgreSQL HBA config file
pg_reloadReload PostgreSQL config (needs pg_reload=true)
pgbouncer_hbaRender Pgbouncer HBA config file
pgbouncer_reloadReload Pgbouncer config

Verify HBA Rules

After you refresh HBA rules, verify that the configuration applies correctly.

View current HBA rules

SQL
-- View PostgreSQL HBA rules (recommended)
TABLE pg_hba_file_rules;

-- View matching rules for specific database
SELECT * FROM pg_hba_file_rules WHERE database @> ARRAY['mydb']::text[];
Bash
# View PostgreSQL HBA config file
cat /pg/data/pg_hba.conf

# View Pgbouncer HBA config file
cat /etc/pgbouncer/pgb_hba.conf

# View config file header (confirm if updated)
head -20 /pg/data/pg_hba.conf
Test Connection
# Test connection for specific user from specific address
psql -h <host> -p 5432 -U <user> -d <database> -c "SELECT 1"

# Test connection through Pgbouncer
psql -h <host> -p 6432 -U <user> -d <database> -c "SELECT 1"

Check HBA config syntax

# Reload config (validates syntax)
psql -c "SELECT pg_reload_conf()"

# If syntax errors, check logs
tail -f /pg/log/postgresql-*.log

Common Scenarios

Add New HBA Rule

Add the rule to pg_hba_rules in the cluster configuration. Then refresh the rules:

pg-meta:
  vars:
    pg_hba_rules:
      - {user: new_user, db: new_db, addr: '192.168.1.0/24', auth: pwd, title: 'new app access'}
bin/pgsql-hba pg-meta

Emergency IP Block

If you detect a malicious IP address, add a deny rule with high priority (order: 0):

pg_hba_rules:
  - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'emergency block'}
bin/pgsql-hba pg-meta    # Refresh immediately

Role-Based Rules

Configure different HBA rules for primary and replica using role parameter:

pg_hba_rules:
  # Only primary allows write users
  - {user: writer, db: all, addr: intra, auth: pwd, role: primary, title: 'writer on primary'}
  # Replicas allow read-only users
  - {user: reader, db: all, addr: world, auth: ssl, role: replica, title: 'reader on replica'}

After a refresh, the instance pg_role determines which rules are enabled.

Refresh HBA After Expansion

After you add cluster instances, refresh rules that use addr: cluster. The refreshed rules include the new members:

./pgsql.yml -l 10.10.10.14       # Add new instance
bin/pgsql-hba pg-meta            # Refresh all instances' HBA (includes new member IPs)

Refresh HBA After Failover

After a Patroni failover, instance roles may differ from pg_role values in the configuration. If HBA rules filter by role, update the configuration and refresh the rules:

# Update role definitions in pigsty.yml then refresh
bin/pgsql-hba pg-meta

Troubleshooting

Connection Rejected

Symptom: FATAL: no pg_hba.conf entry for host "x.x.x.x", user "xxx", database "xxx"

Steps:

  1. Check current HBA rules, confirm if matching rule exists:
psql -c "TABLE pg_hba_file_rules"
  1. Confirm client IP, username, database matches any rule

  2. Check rule order (HBA uses first-match-wins)

  3. Add corresponding rule and refresh:

bin/pgsql-hba <cls>

Authentication Failed

Symptom: FATAL: password authentication failed for user "xxx"

Steps:

  1. Confirm password is correct
  2. Check password encryption method (pg_pwd_enc) compatibility with client
  3. Check if user exists:
SELECT * FROM pg_roles WHERE rolname = 'xxx';

HBA Rules Not Applied

Steps:

  1. Confirm refresh command was executed
  2. Check if Ansible execution succeeded
  3. Confirm PostgreSQL reloaded:
psql -c "SELECT pg_reload_conf()"
  1. Check if config file was updated:
head -20 /pg/data/pg_hba.conf

Rule Order Issues

HBA uses the first matching rule. If rules do not work as expected:

  1. Check order values in rule definitions
  2. Use psql -c "TABLE pg_hba_file_rules" to view actual order
  3. Adjust order values (lower numbers = higher priority)

Pgbouncer HBA

Pgbouncer HBA management is similar to PostgreSQL, with some differences.

Config differences

DifferencePostgreSQLPgbouncer
Config file/pg/data/pg_hba.conf/etc/pgbouncer/pgb_hba.conf
ReplicationSupports db: replicationNot supported
Local authUses identUses peer

Refresh Pgbouncer HBA

Script
bin/pgsql-hba <cls>    # Refresh both PostgreSQL and Pgbouncer
Playbook
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
View
cat /etc/pgbouncer/pgb_hba.conf    # View Pgbouncer HBA rules

Best Practices

  1. Manage rules in the inventory: Make all changes in pigsty.yml. Do not edit pg_hba.conf directly.
  2. Test first: Verify HBA changes in a test environment first. Changes can cause connection problems.
  3. Set rule priority: Use order: 0 for blocklist rules to match them first.
  4. Refresh promptly: Refresh HBA after you add or remove instances, or after failover.
  5. Grant minimum access: Allow only necessary access. Avoid addr: world with auth: trust.
  6. Monitor authentication failures: Check pg_stat_activity for authentication failures.
  7. Back up configuration: Back up pigsty.yml before important changes.

4.6 - Pgbouncer Connection Pooling

Manage Pgbouncer connection pool, including pause, resume, disable, enable, reconnect, kill, and reload operations.

Overview

Pigsty uses Pgbouncer for PostgreSQL connection pooling. By default, it listens on port 6432 and proxies connections to local PostgreSQL on port 5432.

This component is optional. If you do not need many connections, transaction pooling, or query metrics, you can disable it. You can also connect directly to the database or leave Pgbouncer unused.


User & Database Management

Pigsty automatically manages Pgbouncer users and databases. When you create databases or users, it applies the database configuration and user configuration.

Database Management: By default, Pigsty adds databases from pg_databases to Pgbouncer. To exclude a database, set pgbouncer: false.

pg_databases:
  - name: mydb                # Added to connection pool by default
    pool_auth_user: dbuser_meta # Optional, auth query user (with pgbouncer_auth_query)
    pool_mode: transaction    # Database-level pool mode
    pool_size: 50             # Default pool size
    pool_reserve: 30          # Reserve pool size
    pool_size_min: 0          # Minimum pool size
    pool_connlimit: 100       # Max database connections
  - name: internal
    pgbouncer: false          # Excluded from connection pool

User Management: To add a user from pg_users to the pool user list, explicitly set pgbouncer: true.

pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true           # Add to connection pool user list
    pool_mode: transaction    # User-level pool mode
    pool_connlimit: 50        # User-level max connections

Since Pigsty v4.1.0, use pool_reserve and pool_connlimit for database pool fields. These names replace the legacy aliases pool_size_reserve and pool_max_db_conn.


Service Management

By default, the PostgreSQL Primary Service and Replica Service connect to Pgbouncer on port 6432. To connect directly to PostgreSQL, customize pg_services. Alternatively, set pg_default_service_dest to postgres.


Config Management

Pgbouncer config files are in /etc/pgbouncer/, generated and managed by Pigsty:

FileDescription
pgbouncer.iniMain config, pool-level params
database.txtDatabase list, database-level params
userlist.txtUser password list
useropts.txtUser-level pool params
pgb_hba.confHBA access control rules

Pigsty automatically manages database.txt and userlist.txt. It updates them when you create databases or users.

You can edit configuration manually, then run RELOAD to apply the changes:

# Edit config
$ vim /etc/pgbouncer/pgbouncer.ini

# Reload via systemctl
$ sudo systemctl reload pgbouncer

# Reload as pg_dbsu / postgres user
$ pgb -c "RELOAD;"

Pool Management

Pgbouncer runs as the same dbsu as PostgreSQL. The default is the postgres operating system user. Pigsty provides the pgb alias for administration:

alias pgb='psql -p6432 -dpgbouncer'

On database nodes, use pgb to connect to the Pgbouncer administration console. You can then run administration commands and monitoring queries.

$ pgb
pgbouncer=# SHOW POOLS;
pgbouncer=# SHOW CLIENTS;
pgbouncer=# SHOW SERVERS;
CommandFunctionDescription
PAUSEPausePause database, wait for txn completion then disconnect
RESUMEResumeResume database paused by PAUSE/KILL/SUSPEND
DISABLEDisableReject new client connections for database
ENABLEEnableAllow new client connections for database
RECONNECTReconnectGracefully close and rebuild server connections
KILLKillImmediately disconnect all client and server connections
KILL_CLIENTKill ClientTerminate specific client connection
SUSPENDSuspendFlush buffers and stop listening, for online restart
SHUTDOWNShutdownShutdown Pgbouncer process
RELOADReloadReload config files
WAIT_CLOSEWait CloseWait for server connections to close after RECONNECT/RELOAD
Monitor CommandsMonitorView pool status, clients, servers, etc.

PAUSE

Use PAUSE to pause database connections. Pgbouncer waits for the active transaction or session to finish, according to the pool mode. It then disconnects server connections. New client requests wait until RESUME.

PAUSE [db];           -- Pause specified database, or all if not specified

Typical use cases:

  • Online backend database switch (e.g., update connection target after switchover)
  • Maintenance operations requiring all connections disconnected
  • Combined with SUSPEND for Pgbouncer online restart
$ pgb -c "PAUSE mydb;"        # Pause mydb database
$ pgb -c "PAUSE;"             # Pause all databases

After pause, SHOW DATABASES shows paused status:

pgbouncer=# SHOW DATABASES;
   name   |   host    | port | database | ... | paused | disabled
----------+-----------+------+----------+-----+--------+----------
 mydb     | /var/run  | 5432 | mydb     | ... |      1 |        0

RESUME

Use RESUME to restore databases paused by PAUSE, KILL, or SUSPEND. This allows new connections and resumes normal service.

RESUME [db];          -- Resume specified database, or all if not specified
$ pgb -c "RESUME mydb;"       # Resume mydb database
$ pgb -c "RESUME;"            # Resume all databases

DISABLE

Use DISABLE to reject all new client connections for a database. Existing connections continue.

DISABLE db;           -- Disable specified database (database name required)

Typical use cases:

  • Temporarily offline a database for maintenance
  • Block new connections for safe database migration
  • Gradually decommission a database being removed
$ pgb -c "DISABLE mydb;"      # Disable mydb, new connections rejected

ENABLE

Use ENABLE to enable a database that DISABLE stopped. The database accepts new client connections again.

ENABLE db;            -- Enable specified database (database name required)
$ pgb -c "ENABLE mydb;"       # Enable mydb, allow new connections

RECONNECT

Use RECONNECT to rebuild server connections gracefully. Pgbouncer closes connections when they return to the pool. It creates new connections as needed.

RECONNECT [db];       -- Rebuild server connections for database, or all if not specified

Typical use cases:

  • Refresh connections after backend database IP change
  • Reroute traffic after switchover
  • Rebuild connections after DNS update
$ pgb -c "RECONNECT mydb;"    # Rebuild mydb server connections
$ pgb -c "RECONNECT;"         # Rebuild all server connections

After RECONNECT, use WAIT_CLOSE to wait for old connections to fully release.


KILL

Use KILL to disconnect all client and server connections for a database immediately. Unlike PAUSE, it does not wait for transactions to finish.

KILL [db];            -- Kill all connections for database, or all (except admin) if not specified
$ pgb -c "KILL mydb;"         # Force disconnect all mydb connections
$ pgb -c "KILL;"              # Force disconnect all database connections (except admin)

After KILL, new connections are blocked until RESUME.


KILL_CLIENT

Use KILL_CLIENT to terminate a specific client connection. Get the client ID from SHOW CLIENTS.

KILL_CLIENT id;       -- Terminate client connection with specified ID
# View client connections
$ pgb -c "SHOW CLIENTS;"

# Terminate specific client (assuming ptr column shows ID 0x1234567890)
$ pgb -c "KILL_CLIENT 0x1234567890;"

SUSPEND

Use SUSPEND to suspend Pgbouncer. It flushes all socket buffers and stops listening until RESUME.

SUSPEND;              -- Suspend Pgbouncer

SUSPEND is mainly for Pgbouncer online restart (zero-downtime upgrade):

# 1. Suspend current Pgbouncer
$ pgb -c "SUSPEND;"

# 2. Start new Pgbouncer process (with -R option to take over sockets)
$ pgbouncer -R /etc/pgbouncer/pgbouncer.ini

# 3. New process takes over, old process exits automatically

SHUTDOWN

Use SHUTDOWN to stop the Pgbouncer process. The command supports several shutdown modes:

SHUTDOWN;                      -- Immediate shutdown
SHUTDOWN WAIT_FOR_SERVERS;     -- Wait for server connections to release
SHUTDOWN WAIT_FOR_CLIENTS;     -- Wait for clients to disconnect (zero-downtime rolling restart)
ModeDescription
SHUTDOWNImmediately shutdown Pgbouncer
WAIT_FOR_SERVERSStop accepting new connections, wait for server release
WAIT_FOR_CLIENTSStop accepting new connections, wait for all clients disconnect, for rolling restart
$ pgb -c "SHUTDOWN WAIT_FOR_CLIENTS;"   # Graceful shutdown, wait for clients

RELOAD

Use RELOAD to read the Pgbouncer configuration files again. It dynamically updates most parameters without a process restart.

RELOAD;               -- Reload config files
$ pgb -c "RELOAD;"              # Reload via admin console
$ systemctl reload pgbouncer    # Reload via systemd
$ kill -SIGHUP $(cat /run/postgresql/pgbouncer.pid)  # Reload via signal

Pigsty provides playbook task to reload Pgbouncer config:

./pgsql.yml -l <cls> -t pgbouncer_reload    # Reload cluster Pgbouncer config

WAIT_CLOSE

Use WAIT_CLOSE to wait until server connections finish closing. Usually, run it after RECONNECT or RELOAD to verify that old connections have closed.

WAIT_CLOSE [db];      -- Wait for server connections to close, or all if not specified
# Complete connection rebuild flow
$ pgb -c "RECONNECT mydb;"
$ pgb -c "WAIT_CLOSE mydb;"    # Wait for old connections to release

Monitoring

Pgbouncer provides these SHOW commands to monitor pool status:

CommandDescription
SHOW HELPShow available commands
SHOW DATABASESShow database config and status
SHOW POOLSShow pool statistics
SHOW CLIENTSShow client connection list
SHOW SERVERSShow server connection list
SHOW USERSShow user config
SHOW STATSShow statistics (requests, bytes)
SHOW STATS_TOTALSShow cumulative statistics
SHOW STATS_AVERAGESShow average statistics
SHOW CONFIGShow current config params
SHOW MEMShow memory usage
SHOW DNS_HOSTSShow DNS cached hostnames
SHOW DNS_ZONESShow DNS cached zones
SHOW SOCKETSShow open socket info
SHOW ACTIVE_SOCKETSShow active sockets
SHOW LISTSShow internal list counts
SHOW FDSShow file descriptor usage
SHOW STATEShow Pgbouncer running state
SHOW VERSIONShow Pgbouncer version

Common monitoring examples:

# View pool status
$ pgb -c "SHOW POOLS;"

# View client connections
$ pgb -c "SHOW CLIENTS;"

# View server connections
$ pgb -c "SHOW SERVERS;"

# View statistics
$ pgb -c "SHOW STATS;"

# View database status
$ pgb -c "SHOW DATABASES;"

For more monitoring command details, see Pgbouncer official docs.


Unix Signals

Use Unix signals to control Pgbouncer when the administration console is unavailable:

SignalEquivalent CommandDescription
SIGHUPRELOADReload config files
SIGTERMSHUTDOWN WAIT_FOR_CLIENTSGraceful shutdown, wait clients
SIGINTSHUTDOWN WAIT_FOR_SERVERSGraceful shutdown, wait servers
SIGQUITSHUTDOWNImmediate shutdown
SIGUSR1PAUSEPause all databases
SIGUSR2RESUMEResume all databases
# Reload config via signal
$ kill -SIGHUP $(cat /run/postgresql/pgbouncer.pid)

# Graceful shutdown via signal
$ kill -SIGTERM $(cat /run/postgresql/pgbouncer.pid)

# Pause via signal
$ kill -SIGUSR1 $(cat /run/postgresql/pgbouncer.pid)

# Resume via signal
$ kill -SIGUSR2 $(cat /run/postgresql/pgbouncer.pid)

Traffic Switching

Pigsty stores managed database routes in /etc/pgbouncer/database.txt. To move Pgbouncer traffic for one database to another node, edit this file and reload the configuration. Then drain and rebuild the existing server connections:

# 1. Change only mydb's backend target to 10.10.10.12
$ sed -i -E '/^mydb[[:space:]]*=/ s#host=[^[:space:]]+#host=10.10.10.12#' /etc/pgbouncer/database.txt

# 2. Reload config
$ pgb -c "RELOAD;"

# 3. Rebuild this database's connections and wait for old connections to close
$ pgb -c "RECONNECT mydb;"
$ pgb -c "WAIT_CLOSE mydb;"

The pgb-route function currently shipped in the source edits only /etc/pgbouncer/pgbouncer.ini. That file includes database.txt but does not contain the generated host= routes for each database. Thus, the function does not change managed database backends. Use the procedure above instead.

4.7 - Managing PostgreSQL Component Services

Use systemctl to manage PostgreSQL cluster component services - start, stop, restart, reload, and status check.

Overview

The Pigsty PGSQL module has multiple components. Each component runs as a systemd service on nodes, except pgBackRest.

To maintain production PostgreSQL clusters, learn how these components work and how to manage them.

ComponentPortService NameDescription
Patroni8008patroniHA manager, manages PostgreSQL lifecycle
PostgreSQL5432postgresPlaceholder service, not used, for emergency
Pgbouncer6432pgbouncerConnection pooling middleware, traffic entry
PgBackRest--pgBackRest has no daemon service
HAProxy543xhaproxyLoad balancer, exposes database services
pg_exporter9630pg_exporterPostgreSQL metrics exporter
pgbouncer_exporter9631pgbouncer_exporterPgbouncer metrics exporter
vip-manager-vip-managerOptional, manages L2 VIP address floating
Important

Do not manage the PostgreSQL service directly with systemctl. Patroni manages PostgreSQL. Use patronictl commands. Direct PostgreSQL operations can make Patroni state inconsistent and trigger an unexpected failover. Use the postgres service only for emergencies when Patroni fails.


Quick Reference

OperationCommand
Startsystemctl start <service>
Stopsystemctl stop <service>
Restartsystemctl restart <service>
Reloadsystemctl reload <service>
Statussystemctl status <service>
Logsjournalctl -u <service> -f
Enablesystemctl enable <service>
Disablesystemctl disable <service>

Common service names: patroni, pgbouncer, haproxy, pg_exporter, pgbouncer_exporter, vip-manager


Patroni

Patroni manages PostgreSQL high availability. It controls startup, shutdown, failure detection, and automatic failover. It is the core component of the PGSQL module and manages the PostgreSQL process. Do not manage the postgres service directly with systemctl.

Start Patroni

systemctl start patroni     # Start Patroni (also starts PostgreSQL)

After startup, Patroni automatically starts PostgreSQL. On the first startup, the instance role determines the action:

  • Primary: Initialize or recover the data directory.
  • Replica: Clone data from the primary and establish replication.

Stop Patroni

systemctl stop patroni      # Stop Patroni (also stops PostgreSQL)

Stopping Patroni shuts down PostgreSQL gracefully. If the instance is the primary and automatic failover is active, this operation may trigger a failover.

Restart Patroni

systemctl restart patroni   # Restart Patroni (also restarts PostgreSQL)

A restart causes a short service interruption. For production clusters, use pg restart for a rolling restart.

Reload Patroni

systemctl reload patroni    # Reload Patroni config

A reload reads the configuration file again. It applies parameters that PostgreSQL can change without a restart.

View Status & Logs

systemctl status patroni    # View Patroni service status
journalctl -u patroni -f    # Real-time Patroni logs
journalctl -u patroni -n 100 --no-pager  # Last 100 lines

Config file: /etc/patroni/patroni.yml

Best Practice: Use patronictl instead of systemctl to manage PostgreSQL clusters.


Pgbouncer

Pgbouncer provides lightweight PostgreSQL connection pooling. Business traffic usually goes through Pgbouncer on port 6432, rather than directly to PostgreSQL on port 5432. This reuses connections and protects the database.

Start Pgbouncer

systemctl start pgbouncer

Stop Pgbouncer

systemctl stop pgbouncer

Note: Stopping Pgbouncer disconnects all pooled business connections.

Restart Pgbouncer

systemctl restart pgbouncer

A restart disconnects all existing connections. To apply only configuration changes, use reload.

Reload Pgbouncer

systemctl reload pgbouncer

A reload reads configuration files, such as user lists and pool parameters, again. It does not disconnect existing connections.

View Status & Logs

systemctl status pgbouncer
journalctl -u pgbouncer -f

Config files:

  • Main config: /etc/pgbouncer/pgbouncer.ini
  • HBA rules: /etc/pgbouncer/pgb_hba.conf
  • User list: /etc/pgbouncer/userlist.txt
  • Database list: /etc/pgbouncer/database.txt

Admin Console

psql -p 6432 -U postgres -d pgbouncer  # Connect to Pgbouncer admin console

Common admin commands:

SHOW POOLS;      -- View pool status
SHOW CLIENTS;    -- View client connections
SHOW SERVERS;    -- View backend server connections
SHOW STATS;      -- View statistics
RELOAD;          -- Reload config
PAUSE;           -- Pause all pools
RESUME;          -- Resume all pools

HAProxy

HAProxy is a high-performance load balancer. It routes traffic to the correct PostgreSQL instances. Pigsty uses HAProxy to expose services. HAProxy routes traffic by instance role (primary or replica) and health status.

Start HAProxy

systemctl start haproxy

Stop HAProxy

systemctl stop haproxy

Note: Stopping HAProxy disconnects all load-balanced connections.

Restart HAProxy

systemctl restart haproxy

Reload HAProxy

systemctl reload haproxy

HAProxy can reload without disconnecting existing connections. Use reload to apply configuration changes.

View Status & Logs

systemctl status haproxy
journalctl -u haproxy -f

Config files: The main configuration is /etc/haproxy/haproxy.cfg. Pigsty stores generated service fragments in /etc/haproxy/conf.d/.

Admin Interface

HAProxy provides a web admin interface, default port 9101:

http://<node_ip>:9101/haproxy

The default administration username is admin. Set its password with haproxy_admin_password.


pg_exporter

pg_exporter exports PostgreSQL performance metrics for Prometheus.

Start pg_exporter

systemctl start pg_exporter

Stop pg_exporter

systemctl stop pg_exporter

After you stop the exporter, Prometheus cannot collect PostgreSQL metrics from this instance.

Restart pg_exporter

systemctl restart pg_exporter

View Status & Logs

systemctl status pg_exporter
journalctl -u pg_exporter -f

Config file: /etc/pg_exporter.yml

Verify Metrics

curl -s localhost:9630/metrics | head -20

pgbouncer_exporter

pgbouncer_exporter is Pgbouncer’s Prometheus metrics exporter.

Start/Stop/Restart

systemctl start pgbouncer_exporter
systemctl stop pgbouncer_exporter
systemctl restart pgbouncer_exporter

View Status & Logs

systemctl status pgbouncer_exporter
journalctl -u pgbouncer_exporter -f

Verify Metrics

curl -s localhost:9631/metrics | head -20

vip-manager

vip-manager is an optional component that moves an L2 VIP between nodes. When pg_vip_enabled is enabled, vip-manager binds the VIP to the current primary node.

Start vip-manager

systemctl start vip-manager

Stop vip-manager

systemctl stop vip-manager

After you stop vip-manager, it releases the VIP address from the current node.

Restart vip-manager

systemctl restart vip-manager

View Status & Logs

systemctl status vip-manager
journalctl -u vip-manager -f

Config file: /etc/default/vip-manager

Verify VIP Binding

ip addr show           # Check network interfaces, verify VIP binding
pg list <cls>          # Confirm primary location

Startup Order & Dependencies

Recommended PGSQL module component startup order:

1. patroni          # Start Patroni first (auto-starts PostgreSQL)
2. pgbouncer        # Then start connection pool
3. haproxy          # Start load balancer
4. pg_exporter      # Start metrics exporters
5. pgbouncer_exporter
6. vip-manager      # Finally start VIP manager (if enabled)

Stop the components in reverse order. Pigsty playbooks handle these dependencies automatically.

Batch Start All Services

systemctl start patroni pgbouncer haproxy pg_exporter pgbouncer_exporter

Batch Stop All Services

systemctl stop pgbouncer_exporter pg_exporter haproxy pgbouncer patroni

Common Troubleshooting

Service Startup Failure

systemctl status <service>        # View service status
journalctl -u <service> -n 50     # View recent logs
journalctl -u <service> --since "5 min ago"  # Last 5 minutes logs

Patroni Won’t Start

SymptomPossible CauseSolution
Can’t connect to etcdetcd cluster unavailableCheck etcd service status
Data dir permission errorFile ownership not postgreschown -R postgres:postgres /pg/data
Port in useLeftover PostgreSQL processpg_ctl stop -D /pg/data or kill

Pgbouncer Won’t Start

SymptomPossible CauseSolution
Config syntax errorINI format errorCheck /etc/pgbouncer/pgbouncer.ini
Port in usePort 6432 already usedlsof -i :6432
userlist.txt permissionsIncorrect file permissionschmod 600 /etc/pgbouncer/userlist.txt

HAProxy Won’t Start

SymptomPossible CauseSolution
Config syntax errorInvalid main configuration or service fragmenthaproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c -q
Port in useService port conflictlsof -i :5433

4.8 - Manage PostgreSQL Cron Jobs

Configure crontab to schedule PostgreSQL backups, vacuum freeze, and bloat maintenance tasks

Pigsty uses crontab to schedule routine backups, transaction freezing, and reorganization of bloated tables and indexes.

Quick Reference

OperationQuick CommandDescription
Configure Cron Jobs./pgsql.yml -t pg_crontab -l <cls>Apply pg_crontab config
View Cron Jobscrontab -lView as postgres user
Physical Backuppg-backup [full|diff|incr]Execute backup with pgBackRest
Transaction Freezepg-vacuum [database...]Freeze aging transactions, prevent XID wraparound
Bloat Maintenancepg-repack [database...]Online reorganize bloated tables and indexes

For other management tasks, see: Backup Management, Monitoring System, HA Management.


Configure Cron Jobs

Use pg_crontab to configure cron jobs for the PostgreSQL database superuser. The pg_dbsu parameter selects this user; the default is postgres.

Example Configuration

The following pg-meta configuration schedules a full backup daily at 1:00 AM. The pg-test configuration schedules a full backup on Monday and incremental backups on other days.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_crontab:
      - '00 01 * * * /pg/bin/pg-backup'
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_crontab:
      - '00 01 * * 1            /pg/bin/pg-backup full'
      - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup'

Recommended Maintenance Schedule

pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'    # Daily full backup at 1:00 AM
  - '00 03 * * 0 /pg/bin/pg-vacuum'         # Weekly vacuum freeze on Sunday at 3:00 AM
  - '00 04 * * 1 /pg/bin/pg-repack'         # Weekly repack on Monday at 4:00 AM
TaskFrequencyTimingDescription
pg-backupDailyEarly morningFull or incremental backup, depending on business needs
pg-vacuumWeeklySunday early morningFreeze aging transactions, prevent XID wraparound
pg-repackWeekly/MonthlyOff-peak hoursReorganize bloated tables/indexes, reclaim space
Primary Only Execution

The pg-backup, pg-vacuum, and pg-repack scripts detect the current node role automatically. They run only on the primary. Replicas exit immediately. Thus, you can configure the same cron jobs on all nodes. After failover, the new primary automatically continues the scheduled maintenance tasks.


Apply Cron Jobs

The pg_crontab task in pgsql.yml writes cron jobs to the default location for the operating system:

  • EL (RHEL/Rocky/Alma): /var/spool/cron/postgres
  • Debian/Ubuntu: /var/spool/cron/crontabs/postgres
Playbook
./pgsql.yml -l pg-meta -t pg_crontab     # Apply pg_crontab config to specified cluster
./pgsql.yml -l 10.10.10.10 -t pg_crontab # Target specific host only
Manual
# Edit cron jobs as postgres user
sudo -u postgres crontab -e

# Or edit crontab file directly
sudo vi /var/spool/cron/postgres           # EL series
sudo vi /var/spool/cron/crontabs/postgres  # Debian/Ubuntu

Each playbook execution will fully overwrite the cron job configuration.


View Cron Jobs

Execute the following command as the pg_dbsu OS user to view cron jobs:

crontab -l

# Pigsty Managed Crontab for postgres
SHELL=/bin/bash
PATH=/usr/pgsql/bin:/pg/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin
MAILTO=""
00 01 * * * /pg/bin/pg-backup

For explanations of crontab syntax, see Crontab Guru.


pg-backup

pg-backup is the Pigsty physical backup script. It uses pgBackRest and supports full, differential, and incremental backups.

Basic Usage

pg-backup                # Execute incremental backup (default), auto full if no existing full backup
pg-backup full           # Execute full backup
pg-backup diff           # Execute differential backup (based on most recent full backup)
pg-backup incr           # Execute incremental backup (based on most recent any backup)

Backup Types

TypeParameterDescription
Full BackupfullComplete backup of all data, only this backup needed for recovery
DifferentialdiffBackup changes since last full backup, recovery needs full + diff
IncrementalincrBackup changes since last any backup, recovery needs complete chain

Execution Requirements

  • Script must run on primary as postgres user
  • Script auto-detects current node role, exits (exit 1) when run on replica
  • Auto-retrieves stanza name from /etc/pgbackrest/pgbackrest.conf

Common Cron Configurations

Daily Full
pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'    # Daily full backup at 1:00 AM
Weekly Full + Daily Incr
pg_crontab:
  - '00 01 * * 1            /pg/bin/pg-backup full'  # Monday full backup
  - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup'       # Other days incremental
Weekly Full + Daily Diff
pg_crontab:
  - '00 01 * * 1            /pg/bin/pg-backup full'  # Monday full backup
  - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup diff'  # Other days differential

For more backup and recovery operations, see the Backup Management section.


pg-vacuum

pg-vacuum is the Pigsty transaction freeze script. It runs VACUUM FREEZE to prevent database shutdown caused by transaction ID (XID) wraparound.

Basic Usage

Basic
pg-vacuum                    # Freeze aging tables in all databases
pg-vacuum mydb               # Process specified database only
pg-vacuum mydb1 mydb2        # Process multiple databases
Options
pg-vacuum -n mydb            # Dry run mode, display only without executing
pg-vacuum -a 80000000 mydb   # Use custom age threshold (default 100M)
pg-vacuum -r 50 mydb         # Use custom aging ratio threshold (default 40%)
Manual SQL
-- Execute VACUUM FREEZE on entire database
VACUUM FREEZE;

-- Execute VACUUM FREEZE on specific table
VACUUM FREEZE schema.table_name;

Command Options

OptionDescriptionDefault
-h, --helpShow help message-
-n, --dry-runDry run mode, display onlyfalse
-a, --ageAge threshold, tables exceeding need freeze100000000
-r, --ratioAging ratio threshold, full freeze if exceeded (%)40

Logic

  1. Check the database datfrozenxid age. Skip the database if its age is below the threshold.
  2. Calculate the aging page ratio: table pages above the age threshold as a percentage of all pages.
  3. If the aging ratio exceeds 40%, run VACUUM FREEZE ANALYZE on the full database.
  4. Otherwise, run VACUUM FREEZE ANALYZE only on tables above the age threshold.

Script sets vacuum_cost_limit = 10000 and vacuum_cost_delay = 1ms to control I/O impact.

Execution Requirements

  • Script must run on primary as postgres user
  • Uses file lock /tmp/pg-vacuum.lock to prevent concurrent execution
  • Auto-skips template0, template1, postgres system databases

Common Cron Configuration

pg_crontab:
  - '00 03 * * 0 /pg/bin/pg-vacuum'     # Weekly Sunday at 3:00 AM

pg-repack

pg-repack is the Pigsty bloat maintenance script. It uses the pg_repack extension to reorganize bloated tables and indexes online.

Basic Usage

Basic
pg-repack                    # Reorganize bloated tables and indexes in all databases
pg-repack mydb               # Reorganize specified database only
pg-repack mydb1 mydb2        # Reorganize multiple databases
Options
pg-repack -n mydb            # Dry run mode, display only without executing
pg-repack -t mydb            # Reorganize tables only
pg-repack -i mydb            # Reorganize indexes only
pg-repack -T 30 -j 4 mydb    # Custom lock timeout (seconds) and parallelism
Manual
# Use pg_repack command directly to reorganize specific table
pg_repack dbname -t schema.table

# Use pg_repack command directly to reorganize specific index
pg_repack dbname -i schema.index

Command Options

OptionDescriptionDefault
-h, --helpShow help message-
-n, --dry-runDry run mode, display onlyfalse
-t, --tableReorganize tables onlyfalse
-i, --indexReorganize indexes onlyfalse
-T, --timeoutLock wait timeout (seconds)10
-j, --jobsParallel jobs2

Auto-Selection Thresholds

Script auto-selects objects to reorganize based on table/index size and bloat ratio:

Table Bloat Thresholds

Size RangeBloat ThresholdMax Count
< 256MB> 40%64
256MB - 2GB> 30%16
2GB - 8GB> 20%4
8GB - 64GB> 15%1

Index Bloat Thresholds

Size RangeBloat ThresholdMax Count
< 128MB> 40%64
128MB - 1GB> 35%16
1GB - 8GB> 30%4
8GB - 64GB> 20%1

The script skips tables and indexes larger than 64GB and shows a warning. Handle these objects manually.

Execution Requirements

  • Script must run on primary as postgres user
  • Requires pg_repack extension installed (installed by default in Pigsty)
  • Requires pg_table_bloat and pg_index_bloat views in monitor schema
  • Uses file lock /tmp/pg-repack.lock to prevent concurrent execution
  • Auto-skips template0, template1, postgres system databases
Lock Waiting

Normal reads and writes continue during reorganization. However, the final switch requires an AccessExclusive lock on the table, which blocks all access. For high-throughput workloads, run the script during off-peak hours or maintenance windows.

Common Cron Configuration

pg_crontab:
  - '00 04 * * 1 /pg/bin/pg-repack'     # Weekly Monday at 4:00 AM

Use the PGCAT Database - Table Bloat panel to check database bloat. Then select tables and indexes with high bloat for reorganization.

For more details see: Managing Relation Bloat


Remove Cron Jobs

When pgsql-rm.yml removes a PostgreSQL cluster, it automatically deletes the postgres user crontab file.

./pgsql-rm.yml -l <cls> -t pg_crontab    # Remove cron jobs only
./pgsql-rm.yml -l <cls> --check          # Preflight full removal; execute only after backup and target confirmation

4.9 - Managing PostgreSQL Extensions

Extension management - download, install, configure, enable, update, and remove extensions

Quick Start

Pigsty provides 600 extensions. To use an extension, complete four steps: download, install, configure, and enable.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector ]           # <--- Install extension packages
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'    # <--- Configure preload extensions
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]            # <--- Enable in database
Script
bin/pgsql-ext <cls>           # Install extensions defined in config on <cls> cluster
bin/pgsql-ext <cls> [ext...]  # Install extensions specified on command line
Playbook
./pgsql.yml -l pg-meta -t pg_ext    # Use playbook to install extensions
Example
bin/pgsql-ext pg-meta                         # Install defined extensions on pg-meta cluster
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions

For complete extension reference, see Extensions. For available extensions, see Extension Catalog.

ActionCommandDescription
Download Extensions./infra.yml -t repo_buildDownload extensions to local repo
Install Extensionsbin/pgsql-ext <cls>Install extension packages on cluster
Configure Extensionspg edit-config <cls> -pAdd to preload libs (requires restart)
Enable Extensionspsql -c 'CREATE EXT ...'Create extension objects in database
Update ExtensionsALTER EXTENSION UPDATEUpdate packages and extension objects
Remove ExtensionsDROP EXTENSIONDrop extension objects, uninstall pkgs

demo/pgsql-ext.cast — /demo/pgsql-ext.cast


Install Extensions

During PostgreSQL cluster creation, the pg_extension task automatically installs extensions from pg_extensions.

To install extensions on an existing cluster, add them to all.children.<cls>.pg_extensions. Then run:

Script
bin/pgsql-ext <cls>   # Install extensions on <cls> cluster
Playbook
./pgsql.yml -l <cls> -t pg_extension   # Use Ansible playbook
Example
bin/pgsql-ext pg-meta    # Install extensions defined in config on pg-meta

Example: Install PostGIS, TimescaleDB and PGVector on cluster

#all.children.pg-meta.vars:
pg_extensions: [ postgis, timescaledb, pgvector ]

Result: Pigsty installs extension packages on all cluster nodes. It translates package aliases to package names for the operating system and PostgreSQL version.

Ensure repos available before install

Before installation, verify that the nodes have the correct software repositories. Download extensions to the local repository or configure upstream repositories.


Manual Install

To install extensions without managing them in Pigsty configuration, pass the extension list on the command line:

Script
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions on pg-meta
Playbook
./pgsql.yml -l pg-meta -t pg_ext -e '{"pg_extensions": ["pg_duckdb", "pg_mooncake"]}'

You can also use the pig package manager CLI to install extensions on one node. It resolves package aliases automatically.

pig install postgis timescaledb       # Install multiple extensions
pig install pgvector -v 18            # Install for specific PG major version

ansible pg-test -b -a 'pig install pg_duckdb'   # Batch install on cluster with Ansible

You can also use the operating system package manager directly (apt/dnf). You must know the exact RPM or DEB package name for your operating system and PostgreSQL version:

# EL systems (RHEL, Rocky, Alma, Oracle Linux)
sudo yum install -y pgvector_18*

# Debian / Ubuntu
sudo apt install -y postgresql-18-pgvector

Download Extensions

Before installation, verify that the node can get the extension from its software repositories:

The default Pigsty configuration downloads common extensions during installation. For additional extensions, add them to repo_extra_packages and rebuild the repository:

repo_extra_packages: [ pgvector, postgis, timescaledb ]
Script
make repo         # Shortcut = repo-build + node-repo
make repo-build   # Rebuild Infra repo (download packages and deps)
make node-repo    # Refresh node repo cache, update Infra repo reference
Playbook
./deploy.yml -t repo_build,node_repo  # Execute both tasks at once
./infra.yml -t repo_build     # Re-download packages to local repo
./node.yml  -t node_repo      # Refresh node repo cache

Configure Repos

You can configure all nodes to install directly from upstream extension repositories. This skips the local download step. It is not recommended for production:

./node.yml -t node_repo -e node_repo_modules=node,pgsql   # Add PGDG and Pigsty upstream repos

Configure Extensions

Some extensions must be in shared_preload_libraries. Changes to this parameter require a database restart.

Use pg_libs to set the default list of preload extensions. This value applies only during cluster initialization. Changes to this value have no effect after initialization.

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'   # Preload extensions
    pg_extensions: [ timescaledb, postgis, pgvector ]          # Install packages

For existing clusters, refer to Modify Config to modify shared_preload_libraries:

pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'
pg restart pg-meta   # Modify pg-meta params and restart to apply

Before you add preload configuration, verify that the extension packages are installed correctly. If an extension in shared_preload_libraries is missing or cannot load, PostgreSQL will not start. Manage cluster configuration changes through Patroni. Avoid ALTER SYSTEM or pg_parameters for separate instance configuration changes. Different primary and replica configurations can cause startup failure or interrupt replication.


Enable Extensions

After you install the packages, run CREATE EXTENSION in the database to enable extension features.

Enable during cluster init

Declare extensions to enable in database definition via extensions array:

pg_databases:
  - name: meta
    extensions:
      - vector                             # Simple form
      - { name: postgis, schema: public }  # Specify schema

Manual enable

SQL
CREATE EXTENSION vector;                      -- Create extension
CREATE EXTENSION postgis SCHEMA public;       -- Specify schema
CREATE EXTENSION IF NOT EXISTS vector;        -- Idempotent creation
CREATE EXTENSION postgis_topology CASCADE;    -- Auto-install dependencies
psql
psql -d meta -c 'CREATE EXTENSION vector;'                  # Create extension in meta database
psql -d meta -c 'CREATE EXTENSION postgis SCHEMA public;'   # Specify schema
Playbook
# After modifying database definition, use playbook to enable extensions
bin/pgsql-db pg-meta meta    # Creating/modifying database auto-enables defined extensions

Result: Pigsty creates extension objects in the database, such as functions, types, operators, and index methods. You can then use the extension features.


Update Extensions

Extension updates involve two layers: package update and extension object update.

Update packages

pig
pig update pgvector                           # Update extension with pig
yum
sudo yum update pgvector_18 # EL
apt
sudo apt upgrade postgresql-18-pgvector  # Debian/Ubuntu

Update extension objects

-- View upgradeable extensions
SELECT name, installed_version, default_version FROM pg_available_extensions
WHERE installed_version IS NOT NULL AND installed_version <> default_version;

-- Update extension to latest version
ALTER EXTENSION vector UPDATE;

-- Update to specific version
ALTER EXTENSION vector UPDATE TO '0.8.1';
Update Notes

Back up the database before you update extensions. Preloaded extensions may require a PostgreSQL restart after an update. Some version upgrades may be incompatible. Check the extension documentation.


Remove Extensions

Removing extensions involves two layers: drop extension objects and uninstall packages.

Drop extension objects

DROP EXTENSION vector;              -- Drop extension
DROP EXTENSION vector CASCADE;      -- Cascade drop (drops dependent objects)

Remove from preload

For preloaded extensions, remove from shared_preload_libraries and restart:

pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'
pg restart pg-meta   # Restart to apply config

Uninstall packages (optional)

pig
pig remove pgvector                           # Uninstall with pig
yum
sudo yum remove pgvector_18*                  # EL systems
apt
sudo apt remove postgresql-18-pgvector        # Debian/Ubuntu
CASCADE Warning

CASCADE also deletes all objects that depend on the extension, such as tables, indexes, and views. Check dependencies before you run the command.


Query Extensions

Common SQL queries for extension info:

View enabled extensions

SELECT extname, extversion, nspname AS schema
FROM pg_extension e JOIN pg_namespace n ON e.extnamespace = n.oid
ORDER BY extname;

View available extensions

SELECT name, default_version, installed_version, comment
FROM pg_available_extensions
WHERE installed_version IS NOT NULL   -- Only show installed
ORDER BY name;

Check if extension is available

SELECT * FROM pg_available_extensions WHERE name = 'vector';

View extension dependencies

SELECT e.extname, d.refobjid::regclass AS depends_on
FROM pg_extension e
JOIN pg_depend d ON d.objid = e.oid
WHERE d.deptype = 'e' AND e.extname = 'postgis_topology';

View extension objects

SELECT classid::regclass, objid, deptype
FROM pg_depend
WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'vector');

psql shortcuts

\dx                    # List enabled extensions
\dx+ vector            # Show extension details

Add Repos

To install directly from upstream, manually add repos.

Using Pigsty playbook

./node.yml -t node_repo -e node_repo_modules=node,pgsql        # Add PGDG and Pigsty repos
./node.yml -t node_repo -e node_repo_modules=node,pgsql,local  # Including local repo

YUM repos (EL systems)

# Pigsty repo
curl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.io/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

# China mainland mirror
curl -fsSL https://repo.pigsty.cc/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.cc/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

APT repos (Debian/Ubuntu)

curl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql $(lsb_release -cs) main
EOF
sudo apt update

# China mainland mirror: replace repo.pigsty.io with repo.pigsty.cc

FAQ

Difference between extension name and package name

NameDescriptionExample
Extension nameName used with CREATE EXTENSIONvector
Package aliasStandardized name in Pigsty configpgvector
Package nameActual OS package namepgvector_18* or postgresql-18-pgvector

Preloaded extension prevents startup

If an extension in shared_preload_libraries is missing or cannot load, PostgreSQL will not start. Use one of these solutions:

  1. Ensure extension package is correctly installed
  2. Or remove extension from shared_preload_libraries (edit /pg/data/postgresql.conf)

Extension dependencies

Some extensions depend on other extensions. Create the dependencies first, or use CASCADE:

CREATE EXTENSION postgis;                    -- Create base extension first
CREATE EXTENSION postgis_topology;           -- Then create dependent extension
-- Or
CREATE EXTENSION postgis_topology CASCADE;   -- Auto-create dependencies

Extension version incompatibility

View extension versions supported by current PostgreSQL:

SELECT * FROM pg_available_extension_versions WHERE name = 'vector';

4.10 - Upgrading PostgreSQL Major/Minor Versions

Version upgrade - minor version rolling upgrade, major version migration, extension upgrade

Quick Start

PostgreSQL has two types of version upgrade: minor and major. Their risk and complexity differ substantially.

TypeExampleDowntimeData CompatibilityRisk
Minor upgrade17.2 → 17.3Seconds (rolling)Fully compatibleLow
Major upgrade17 → 18MinutesRequires data dir upgradeMedium
Minor
# Rolling upgrade: replicas first, then primary
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
pg restart --role replica --force <cls>
pg switchover <cls>
pg restart <cls> <old-primary> --force
Major
# Recommended: Logical replication migration
bin/pgsql-add pg-new              # Create new version cluster
# Configure logical replication to sync data...
# Switch traffic to new cluster
Extension
ansible <cls> -b -a 'yum upgrade -y postgis36_17*'
psql -c 'ALTER EXTENSION postgis UPDATE;'

For detailed online migration process, see Online Migration documentation.

ActionDescriptionRisk
Minor Version UpgradeUpdate packages, rolling restartLow
Minor Version DowngradeRollback to previous minor versionLow
Major Version UpgradeLogical replication or pg_upgradeMedium
Extension UpgradeUpgrade extension packages and objectsLow

Minor Version Upgrade

Minor version upgrades, such as 17.2 → 17.3, are the most common upgrade type. They usually contain security patches and bug fixes. The data directory remains fully compatible. Apply the upgrade with a rolling restart.

Strategy: Use a rolling upgrade to minimize service interruption. Upgrade replicas first. Then switch the primary role and upgrade the original primary.

1. Update repo → 2. Upgrade replica packages → 3. Restart replicas
4. Switchover → 5. Upgrade original primary packages → 6. Restart original primary

Step 1: Prepare packages

Verify that the local software repository has the latest PostgreSQL packages. Then refresh the node package cache:

Repo
cd ~/pigsty
./infra.yml -t repo_upstream      # Add upstream repos (needs internet)
./infra.yml -t repo_build         # Rebuild local repo
EL
ansible <cls> -b -a 'yum clean all'
ansible <cls> -b -a 'yum makecache'
Debian
ansible <cls> -b -a 'apt clean'
ansible <cls> -b -a 'apt update'

Step 2: Upgrade replicas

Upgrade packages on all replicas and verify version:

EL
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
ansible <cls> -b -a '/usr/pgsql/bin/pg_ctl --version'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17'
ansible <cls> -b -a '/usr/lib/postgresql/17/bin/pg_ctl --version'

Restart all replicas to apply new version:

pg restart --role replica --force <cls>

Step 3: Switchover

Execute switchover to transfer primary role to upgraded replica:

pg switchover <cls>
# Or non-interactive:
pg switchover --leader <old-primary> --candidate <new-primary> --scheduled=now --force <cls>

Step 4: Upgrade original primary

The original primary is now a replica. Upgrade its packages and restart it:

EL
ansible <old-primary-ip> -b -a 'yum upgrade -y postgresql17*'
Debian
ansible <old-primary-ip> -b -a 'apt install -y postgresql-17'
pg restart <cls> <old-primary-name> --force

Step 5: Verify

Confirm all instances have consistent version:

pg list <cls>
pg query <cls> -c "SELECT version()"

Minor Version Downgrade

In rare cases, you may need to return to the previous PostgreSQL version. For example, the new version may introduce bugs.

Step 1: Get old version packages

EL
cd ~/pigsty; ./infra.yml -t repo_upstream     # Add upstream repos
cd /www/pigsty; repotrack postgresql17-*-17.1 # Download specific version packages
cd ~/pigsty; ./infra.yml -t repo_create       # Rebuild repo metadata
Refresh Cache
ansible <cls> -b -a 'yum clean all'
ansible <cls> -b -a 'yum makecache'

Step 2: Execute downgrade

EL
ansible <cls> -b -a 'yum downgrade -y postgresql17*'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17=17.1*'

Step 3: Restart cluster

pg restart --force <cls>

Major Version Upgrade

Major version upgrades, such as 17 → 18, change the data format. They require specialized data migration tools.

MethodDowntimeComplexityUse Case
Logical Replication MigrationSeconds (switch)HighProduction, minimal downtime required
pg_upgrade In-Place UpgradeMinutes~HoursMediumTest env, smaller data
Recommended Approach

For production, use logical replication migration. Create a cluster with the new version and synchronize data through logical replication. Then switch traffic from the old cluster to the new cluster. This gives the shortest downtime and supports rollback. See Online Migration.

Logical Replication Migration

Logical replication is the recommended approach for production major version upgrades. Core steps:

1. Create new version target cluster → 2. Configure logical replication → 3. Verify data consistency
4. Switch app traffic to new cluster → 5. Decommission old cluster

Step 1: Create new version cluster

pg-meta-new:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta-new
    pg_version: 18                    # New version
bin/pgsql-add pg-meta-new

Step 2: Configure logical replication

-- Source cluster (old version) primary: create publication
CREATE PUBLICATION upgrade_pub FOR ALL TABLES;

-- Target cluster (new version) primary: create subscription
CREATE SUBSCRIPTION upgrade_sub
  CONNECTION 'host=10.10.10.11 port=5432 dbname=mydb user=replicator password=xxx'
  PUBLICATION upgrade_pub;

Step 3: Wait for sync completion

-- Target cluster: check subscription status
SELECT * FROM pg_stat_subscription;

-- Source cluster: check replication slot LSN
SELECT slot_name, confirmed_flush_lsn FROM pg_replication_slots;

Step 4: Switch traffic

After you verify that data synchronization is complete, stop application writes to the source. Wait for final synchronization, then move application connections to the new cluster. Drop the subscription and decommission the source:

-- Target cluster: drop subscription
DROP SUBSCRIPTION upgrade_sub;

For detailed migration process, see Online Migration documentation.

pg_upgrade In-Place Upgrade

pg_upgrade is the official PostgreSQL tool for major version upgrades. Use it for test environments or when longer downtime is acceptable.

Important Warning

An in-place upgrade causes longer downtime and is difficult to roll back. For production, prefer logical replication migration.

Step 1: Install new version packages

./pgsql.yml -l <cls> -t pg_pkg -e pg_version=18

Step 2: Stop Patroni

pg pause <cls>                        # Pause auto-failover
systemctl stop patroni                # Stop Patroni (stops PostgreSQL)

Step 3: Run pg_upgrade

sudo su - postgres
mkdir -p /data/postgres/pg-meta-18/data

# Pre-check (-c parameter: check only, don't execute)
/usr/pgsql-18/bin/pg_upgrade \
  -b /usr/pgsql-17/bin -B /usr/pgsql-18/bin \
  -d /data/postgres/pg-meta-17/data \
  -D /data/postgres/pg-meta-18/data \
  -v -c

# Execute upgrade
/usr/pgsql-18/bin/pg_upgrade \
  -b /usr/pgsql-17/bin -B /usr/pgsql-18/bin \
  -d /data/postgres/pg-meta-17/data \
  -D /data/postgres/pg-meta-18/data \
  --link -j 8 -v

Step 4: Update links and start

rm -rf /usr/pgsql && ln -s /usr/pgsql-18 /usr/pgsql
rm -rf /pg && ln -s /data/postgres/pg-meta-18 /pg
# Edit /etc/patroni/patroni.yml to update paths
systemctl start patroni
pg resume <cls>

Step 5: Post-processing

/usr/pgsql-18/bin/vacuumdb --all --analyze-in-stages
./delete_old_cluster.sh   # Cleanup script generated by pg_upgrade

Extension Upgrade

When you upgrade PostgreSQL, you usually also need to upgrade its extensions.

Upgrade extension packages

EL
ansible <cls> -b -a 'yum upgrade -y postgis36_17 timescaledb-2-postgresql-17* pgvector_17*'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17-postgis-3 postgresql-17-pgvector'

Upgrade extension objects

After package upgrade, execute extension upgrade in database:

-- View upgradeable extensions
SELECT name, installed_version, default_version FROM pg_available_extensions
WHERE installed_version IS NOT NULL AND installed_version <> default_version;

-- Upgrade extensions
ALTER EXTENSION postgis UPDATE;
ALTER EXTENSION timescaledb UPDATE;
ALTER EXTENSION vector UPDATE;

-- Check extension versions
SELECT extname, extversion FROM pg_extension;
Extension Compatibility

Before a major version upgrade, verify that all extensions support the target PostgreSQL version. Some extensions may require removal and reinstallation. Check the extension documentation.


Important Notes

  1. Back up first: Make a complete backup before every upgrade.
  2. Test first: Verify the upgrade procedure in a test environment.
  3. Check extensions: Verify that all extensions support the target version.
  4. Plan rollback: Prepare a rollback plan, especially for major upgrades.
  5. Monitor closely: Monitor database performance and error logs after the upgrade.
  6. Keep records: Record all operations and problems during the upgrade.

5 - Backup & Restore

Configure repositories and policies, manage pgBackRest backups, and perform point-in-time recovery safely.

Pigsty uses pgBackRest for PostgreSQL backups. It supports full, differential, and incremental backups, parallel processing, encryption, and Silo/S3 object storage. By default, Pigsty configures backup and WAL archiving for every PGSQL cluster.

This chapter is the operational manual for backup configuration, management, recovery, and drills. For design concepts and tradeoffs, see Point-in-Time Recovery.

All backup and recovery operations call pgBackRest. Pigsty provides these wrapper layers:

LayerInterfaceFormScope
Cluster orchestrationpg_pitr + pgsql-pitr.ymlAnsible playbookHA, etcd, and multi-node recovery
Instance orchestrationpig pitrCLILocal-node recovery without the admin node
Command primitivespig pb, pb, and pg-backuppgBackRest wrappersBackup, inspection, expiry, and unmanaged restore
EnginepgbackrestNative CLIUnderlying backup, archive, and restore engine
SectionContent
MechanismStanzas, repositories, retention, timelines, and Pigsty wrapper mapping
PolicyScheduling, recovery windows, and storage planning
RepositoryLocal, Silo, and external S3 repositories; encryption, versioning, and locking
AdministrationBackup commands, inspection, expiration, and stanza management
RestoreRecovery targets, staged PITR, and complete parameter reference
CloneRestore production history into another cluster and perform drills
TutorialA sandbox restore using pgBackRest primitives
Disclaimer

Pigsty makes every effort to provide a reliable PITR solution. It accepts no liability for data loss caused by restore operations. If you need assistance, consider professional services.

Recovery overwrites target data

Before PITR, inspect pig pg list <target-cluster> and pig pb info. Verify a recent usable backup and its recovery window. Have the operator state the exact target cluster and recovery point. Then run the target-scoped ./pgsql-pitr.yml -l <target-cluster> ... command. pgsql-pitr.yml prints a plan but does not pause for approval. Production recovery also requires a maintenance window and an independently verified backup.


Quick Start

  1. Design a backup policy: declare scheduled backups in pg_crontab and select a repository with pgbackrest_repo.
  2. Manage backups: run pg-backup and inspect recovery coverage with pb info.
  3. Perform recovery: declare pg_pitr, then run pgsql-pitr.yml.
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'

5.1 - Backup Mechanism

pgBackRest concepts—stanzas, repositories, backup chains, retention, and timelines—and how Pigsty maps parameters to commands.

Pigsty’s backup and restore operations execute pgBackRest commands. For safe operation, understand the pgBackRest model and how Pigsty maps its orchestration parameters to native options.


Core pgBackRest Concepts

Stanza: the Cluster’s Backup Identity

A stanza names one PostgreSQL backup configuration and isolates that cluster inside a repository. Pigsty uses pg_cluster as the stanza name. The pg-meta stanza stores data under backup/pg-meta/ and archive/pg-meta/. Thus, several clusters can share one repository.

The stanza records the source system identifier and major version. pgBackRest checks these values before a backup. Because of this check, a cross-cluster clone needs stanza-upgrade afterward. Pigsty creates the stanza during cluster initialization. stanza-upgrade updates it after a major-version change or clone.

Repository: Where Backups Live

A repository stores backup files and WAL archives. repo1-type selects POSIX, S3, Azure, GCS, or SFTP. repo1-path, repo1-cipher-*, and repo1-retention-* define location, encryption, and retention. Pigsty generates these settings from pgbackrest_repo. See Backup Repository.

Backup Chains and Labels

TypeContentsLabel suffix
FullComplete database-cluster copyF
DifferentialChanges since the latest fullD
IncrementalChanges since the latest backup of any typeI

Labels identify the backup chain. 20250715-013657F is a full backup. 20250715-013657F_20250715-013724D and ..._20250715-013730I depend on the full backup identified before the underscore. Use --set to select the starting backup explicitly. Otherwise, pgBackRest selects the newest usable set before the target.

Retention: Keeping the Repository Bounded

repo1-retention-full and repo1-retention-full-type (count or time) determine when full backup chains expire. Dependent differential/incremental backups and WAL needed only by that chain expire with the full backup. Pigsty enables expire-auto. Use pig pb expire --plan to preview a manual run.

Time retention defines a minimum window. It does not remove every full backup older than N days. An old full backup expires only when another retained full backup has reached that age. With weekly full backups, a 14-day setting therefore commonly retains three full chains and approximately 14–21 days of history.

WAL Archiving

PostgreSQL calls archive-push when a WAL segment fills or archive_timeout elapses. During recovery, restore_command calls archive-get. Pigsty enables asynchronous archiving through /pg/spool. Thus, a temporary repository delay does not block the primary directly.

Timelines

Each promotion after recovery or failover creates a new timeline. Older timeline history remains in the repository. --target-timeline selects the recovery branch and defaults to latest. See the conceptual PITR mechanism.

What restore Actually Does

restore first reconstructs the data directory. Pigsty enables --delta, so pgBackRest validates existing files and rewrites only files that differ. It then writes the recovery state (recovery.signal, restore_command, and recovery_target_*). WAL replay starts after PostgreSQL starts.

A successful restore command does not complete PITR. --target-action controls what happens when replay reaches the target: pause, promote, or shutdown.

pgbackrest --stanza=pg-meta restore
pgbackrest --stanza=pg-meta --type=immediate restore
pgbackrest --stanza=pg-meta --type=time --target='2025-07-13 10:00:00+00' restore
pgbackrest --stanza=pg-meta --type=xid --target='250000' --target-exclusive restore
pgbackrest --stanza=pg-meta --type=name --target='my-restore-point' restore
pgbackrest --stanza=pg-meta --type=lsn --target='0/4001C80' --target-action=promote restore

Observe a Backup Chain

Run read-only info after full, differential, and incremental backups to inspect labels, size, WAL bounds, and references:

pig pb info

A representative sequence looks like:

full backup: 20250715-013657F
diff backup: 20250715-013657F_20250715-013724D
    backup reference total: 1 full
incr backup: 20250715-013657F_20250715-013730I
    backup reference total: 1 full, 1 diff

Pigsty’s Wrapper Layers

LayerInterfaceWhat it does
Cluster orchestrationpg_pitr + pgsql-pitr.ymlPause HA, stop nodes, render configuration, restore/replay, inspect control data, clean etcd, and rebuild HA
Instance orchestrationpig pitrPreflight, keep one target offline, restore, optionally start PostgreSQL, and leave Patroni stopped for inspection
Command primitivepig pb, pb, pg-backupSupply stanza/DBSU context and call the corresponding pgBackRest command
EnginepgbackrestRead /etc/pgbackrest/pgbackrest.conf and perform backup, archive, and restore operations

Command Primitives

pb is a login-shell function that reads the first stanza from the local configuration and forwards arguments:

pb() (
    stanza=$(grep -o '\[[^][]*]' /etc/pgbackrest/pgbackrest.conf | head -n1 | sed 's/.*\[\([^]]*\)].*/\1/')
    pgbackrest --stanza="${stanza}" "$@"
)
pb info     # pgbackrest --stanza=pg-meta info
pb backup   # pgbackrest --stanza=pg-meta backup

pg-backup adds a primary-role check for scheduled use:

pg-backup full
pg-backup diff
pg-backup incr   # the default; pgBackRest promotes it when no full exists

pig pb adds stanza detection, DBSU privilege handling, primary checks for backup, and plan/confirmation guards for destructive primitives. See Admin Commands.

Parameter Mapping

pg_pitr fieldpig pitr optionpgBackRest optionMeaning
cluster--stanza--stanzaSource cluster/stanza
type plus time/xid/lsn/namecorresponding target option--type + --targetRecovery target
default--defaultno --type/--targetReplay to archive end
immediate--immediate--type=immediateStop at the first consistent point
exclusive--exclusive / -X--target-exclusiveStop before the target
action--target-action--target-actionpause, promote, or shutdown
timeline--target-timeline / -T--target-timelineTarget timeline
set--set / -b--setStarting backup set
db_include / db_exclude—--db-include / --db-excludeSelect databases in a physical restore
link_map—--link-mapRemap directory or tablespace links
process—process-maxParallel restore processes
data--data / -D--pg1-pathTarget data directory
reporepository number only in pig pitrrendered repo1-*Override repository definition in the playbook

A selective restore is still a physical restore. Excluded databases receive sparse zeroed files so PostgreSQL can complete recovery. These databases are inaccessible. You must remove them explicitly afterward. This is not a logical subset as with pg_dump.

How Configuration Is Rendered

pgbackrest_method selects the entry that Pigsty writes to /etc/pgbackrest/pgbackrest.conf. Pigsty converts underscores to hyphens and adds the repo1- prefix to keys.

pgbackrest_repo:
  minio:
    type: s3                  # repo1-type=s3
    s3_endpoint: sss.pigsty   # repo1-s3-endpoint=sss.pigsty
    cipher_type: aes-256-cbc  # repo1-cipher-type=aes-256-cbc
    retention_full: 14        # repo1-retention-full=14

pgsql-pitr.yml generates a separate temporary configuration at /pg/conf/pitr.conf. PostgreSQL recovery output goes to /pg/tmp/recovery.log.


Scheduled Backups

Pigsty installs pg_crontab entries for the postgres OS user on every cluster node. pg-backup checks the current role, so only the primary makes backups. After promotion, the new primary runs future scheduled backups.

pg_crontab:
  - '00 01 * * 1 /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
./pgsql.yml -t pg_crontab -l pg-meta

See Backup Policy for frequency and retention design.


Deployment Details

The pg_backup subtask installs and configures pgBackRest, then creates the stanza. If pgbackrest_init_backup is enabled, it attempts an initial full backup. It writes /etc/pgbackrest/initial.done only after that backup succeeds.

PathPurpose
/usr/bin/pgbackrestpgBackRest binary
/etc/pgbackrest/pgbackrest.confMain stanza and repository configuration
/pg/backupLocal repository path
/pg/spoolAsynchronous archive spool
/pg/log/pgbackrest/Backup, archive, and restore logs
/pg/conf/pitr.confTemporary PITR configuration
/pg/tmp/recovery.logPostgreSQL recovery log

pgbackrest_exporter exports backup metrics and listens on pgbackrest_exporter_port, which defaults to 9854. To disable it, use pgbackrest_exporter_enabled. To customize it, use pgbackrest_exporter_options.

5.2 - Backup Policy

Design backup policies according to your needs

The chart below shows the “Recovery Window” and “Backup Storage Usage” on one timeline (0~108h). Use it to compare the two metrics.

The 30-day example uses a database size of 100GB and daily writes of 10GB. It shows both metrics with weekly full backups, daily incremental backups, and 14-day full-backup retention.

  • When: Backup schedule
  • Where: Backup repository
  • How: Backup method

When to Backup

First, choose when to back up the database. Backup frequency affects recovery time. Recovery replays WAL from the last backup to the target point. More frequent backups reduce the amount of WAL to replay and shorten recovery.

Daily Full Backup

For production databases, we recommend a daily full backup as the simplest initial policy. This is also Pigsty’s default backup policy. Pigsty implements it through crontab.

pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
pgbackrest_method: local          # Choose backup repository method: `local`, `minio`, or other custom repository
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  local:                          # Default pgbackrest repository using local POSIX filesystem
    path: /pg/backup              # Local backup directory, defaults to `/pg/backup`
    retention_full_type: count    # Retain full backups by count
    retention_full: 2             # Keep 2, up to 3 full backups when using local filesystem repository

Assume a database size of 100GB, daily writes of 10GB, and a duration of 1 hour for each full backup. The chart below shows the recovery window and storage usage for this daily full backup policy with a local repository.

The recovery window cycles between 25-49 hours, and storage usage is roughly 2 full backups plus around 2 days of WAL archives. In practice, prepare at least 3~5 times the base database size as backup disk capacity for the default policy.

tooltip: { trigger: axis, formatter: $fn:tipMerged, axisPointer: { type: line, snap: true, label: { show: false } } }
axisPointer: { link: [ { xAxisIndex: [0, 1] } ] }
legend: { show: false, bottom: 10, itemGap: 18, data: ["Primary Backup", "Secondary Backup", "WAL Archive", "Transient Backup"] }
grid:
  - { left: 82, right: "10%", top: 42, height: 218, containLabel: false }
  - { left: 82, right: "10%", top: 286, height: 218, containLabel: false }
xAxis:
  - type: category
    gridIndex: 0
    position: bottom
    boundaryGap: false
    data: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108]
    name: Time h
    nameLocation: end
    nameGap: 10
    nameTextStyle: { align: left, verticalAlign: top, padding: [8, 0, 0, 0] }
    axisLabel: { interval: 11, formatter: $fn:fmtHour }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.28, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 12, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.14, color: "#9ca3af" } }
  - type: category
    gridIndex: 1
    position: top
    boundaryGap: true
    data: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108]
    axisLabel: { show: false }
    axisLine: { show: true, lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.22, color: "#9ca3af" } }
yAxis:
  - type: value
    gridIndex: 0
    min: 0
    max: 52
    interval: 5
    name: Recovery Window h
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: bottom, padding: [0, 0, 8, 4] }
    axisLabel: { formatter: $fn:fmtWin }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.35, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 5, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.18, color: "#9ca3af" } }
  - type: value
    gridIndex: 1
    min: 0
    max: 350
    interval: 50
    inverse: true
    name: Backup Storage GB
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: top, padding: [10, 0, 0, 4] }
    axisLabel: { formatter: $fn:fmtGbTick }
    axisLine: { show: true, symbol: [arrow, none], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.32, color: "#9ca3af" } }
series: [ { name: Recovery Window, type: line, smooth: false, symbol: none, showSymbol: false, xAxisIndex: 0, yAxisIndex: 0, lineStyle: { width: 3, color: "#f2a000" }, itemStyle: { color: "#f2a000" }, data: [[0,0],[1,0],[1,1],[2,2],[3,3],[4,4],[5,5],[6,6],[7,7],[8,8],[9,9],[10,10],[11,11],[12,12],[13,13],[14,14],[15,15],[16,16],[17,17],[18,18],[19,19],[20,20],[21,21],[22,22],[23,23],[24,24],[25,25],[26,26],[27,27],[28,28],[29,29],[30,30],[31,31],[32,32],[33,33],[34,34],[35,35],[36,36],[37,37],[38,38],[39,39],[40,40],[41,41],[42,42],[43,43],[44,44],[45,45],[46,46],[47,47],[48,48],[49,49],[49,25],[50,26],[51,27],[52,28],[53,29],[54,30],[55,31],[56,32],[57,33],[58,34],[59,35],[60,36],[61,37],[62,38],[63,39],[64,40],[65,41],[66,42],[67,43],[68,44],[69,45],[70,46],[71,47],[72,48],[73,49],[73,25],[74,26],[75,27],[76,28],[77,29],[78,30],[79,31],[80,32],[81,33],[82,34],[83,35],[84,36],[85,37],[86,38],[87,39],[88,40],[89,41],[90,42],[91,43],[92,44],[93,45],[94,46],[95,47],[96,48],[97,49],[97,25],[98,26],[99,27],[100,28],[101,29],[102,30],[103,31],[104,32],[105,33],[106,34],[107,35],[108,36]], markLine: { symbol: none, label: { show: false }, data: [ { xAxis: 0, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 24, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 48, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 72, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 96, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 1, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 25, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 49, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 73, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 97, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { yAxis: 25, label: { show: true, formatter: "lower 25h", position: "end", distance: 12, color: "#2563eb" }, lineStyle: { color: "#2563eb", type: "dashdot", width: 1.4, opacity: 0.75 } }, { yAxis: 49, label: { show: true, formatter: "upper 49h", position: "end", distance: 12, color: "#7c3aed" }, lineStyle: { color: "#7c3aed", type: "dashdot", width: 1.4, opacity: 0.75 } } ] } }, { name: Primary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#59a14f" }, data: [0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }, { name: Secondary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#4e79a7" }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }, { name: WAL Archive, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#edc949" }, data: [0,0,0.42,0.83,1.25,1.67,2.08,2.5,2.92,3.33,3.75,4.17,4.58,5,5.42,5.83,6.25,6.67,7.08,7.5,7.92,8.33,8.75,9.17,9.58,10,10.42,10.83,11.25,11.67,12.08,12.5,12.92,13.33,13.75,14.17,14.58,15,15.42,15.83,16.25,16.67,17.08,17.5,17.92,18.33,18.75,19.17,19.58,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20] }, { name: Transient Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#9ca3af", opacity: 0.75 }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0] } ]

Full + Incremental Backup

You can optimize backup space usage by adjusting these parameters.

If using Silo / S3 as a centralized backup repository, storage is no longer limited by the local disk. In this case, consider using full + incremental backup with a 2-week retention policy:

pg_crontab:  # Full backup at 1 AM on Monday, incremental backups on weekdays
  - '00 01 * * 1           /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
pgbackrest_method: minio
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  minio:                          # Optional S3-compatible repository preset
    type: s3                      # Silo uses the S3-compatible repository type
    s3_endpoint: sss.pigsty       # object-storage endpoint, `sss.pigsty` by default
    s3_region: us-east-1          # compatibility region, `us-east-1` by default
    s3_bucket: pgsql              # backup bucket, `pgsql` by default
    s3_key: pgbackrest            # pgBackRest access key
    s3_key_secret: S3User.Backup  # object-storage user secret
    s3_uri_style: path            # use path-style rather than host-style URIs
    path: /pgbackrest             # backup path, `/pgbackrest` by default
    storage_port: 9000            # Silo service port, 9000 by default
    storage_ca_file: /etc/pki/ca.crt  # Pigsty CA path, `/etc/pki/ca.crt` by default
    block: y                      # Enable block-level incremental backup
    bundle: y                     # Bundle small files into a single file
    bundle_limit: 20MiB           # Bundle size limit, recommended 20MiB for object storage
    bundle_size: 128MiB           # Bundle target size, recommended 128MiB for object storage
    cipher_type: aes-256-cbc      # Enable AES encryption for remote backup repository
    cipher_pass: pgBackRest       # AES encryption password, defaults to 'pgBackRest'
    retention_full_type: time     # Retain full backups by time
    retention_full: 14            # Keep full backups from the last 14 days

With weekly full backups and 14-day time-based retention, the steady-state recovery window is approximately 14–21 days. The exact window depends on successful backups and WAL archiving.

For a database of 100GB with daily writes of 10GB, the following chart shows backup size:

tooltip: { trigger: axis, formatter: $fn:tipMerged30, axisPointer: { type: line, snap: true, label: { show: false } } }
axisPointer: { link: [ { xAxisIndex: [0, 1] } ] }
legend: { show: false, bottom: 10, itemGap: 18, data: ["Primary Backup", "Secondary Backup", "Incremental Backup", "WAL Archive", "Transient Backup"] }
grid:
  - { left: 82, right: "10%", top: 42, height: 218, containLabel: false }
  - { left: 82, right: "10%", top: 302, height: 218, containLabel: false }
xAxis:
  - type: value
    gridIndex: 0
    position: bottom
    boundaryGap: false
    min: 0
    max: 31
    interval: 1
    name: Time
    nameLocation: end
    nameGap: 10
    nameTextStyle: { align: left, verticalAlign: top, padding: [8, 0, 0, 0] }
    axisLabel: { formatter: $fn:fmtDay30 }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.28, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 4, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.14, color: "#9ca3af" } }
  - type: category
    gridIndex: 1
    position: top
    boundaryGap: true
    z: 10
    data: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]
    axisLabel: { show: false }
    axisLine: { show: true, lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, alignWithLabel: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.22, color: "#9ca3af" } }
yAxis:
  - type: value
    gridIndex: 0
    min: 0
    max: 360
    interval: 48
    name: Recovery Window h
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: bottom, padding: [0, 0, 8, 4] }
    axisLabel: { formatter: $fn:fmtWin30 }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.35, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 4, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.18, color: "#9ca3af" } }
  - type: value
    gridIndex: 1
    min: 0
    max: 600
    interval: 50
    inverse: true
    z: 10
    name: Storage GB
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: top, padding: [10, 0, 0, 4] }
    axisLabel: { formatter: $fn:fmtGbTick30 }
    axisLine: { show: true, symbol: [arrow, none], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.32, color: "#9ca3af" } }
series:
  - { name: Recovery Window, type: line, smooth: false, symbol: none, showSymbol: false, xAxisIndex: 0, yAxisIndex: 0, lineStyle: { width: 3, color: "#f28e2c" }, itemStyle: { color: "#f28e2c" }, data: [[1,24],[2,48],[3,72],[4,96],[5,120],[6,144],[7,168],[8,192],[9,216],[10,240],[11,264],[12,288],[13,312],[14,336],[14,168],[15,192],[16,216],[17,240],[18,264],[19,288],[20,312],[21,336],[21,168],[22,192],[23,216],[24,240],[25,264],[26,288],[27,312],[28,336],[28,168],[29,192],[30,216]], markLine: { symbol: none, label: { show: false }, data: [ { xAxis: 7, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 14, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 21, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 28, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { yAxis: 168, label: { show: true, formatter: "lower 7d", position: "end", distance: 12, color: "#2563eb" }, lineStyle: { color: "#2563eb", type: "dashdot", width: 1.4, opacity: 0.72 } }, { yAxis: 336, label: { show: true, formatter: "upper 14d", position: "end", distance: 12, color: "#7c3aed" }, lineStyle: { color: "#7c3aed", type: "dashdot", width: 1.4, opacity: 0.72 } } ] } }
  - { name: Primary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#59a14f" }, data: [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }
  - { name: Secondary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#4e79a7" }, data: [0,0,0,0,0,0,0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }
  - { name: Incremental Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#76b7b2" }, data: [0,10,20,30,40,50,60,70,80,90,100,110,120,130,70,80,90,100,110,120,130,70,80,90,100,110,120,130,70,80] }
  - { name: WAL Archive, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#edc949" }, data: [10,20,30,40,50,60,70,80,90,100,110,120,130,140,80,90,100,110,120,130,140,80,90,100,110,120,130,140,80,90] }
  - { name: Transient Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#9ca3af", opacity: 0.75 }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,0,0,0,0,0,110,0,0,0,0,0,0,110,0,0] }

Backup Location

Pigsty provides two default backup repository definitions: local and minio.

  • local: Default option, uses local /pg/backup directory (symlink to pg_fs_backup: /data/backups)
  • minio: Uses SNSD single-node MinIO cluster (supported by Pigsty, but not enabled by default)
pgbackrest_method: local          # Choose backup repository method: `local`, `minio`, or other custom repository
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  local:                          # Default pgbackrest repository using local POSIX filesystem
    path: /pg/backup              # Local backup directory, defaults to `/pg/backup`
    retention_full_type: count    # Retain full backups by count
    retention_full: 2             # Keep 2, up to 3 full backups when using local filesystem repository
  minio:                          # Optional minio repository
    type: s3                      # minio is S3 compatible
    s3_endpoint: sss.pigsty       # minio endpoint domain, defaults to `sss.pigsty`
    s3_region: us-east-1          # minio region, defaults to us-east-1, meaningless for minio
    s3_bucket: pgsql              # minio bucket name, defaults to `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret for pgbackrest
    s3_uri_style: path            # minio uses path-style URIs instead of host-style
    path: /pgbackrest             # minio backup path, defaults to `/pgbackrest`
    storage_port: 9000            # minio port, defaults to 9000
    storage_ca_file: /etc/pki/ca.crt  # minio CA certificate path, defaults to `/etc/pki/ca.crt`
    block: y                      # Enable block-level incremental backup
    bundle: y                     # Bundle small files into a single file
    bundle_limit: 20MiB           # Bundle size limit, recommended 20MiB for object storage
    bundle_size: 128MiB           # Bundle target size, recommended 128MiB for object storage
    cipher_type: aes-256-cbc      # Enable AES encryption for remote backup repository
    cipher_pass: pgBackRest       # AES encryption password, defaults to 'pgBackRest'
    retention_full_type: time     # Retain full backups by time
    retention_full: 14            # Keep full backups from the last 14 days

5.3 - Backup Repository

Configure local, Silo, and external S3 backup repositories, including retention, encryption, versioning, and Object Lock.

Two parameters select the backup location. pgbackrest_repo defines candidate repositories. pgbackrest_method selects one repository. Pigsty converts repository keys to pgBackRest repo1-* options, so you can use any supported pgBackRest repository option directly. Pigsty v4.5.0 writes only the entry selected by pgbackrest_method as repo1. Multiple candidate keys do not enable multi-repository backup.


Default Repositories

Pigsty supplies two definitions: local and minio.

  • local is the default. /pg/backup points at pg_fs_backup, /data/backups by default.
  • minio uses Silo from the MINIO module or another compatible S3 service. It is supported but not selected by default.
pgbackrest_method: local
pgbackrest_repo:
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2
  minio:
    type: s3
    s3_endpoint: sss.pigsty
    s3_region: us-east-1
    s3_bucket: pgsql
    s3_key: pgbackrest
    s3_key_secret: S3User.Backup
    s3_uri_style: path
    path: /pgbackrest
    storage_port: 9000
    storage_ca_file: /etc/pki/ca.crt
    block: y
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: pgBackRest
    retention_full_type: time
    retention_full: 14

The presets use different settings. local provides simple, fast restores with count-based retention and no encryption or bundling. minio enables AES-256-CBC encryption, bundles small files, uses block incremental backup, and retains full backups by time.

Replace public defaults in production

For a remote repository, replace both cipher_pass and s3_key_secret. pgBackRest and S3User.Backup are public example defaults. If you lose the encryption passphrase, you cannot recover the repository. Store the passphrase separately from the backups under controlled recovery procedures. See Deployment Security.


Retention Policy

pgBackRest applies retention after each backup (expire-auto). When a full backup expires, its dependent differential/incremental backups expire with it. WAL needed only by that chain also expires.

  • retention_full_type: count with retention_full: 2 keeps the two newest full chains. A third chain can exist briefly while a new full backup completes.
  • retention_full_type: time with retention_full: 14 defines a minimum time window. An old full backup expires only when another full backup is at least 14 days old. With weekly full backups, this usually retains three chains and approximately 14–21 days of recovery history.

See Backup Policy for recovery-window and space calculations, and Admin Commands for a safe expiration preview.


Use a Silo Repository

The MINIO module currently deploys Silo, an S3-compatible object store. It provides an independent disaster-recovery copy only if it is outside the database host or site failure domain. Deploy it, then select the minio preset:

all:
  vars:
    pgbackrest_method: minio
  children:
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio, minio_type: silo }}

By default, the preset uses the HTTPS endpoint sss.pigsty and validates it with /etc/pki/ca.crt. MINIO initialization creates the default pgsql bucket and pgbackrest user.

For production, deploy and test a fault-tolerant object store with multiple nodes and drives. See MINIO Configuration. The minio preset also supports servers that Pigsty does not manage, including independent MinIO, RustFS, and other compatible implementations. For these servers, installation, upgrades, certificates, and lifecycle management remain outside the MINIO role support scope.


Use S3 or Cloud Object Storage

For a single database node, an off-host cloud object store is often the most useful repository. Define a new entry and select it:

pgbackrest_method: s3
pgbackrest_repo:
  s3:
    type: s3
    s3_endpoint: oss-cn-beijing-internal.aliyuncs.com
    s3_region: oss-cn-beijing
    s3_bucket: <your_bucket_name>
    s3_key: <your_access_key>
    s3_key_secret: <your_secret_key>
    s3_uri_style: host
    path: /pgbackrest
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: <separate-strong-passphrase>
    retention_full_type: time
    retention_full: 14
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2

pgBackRest also supports Azure, GCS, and SFTP repositories.


Share a Repository Across Clusters

A centralized repository can serve several PostgreSQL clusters. Each stanza takes its name from pg_cluster and isolates one cluster’s backup and archive history. This also enables cross-cluster restore.

Cluster names must therefore be globally unique within a shared repository, even across otherwise separate deployment environments.


Repository Versioning

Object-store versioning can preserve earlier versions after an overwrite or deletion. These versions share the same storage system and control plane. Thus, versioning does not replace an independent off-site or offline copy. Enable versioning when you create a bucket:

minio_buckets:
  - { name: pgsql, versioning: true }
  - { name: meta, versioning: true }
  - { name: data }

pgBackRest’s repo-target-time option can read the repository as it existed at an earlier time when the backend retains those versions.


Repository Locking

Some S3-compatible services support Object Lock/WORM. You cannot change or permanently delete a retained object version until its retention period ends. A normal delete can still create a delete marker that hides the current object. Historical versions remain and continue to consume capacity.

The lock flag enables versioning and lock capability only when Pigsty creates the bucket:

minio_buckets:
  - { name: pgsql, lock: true }
  - { name: meta, versioning: true }
  - { name: data }

The flag does not set a default retention period. Configure GOVERNANCE or COMPLIANCE retention with mcli retention set or the storage console. Then verify it with mcli retention info. A sufficiently privileged principal can bypass GOVERNANCE. Even root cannot shorten COMPLIANCE retention.

Locking changes expiration and stanza deletion. pgBackRest may expire objects logically while retained historical versions remain until their deadline. Before you enable locking, test backup, expiration, delete-marker cleanup, and version recovery on a non-production bucket.


Switch Repositories

After you change a repository definition or pgbackrest_method, generate the configuration again and initialize the stanza. Promptly create a recovery point in the new repository:

./pgsql.yml -t pg_backup -l pg-meta           # Apply the repository definition and create the stanza
sudo -iu postgres pg-backup full              # Run on the current primary to establish the first recovery point

Pigsty does not migrate existing backups automatically. While you retain the old repository, you can select it as a restore source through pg_pitr.repo.

5.4 - Admin Commands

Backup administration reference covering setup and removal, manual backups, inspection, expiration, stanza management, logs, and alternative tools.

On a database node, run backup commands as the database superuser (pg_dbsu, which defaults to postgres. Use one of these interfaces:

  • pig pb: the PIG CLI wrapper, with automatic stanza detection, DBSU switching, and safety checks; this is the recommended interface
  • pb: a login-shell function that supplies --stanza and forwards arguments to pgBackRest
  • pgbackrest: the native command; see the pgBackRest command reference

Command Overview

PIG commandAliasNative pgBackRest commandPurpose
pig pb infoiinfoShow backup and archive status
pig pb listls—List repositories, stanzas, or backup sets
pig pb backup [full/diff/incr]bbackupCreate a backup after checking the primary role
pig pb restorerrestoreLow-level restore primitive; see Restore Operations
pig pb expireeexpireExpire backups according to retention (--plan previews)
pig pb createcstanza-createCreate a stanza
pig pb upgradeustanza-upgradeUpgrade a stanza after a major-version change or clone
pig pb deletedstanza-deleteDelete a stanza and all of its backups
pig pb checkckcheckVerify configuration, repository access, and archiving
pig pb startupstartRe-enable pgBackRest operations
pig pb stopdwstopStop new pgBackRest operations
pig pb log [list/show/tail]l—Inspect pgBackRest logs

Enable Backup

If pgbackrest_enabled is true at cluster creation, Pigsty enables backup automatically. If backup was disabled at creation, or repository settings have changed, run the pg_backup subtask:

./pgsql.yml -t pg_backup -l pg-meta   # Configure pgBackRest and create the stanza

After cluster initialization, Pigsty attempts an initial full backup. It writes /etc/pgbackrest/initial.done only after the backup command succeeds. If the attempt fails, the playbook ignores the failure and leaves no marker. This file only prevents the initialization task from repeating. Always verify repository state with pig pb info or pgbackrest info. Define scheduled backups with pg_crontab; see Backup Policy.


Remove Backup

To remove only a backup stanza, use pig pb delete as the preferred interface. It asks for interactive confirmation. With a multi-stanza configuration, you must also specify the target explicitly. Verify the exact target first:

pig pb info -s pg-meta           # Verify the backup chain and recovery window
pig pb delete -s pg-meta         # Executes only after typing the exact stanza name

By default, pgsql-rm.yml also tries to delete the cluster’s backup stanza when it removes a primary instance (pg_role = primary). Each command below changes or deletes state. Do not run a command only because it appears in an example:

./pgsql-rm.yml -l pg-meta                          # Remove the cluster and its backups
./pgsql-rm.yml -l pg-meta -e pg_rm_backup=false    # Remove the cluster while preserving backups
./pgsql-rm.yml -l pg-meta -t pg_backup             # Remove backup-related state only

Before execution, verify a recent usable backup and record the recovery requirement. Have the operator re-enter the exact cluster/stanza name. To preserve backups when you remove the cluster, set pg_rm_backup to false.

pgsql-rm.yml -t pg_backup forcibly runs pgbackrest stanza-delete on the primary. In local mode, it removes the local repository directory. It then removes the pgBackRest configuration and initial-backup marker. The task ignores some deletion errors. Thus, a successful playbook result does not prove that it physically removed the repository objects. To delete only a stanza, prefer pig pb delete, which provides a plan and requires confirmation.

With object versioning and object-lock retention, deletion may create a delete marker. Locked historical versions continue to consume storage until their retention period expires.

Backup Deletion

Deleting backups can permanently destroy recovery options. Confirm the cluster/stanza. Verify a recent backup and an alternative recovery copy. Keep the pig pb info output and deletion plan as an audit record.


Manual Backup

You can start a backup outside the crontab schedule. Both pg-backup and pig pb backup check that the current instance is primary. They exit on a replica:

pg-backup            # Incremental; pgBackRest promotes it to full if no full backup exists
pg-backup full       # Full backup
pg-backup diff       # Differential backup relative to the latest full
pg-backup incr       # Incremental backup relative to the latest backup

pig pb backup full   # Equivalent PIG wrapper with stanza and DBSU detection

Backup consumes disk I/O and network bandwidth. Pigsty limits parallelism to a small number of processes. Still, you should schedule production backups for periods with low traffic.


Inspect Backups

pb info shows backups and WAL archive status for the current stanza:

pb info          # pgbackrest --stanza=pg-meta info
pig pb info      # Equivalent PIG wrapper
pig pb list      # List backup sets; use "list stanza" for all stanzas

Backup labels that end in F, D, and I identify full, differential, and incremental backups. The portion before an underscore identifies the full backup at the start of that chain. The WAL archive range and oldest usable full backup define the recovery window.

The pgbackrest_exporter service on port 9854 continuously exports metrics such as the latest backup time, type, size, and error status.


Expire Old Backups

pgBackRest applies the retention policy automatically after backups (expire-auto). To preview or run expiration manually, use:

pig pb expire --plan    # Show what would be removed; no deletion
pig pb expire           # Apply the configured retention policy

Stanza Management

A stanza records a cluster’s backup identity, including its system identifier and major version. You may need to manage it manually:

pig pb create                    # Create a stanza; cluster initialization normally does this
pig pb upgrade                   # Update after a major-version upgrade or clone recovery
pig pb delete -s pg-meta         # Delete all backups and archives after confirmation; destructive

The usual manual upgrade case is post-clone cleanup. After you restore another cluster’s backup into a new cluster, update the stanza identity before you write new backups.


Check and Control

pig pb check     # End-to-end check of configuration, archive push, and repository access
pig pb stop      # Stop new pgBackRest operations for a maintenance window
pig pb start     # Re-enable pgBackRest operations

check verifies that WAL can reach the repository through the archive path. It is not a purely local or read-only operation.


Logs

pig pb log              # Show the latest log snapshot
pig pb log list         # List log files
pig pb log tail         # Follow the latest log
ls /pg/log/pgbackrest/  # Directory containing backup, archive, and restore logs

For pgsql-pitr.yml, PostgreSQL writes recovery output to /pg/tmp/recovery.log.


Alternative Backup Tools

pg-basebackup

The legacy /pg/bin/pg-basebackup script creates a single-file physical backup with native pg_basebackup. It uses an lz4-compressed tar stream and writes to /pg/backup by default. Use it only for a simple local copy when a pgBackRest repository is unavailable:

pg-basebackup                                      # /pg/backup/backup_<tag>_<date>.tar.lz4
pg-basebackup --dst /tmp --file backup.tar.lz4     # Explicit destination and filename

mkdir -p /tmp/data
cat /pg/backup/backup_pg-meta_20250713.tar.lz4 | unlz4 -d -c | tar -xC /tmp/data
Legacy Encryption

pg-basebackup -e uses the obsolete OpenSSL RC4 cipher. You must not treat it as confidentiality protection. For encrypted backups, use a pgBackRest repository with AES-256 (cipher_type: aes-256-cbc).

Logical Backup

Logical backups made with pg_dump cannot provide PITR. They are suitable for cross-major-version migration, partial exports, and long-term logical snapshots. Production recovery plans commonly use logical and physical backups together. See the PostgreSQL documentation.

5.5 - Restore Operations

Perform PITR with pgsql-pitr.yml, pig pitr, or pig pb restore; select targets, run stages, and verify the result.

Pigsty provides three restore interfaces. They use the same parameter semantics for different scopes:

Entry pointUse caseWhat it controls
pgsql-pitr.ymlProduction cluster recoveryHA pause, multiple nodes, etcd cleanup, restore, and restart
pig pitrA local database nodeSingle-instance orchestration without the admin node
pig pb restoreAn instance not managed by PatroniA direct pgBackRest restore wrapper

For a sandbox drill, see Manual Recovery. To recover into another cluster without changes to production, see Clone a PG Cluster.

PITR overwrites the target cluster

pgsql-pitr.yml pauses HA and stops Patroni/PostgreSQL. It overwrites the target data directory with pgbackrest --force restore. It then deletes the target cluster’s etcd prefix and rebuilds HA. The playbook prints a plan but does not wait for confirmation. Before a real restore, inspect the topology with pig pg list <target-cluster>. Verify a recent usable backup and recovery window with pig pb info. Have the operator state and confirm the exact target cluster and recovery point. For production recovery, schedule a maintenance window and retain an independently verified backup.


Quick Start

To roll pg-meta back to an earlier time, declare pg_pitr:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_pitr: { time: '2025-07-13 10:00:00+00', action: promote }

Before recovery, verify the same target with these checks, then run the playbook:

pig pg list pg-meta
pig pb info
./pgsql-pitr.yml -l pg-meta

You can pass the same object temporarily on the command line:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'
Use valid JSON for command-line variables

The -e value must be valid JSON: quote keys and string values, for example {"pg_pitr": {"time": "...", "archive": true}}. Booleans are not quoted. Invalid quoting can fail parsing or silently produce the wrong value.

The playbook pauses Patroni HA, stops the cluster, and performs a delta pgBackRest restore. It starts PostgreSQL and waits for a consistent recovery state. It then prints control data, removes old etcd metadata, and starts the cluster under Patroni again. The playbook first prints the source, target, and restore command, but does not ask for interactive approval. For a one-shot targeted recovery, explicitly set action: promote. To inspect data at the target, use step-by-step execution with action: pause.


Recovery Targets

pg_pitr supports six recovery target forms. The four target values are mutually exclusive.

default/latest
pg_pitr: { }  # Replay to the end of the WAL archive stream
time
pg_pitr: { time: "2025-07-13 10:00:00+00" }
lsn
pg_pitr: { lsn: "0/4001C80" }
xid
pg_pitr: { xid: "250000" }
name
pg_pitr: { name: "some_restore_point" }
immediate
pg_pitr: { type: "immediate" }

With no target, recovery replays all archived WAL to the latest available state. Pigsty uses the internal type default for this mode. immediate stops at the first consistent point. Use it to get a usable instance as quickly as possible or to test a backup.

Recover by Time

Use a valid PostgreSQL TIMESTAMP. We strongly recommend an explicit time zone:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'

Recover by Name

Create an unambiguous marker before a risky change with pg_create_restore_point:

SELECT pg_create_restore_point('before_migration');
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "name": "before_migration", "action": "promote" }}'

Recover by Transaction ID

If monitoring or the CSVLOG TXID field identifies the offending transaction, use exclusive to stop before that transaction:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "xid": "250000", "exclusive": true, "action": "promote" }}'

Recover by LSN

An LSN identifies a position in the WAL stream. Pigsty’s PG LSN dashboard panel also shows it. If the required position is on a particular timeline, set timeline. The default is latest.

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "lsn": "0/4001C80", "timeline": "1", "action": "promote" }}'
Inclusive and exclusive targets

Targets are inclusive by default, so recovery replays the target transaction. exclusive: true excludes the exact target. It applies only to time, xid, and lsn. It maps to PostgreSQL’s recovery_target_inclusive.


Recovery Source

By default, recovery uses the target cluster’s own backup. Use these three fields to select another source:

  • cluster: the source stanza, including another cluster in a shared repository
  • repo: a temporary repository definition in the same format as a pgbackrest_repo entry
  • set: a specific backup label; otherwise pgBackRest selects a suitable set

For example, recover pg-meta2 from pg-meta:

./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": { "cluster": "pg-meta", "archive": false, "action": "promote" }}'
./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2025-07-14 08:00:00+00", "archive": false, "action": "promote" }}'

Step-by-Step Execution

During an incident, use tags to require a human decision between stages. Confirm the backup, recovery point, and exact target. Then run the stages in order:

./pgsql-pitr.yml -l pg-meta -t down  # Pause HA; stop Patroni and PostgreSQL
./pgsql-pitr.yml -l pg-meta -t pitr  # Restore, replay WAL, and print control information
./pgsql-pitr.yml -l pg-meta -t up    # Remove etcd metadata; start the cluster and resume HA
# down                 : # stop HA and PostgreSQL
#   - pause            : # pause Patroni automatic failover
#   - stop             : # stop Patroni and PostgreSQL
# pitr                 : # perform PITR
#   - config           : # render pgBackRest config and restore script
#   - backup           : # optionally move PGDATA to /pg/data-backup
#   - restore          : # run pgBackRest restore
#   - recovery         : # start PostgreSQL and replay WAL
#   - verify           : # print recovered control data
# up                   : # rebuild HA
#   - etcd             : # remove old cluster metadata
#   - start            : # start Patroni/PostgreSQL
#   - resume           : # resume Patroni automatic failover

After down, confirm that the processes have stopped. After pitr, inspect /pg/tmp/recovery.log and query the recovery state. Then perform the narrowly authorized checks of business data. pg_controldata /pg/data reports checkpoint and timeline metadata. This information alone does not prove that recovery reached a time, XID, or LSN target.

SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(),
       pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();

With action: pause, validate the data before promotion, then run up. If the target is wrong, adjust pg_pitr and repeat pitr before up. pause or shutdown provides an effective human decision point only in this staged procedure. For a one-shot targeted recovery, explicitly set action: promote.

pg_ctl -D /pg/data promote            # only after validating an action: pause recovery
./pgsql-pitr.yml -l pg-meta -t up     # rebuild Patroni HA
Repeating the pitr stage

With backup: true, the playbook moves the current data directory to /pg/data-backup. It deletes any existing /pg/data-backup first. The staged procedure is supported. A restore with backup: true is not generally idempotent.


PITR Parameter Definition

Declare the target, action, and treatment of existing data explicitly:

pg_pitr:
  cluster: pg-meta                 # source cluster/stanza; defaults to pg_cluster
  type: default                    # default | time | xid | name | lsn | immediate
  time: "2025-07-13 10:00:00+00"  # mutually exclusive with xid, name, and lsn
  name: "some_restore_point"       # mutually exclusive with time, xid, and lsn
  xid: "250000"                    # mutually exclusive with time, name, and lsn
  lsn: "0/4001C80"                 # mutually exclusive with time, xid, and name
  exclusive: false                 # exclude the exact target; time/xid/lsn only
  timeline: latest                 # target timeline; integer or latest
  set: latest                      # backup label; auto-select by default
  action: pause                    # pause | promote | shutdown
                                   # a specified target defaults to pause
  archive: true                    # preserve archiving; false sets archive-mode=off
  backup: false                    # move old PGDATA to /pg/data-backup before restore
  db_exclude: []                   # databases to exclude
  db_include: []                   # databases to include
  link_map:                        # tablespace/WAL link remapping
    pg_wal: '/data/wal'
    pg_xact: '/data/pg_xact'
  process: 4                       # restore workers; defaults to node_cpu
  repo: {}                        # temporary source repository definition
  data: /pg/data                  # target data directory
  port: 5432                      # recovery instance port

See Parameter Mapping for the corresponding pgBackRest options.


Single Instance: pig pitr

pig pitr performs a local-node procedure without Ansible. It validates the target, stanza, and backup, then stops Patroni/PostgreSQL and restores the data. It can optionally start PostgreSQL. It then prints instructions for the next steps.

pig pitr -t "2025-07-13 10:00:00+00"    # Recover to a point in time
pig pitr --xid 250000 -X                # Recover to before transaction 250000
pig pitr --name before_migration        # Recover to a named restore point
pig pitr -d                             # Recover to the end of the WAL archive stream
pig pitr -I --no-restart                # Restore for immediate recovery and leave PostgreSQL stopped
pig pitr -t "..." --plan                # Print the execution plan without making changes

Use -b/--set for a backup set, -T/--target-timeline for a timeline, and --target-action for the action after recovery reaches the target. For a side restore, use -D/--data with --no-restart. The command normally attempts a fast stop and aborts if that fails. Only explicit --force-stop permits immediate shutdown and a kill fallback.

For managed PGDATA, the command leaves Patroni stopped. Validate the instance before pig pt start. The command does not remove etcd metadata, rebuild replicas, or rejoin the instance to an HA cluster.


Primitive: pig pb restore

Use pig pb restore for an instance that Patroni does not manage, or that you deliberately removed from management. The command directly wraps pgbackrest restore. It validates the environment and requires PostgreSQL to be stopped. It then shows the plan and asks for confirmation.

pig pb restore --time "2025-07-13 10:00"
pig pb restore --set 20250715-013657F
pig pb restore -d

It rejects a live Patroni-managed target because Patroni could restart a partially restored instance. It also rejects any running PostgreSQL target. Arguments after --, such as --tablespace-map or --link-all, pass through to pgBackRest. You cannot override wrapped options, such as target, stanza, and repository, after --.


Post-Recovery

After restore:

  1. Verify the recovery state and the smallest authorized set of application checks.

  2. After a cross-cluster clone, complete stanza cleanup. Create a full backup on the new timeline as soon as practical:

    pg-backup full
  3. If exploratory recovery used archive: false, restore archiving. The archive_mode setting requires a restart. First confirm the maintenance window, current primary, and replication state. Then obtain explicit approval for the restart:

    psql -c 'ALTER SYSTEM RESET archive_mode;'
    pg restart pg-meta
    pg-backup full

5.6 - Clone a PG Cluster

Restore one cluster’s historical state into another for data recovery, restore drills, and forensic inspection.

Cloning restores production history into another cluster and leaves the production cluster unchanged. This makes cloning a useful recovery method with less risk to production. You can recover accidentally deleted data from the clone, validate backups in a drill, or inspect a historical state. You can also reset a test environment to a production snapshot.

The target must have access to the source backup repository and use a compatible PostgreSQL major version. Recovery may overwrite the target. In a shared Silo/S3 repository, a stanza isolates each cluster’s backups. Targets with the required credentials can access these backups.

A clone overwrites the target cluster

Inspect the target topology with pig pg list <target-cluster>. Verify the source stanza’s recent backups and recovery window with pig pb info. Before the restore, have the operator confirm the exact source cluster, target cluster, and recovery point. Recovery overwrites existing data on the target. Production work requires a maintenance window and an independently verified backup.


Clone an Existing Cluster

Assume the four-node sandbox contains pg-meta and pg-test, sharing a Silo repository. To reset pg-test to the latest state of pg-meta, point pg_pitr to the pg-meta stanza:

pig pg list pg-test
pig pb info
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "archive": false, "action": "promote" }}'

Add a recovery target to clone any state inside the recovery window. For example, reset to 15:30 on December 26, 2025:

./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2025-12-26 15:30:00+08", "archive": false, "action": "promote" }}'

These cross-cluster examples set archive: false to keep the exploratory recovery from archiving under the target stanza. After Patroni takes control, complete the stanza and archive cleanup below.

The target may also be a newly initialized empty cluster, such as pg-meta2. Use the normal cluster creation procedure to create it. Then perform cross-cluster PITR.

pgBackRest restore uses delta mode and rewrites only files that differ from the backup. Repeated drills, or a target already synchronized through a standby cluster, can therefore restore much faster than a first full restore.

For accidental deletion, validate the clone and use pg_dump to export only the affected tables or database back into production. An in-place rollback of the entire production cluster should be the last resort, not the first response.


Post-Clone Cleanup

The clone contains the source cluster’s data, while the target stanza may still record the target’s old PostgreSQL system identifier. pgBackRest refuses a backup when the identifiers do not match, preventing the new cluster from contaminating the source history.

After you validate the clone, complete these steps. A cluster restart changes the running service. First inspect the primary and replication state, schedule the maintenance window, and obtain explicit approval.

pb stanza-upgrade                           # accept the new system-id; cross-cluster clone only
psql -c 'ALTER SYSTEM RESET archive_mode;' # undo archive: false
pg restart pg-test                         # archive_mode requires a restart
pg-backup full                             # establish a recovery point on the new timeline

Until you complete these steps, scheduled backups can fail the identity check. A clone restored with archive: false produces no new WAL archive:

postgres@pg-test-1:~$ pb backup
INFO: backup command begin 2.57.0: --annotation=pg_cluster=pg-test ... --stanza=pg-test --start-fast
ERROR: [051]: PostgreSQL version 18, system-id 7588470953413201282 do not match stanza version 18, system-id 7588470974940466058
       HINT: is this the correct stanza?
INFO: backup command end: aborted with exception [051]

Rebuild Backup Identity

stanza-upgrade lets the new cluster continue writing under its existing stanza. If the clone needs a completely independent backup history, rebuild the stanza instead.

Declarative workflow:

pig pb info -s pg-test
pig pb delete -s pg-test                 # type the exact stanza when prompted
./pgsql.yml -t pg_backup -l pg-test      # create a fresh stanza
pg-backup full

Equivalent low-level workflow:

pig pb stop
pig pb delete -s pg-test                 # confirm the exact stanza
pig pb start
pig pb create -s pg-test
pig pb backup full -s pg-test
Rebuilding permanently discards old recovery history

Before deletion, check recent backups and retain any required independent recovery copy. Have the operator confirm the exact pg-test stanza. Object-locked versions can remain and continue to consume storage. A successful deletion command does not prove that it physically erased every underlying version.


Online Copies: Standby Clusters

A PITR clone is a static snapshot. Use a streaming-replication standby cluster for an online copy that follows the source continuously. Use a delayed cluster for a fixed rollback window, such as one hour.

The three methods complement each other. Standby clusters provide a live copy. Delayed clusters preserve a fixed delay. PITR clones provide any historical state inside the recovery window and do not require an existing online replica.


Restore Drills

A clone provides an end-to-end restore drill without changes to production. It overwrites the designated drill target. Run one quarterly and after major backup changes:

  1. Select a point inside the production recovery window.
  2. Restore it into the drill cluster and record elapsed time as the measured PITR RTO.
  3. Validate integrity with authorized row-count checks, critical-table checks, and application connectivity.
  4. Complete post-clone cleanup and verify that the drill cluster can create a new backup.
  5. Record timing, failures, and any difference between the runbook and reality.

See Manual Recovery for a sandbox exercise using pgBackRest primitives, or Fork an Instance for an XFS snapshot-based local test copy.

6 - Data Migration

How to migrate an existing PostgreSQL cluster to a new Pigsty-managed PostgreSQL cluster with minimal downtime?

Pigsty includes the pgsql-migration.yml playbook for online database migration with logical replication.

The generated automation scripts can reduce application downtime to a few seconds. Logical replication requires PostgreSQL 10 or later.

If the permitted downtime is sufficient, you can use pg_dump | psql for offline migration.


Defining Migration Tasks

To use the online migration playbook, create a definition file that describes the migration task.

Refer to the task definition file example: files/migration/pg-meta.yml.

This task migrates pg-meta.meta to pg-test.test online. The former is the Source Cluster (SRC). The latter is the Destination Cluster (DST).

pg-meta-1	10.10.10.10  --> pg-test-1	10.10.10.11 (10.10.10.12,10.10.10.13)

Logical replication migrates one database at a time. Specify the database name, the source and destination primary IP addresses, and the superuser connection information.

---
#-----------------------------------------------------------------
# PG_MIGRATION
#-----------------------------------------------------------------
context_dir: ~/migration  # Directory for migration manual & scripts
#-----------------------------------------------------------------
# SRC Cluster (Old Cluster)
#-----------------------------------------------------------------
src_cls: pg-meta      # Source cluster name                  <Required>
src_db: meta          # Source database name                 <Required>
src_ip: 10.10.10.10   # Source cluster primary IP            <Required>
#src_pg: ''            # If defined, use this as source dbsu pgurl instead of:
#                      # postgres://{{ pg_admin_username }}@{{ src_ip }}/{{ src_db }}
#                      # e.g.: 'postgres://dbuser_dba:[email protected]:5432/meta'
#sub_conn: ''          # If defined, use this as subscription connection string instead of:
#                      # host={{ src_ip }} dbname={{ src_db }} user={{ pg_replication_username }}'
#                      # e.g.: 'host=10.10.10.10 dbname=meta user=replicator password=DBUser.Replicator'
#-----------------------------------------------------------------
# DST Cluster (New Cluster)
#-----------------------------------------------------------------
dst_cls: pg-test      # Destination cluster name             <Required>
dst_db: test          # Destination database name            <Required>
dst_ip: 10.10.10.11   # Destination cluster primary IP       <Required>
#dst_pg: ''            # If defined, use this as destination dbsu pgurl instead of:
#                      # postgres://{{ pg_admin_username }}@{{ dst_ip }}/{{ dst_db }}
#                      # e.g.: 'postgres://dbuser_dba:[email protected]:5432/test'
#-----------------------------------------------------------------
# PGSQL
#-----------------------------------------------------------------
pg_dbsu: postgres
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor
#-----------------------------------------------------------------
...

By default, Pigsty builds the source and destination superuser connection strings from the global administration user and each primary IP address. Use src_pg and dst_pg to override these defaults. Use sub_conn to override the subscription connection string.


Generating Migration Plan

The playbook generates the operating manual and automation scripts for migration. It does not perform the migration itself.

By default, the migration context directory is ~/migration/pg-meta.meta. Follow README.md and run the scripts in sequence to complete the migration.

# Activate migration context: enable related environment variables
. ~/migration/pg-meta.meta/activate

# These scripts check src cluster status and help generate new cluster definitions in pigsty
./check-user     # Check src users
./check-db       # Check src databases
./check-hba      # Check src hba rules
./check-repl     # Check src replication identity
./check-misc     # Check src special objects

# These scripts establish logical replication between existing src cluster and pigsty-managed dst cluster, data except sequences will sync in real-time
./copy-schema    # Copy schema to destination
./create-pub     # Create publication on src
./create-sub     # Create subscription on dst
./copy-progress  # Print logical replication progress
./copy-diff      # Quick compare src and dst differences by counting tables

# These scripts run during online migration, which stops src cluster and copies sequence numbers (logical replication doesn't replicate sequences!)
./copy-seq [n]   # Sync sequence numbers, if n is given, apply additional offset

# You must switch application traffic to the new cluster based on your access method (dns,vip,haproxy,pgbouncer,etc.)!
#./disable-src   # Restrict src cluster access to admin nodes and new cluster (your implementation)
#./re-routing    # Re-route application traffic from SRC to DST! (your implementation)

# Then cleanup to remove subscription and publication
./drop-sub       # Drop subscription on dst after migration
./drop-pub       # Drop publication on src after migration

Notes

To avoid primary key conflicts when you copy sequence values, you can advance all sequences by an offset, such as +1000. For this example, run ./copy-seq with the argument 1000.

Implement ./re-routing to route application traffic from src to dst. The implementation depends on your access method, such as DNS, VIP, HAProxy, or PgBouncer. You can also change the routing manually.

You can optionally implement ./disable-src to restrict application access to the src cluster. If ./re-routing switches all application traffic completely, you do not need this step.

If traffic from unknown sources prevents a complete switch, use a more complete access restriction. We recommend that you change and reload the HBA rules. You can also stop the postgres, pgbouncer, or haproxy processes on the source primary.

7 - Tutorials

Step-by-step guides for common PostgreSQL tasks and scenarios.

This section provides step-by-step tutorials for common PostgreSQL tasks and scenarios.

7.1 - Troubleshooting

Common failures and analysis troubleshooting approaches

This document lists possible PostgreSQL and Pigsty failures. It gives standard procedures to locate, resolve, and analyze them.


Disk Space Exhausted

Disk space exhaustion is the most common type of failure.

Symptoms

If the database disk is full, PostgreSQL cannot operate normally. Logs repeatedly report “no space left on device”, and new writes fail. PostgreSQL may also report a PANIC and shut down.

The Pigsty NodeFsSpaceFull alert triggers when filesystem free space is less than 10%. To locate the problem, open the NODE Instance dashboard and inspect the FS metrics panel.

Diagnosis

You can also log in to the database node and run df -h. Identify the full partition. Then check these directories and their sizes to identify which files consume the space:

  • Data directory (/pg/data/base): Stores data files for tables and indexes; pay attention to heavy writes and temporary files
  • WAL directory (e.g., pg/data/pg_wal): Stores PG WAL; WAL accumulation/replication slot retention is a common cause of disk exhaustion.
  • Database log directory (e.g., pg/log): If PG logs are not rotated in time and large amounts of errors are written, they may also consume significant space.
  • Local backup directory (e.g., data/backups): When using pgBackRest or similar tools to save backups locally, this may also fill up the disk.

If the issue occurs on the Pigsty admin node or monitoring node, also consider:

  • Monitoring data: VictoriaMetrics time-series metrics and VictoriaLogs log storage both consume disk space; check retention policies.
  • Object storage data: Pigsty’s integrated Silo object storage may be used for PG backup storage.

After you identify the largest directory, use du -sh <directory> to locate large files or subdirectories.

Resolution

Disk exhaustion requires immediate action to free space and keep the database operating. If the data disk and system disk are the same, a full disk may prevent shell commands from running. In this case, delete the /pg/dummy placeholder file to free emergency space for shell commands. If accumulated pg_wal caused a database crash, free space and restart the database service. Then check data integrity carefully.


Transaction ID Wraparound

PostgreSQL reuses 32-bit transaction IDs (XIDs) in a cycle. Exhaustion causes a transaction ID wraparound failure (XID Wraparound).

Symptoms

The first typical sign is age saturation entering the warning zone in PGSQL Persist - Age Usage. Database logs begin to show messages such as WARNING: database "postgres" must be vacuumed within xxxxxxxx transactions.

If the condition gets worse, PostgreSQL enters protection mode. With about 1 million transaction IDs remaining, the database becomes read-only. At the limit of about 2.1 billion (2^31), it rejects new transactions and shuts down to prevent data corruption.

Diagnosis

PostgreSQL and Pigsty enable automatic garbage collection (AutoVacuum) by default. Thus, this failure usually indicates another underlying problem. Common causes include very long transactions (SAGE), incorrect Autovacuum settings, blocked replication slots, and insufficient resources. Storage engine or extension bugs and bad disk blocks can also cause it.

First, identify the database with the highest age. Then inspect table ages in the Pigsty PGCAT Database - Tables panel. Also inspect database error logs for evidence of the cause.

Resolution

  1. Immediately freeze old transactions: If the database is not yet in read-only protection mode, immediately run a manual VACUUM FREEZE. For faster results, first freeze the oldest tables individually. Connect as a superuser and run VACUUM FREEZE table_name; on tables identified with the largest relfrozenxid. Prioritize tables with the highest XID age. This can quickly reclaim transaction ID space.
  2. Single-user mode rescue: If the database rejects writes or has shut down for protection, start it in single-user mode. Run VACUUM FREEZE database_name; to freeze and clean the entire database. Then restart in multi-user mode. This can remove wraparound protection and permit writes again. Operate carefully in single-user mode. Make sure that enough transaction IDs remain to complete the freeze.
  3. Standby node takeover: If problems such as hardware failure prevent vacuum, consider promoting a read-only standby to primary. This can provide a cleaner environment for the freeze. For example, if bad blocks prevent vacuum on the primary, manually fail over to the standby and run emergency vacuum freeze. Verify that the new primary has frozen old transactions before you switch the load back.

Connection Exhaustion

max_connections limits PostgreSQL connections. When clients exceed this limit, PostgreSQL rejects new connections. Applications cannot connect and report FATAL: remaining connection slots are reserved for non-replication superuser connections or too many clients already. Regular connection slots are exhausted. Only slots reserved for superusers or replication remain.

Diagnosis

Connection exhaustion usually results from many concurrent client requests. Inspect active sessions in PGCAT Instance, PGCAT Database, or PGCAT Locks. Identify the query types that occupy the connections. Look for many “Idle in Transaction” connections, long transactions, and slow queries.

Resolution

Kill queries: If connection exhaustion blocks application operations, usually use pg_terminate_backend(pid) for immediate relief. If connections use a pool, adjust its size settings and reload it to reduce database connections.

You can also increase max_connections. Restart the database to apply the change.


etcd Quota Exhausted

An exhausted etcd quota will cause the PG high availability control plane to fail and prevent configuration changes.

Diagnosis

Pigsty uses etcd as the distributed configuration store (DCS) for high availability. etcd has a storage quota of about 2GB by default. At the limit, etcd rejects writes and reports “etcdserver: mvcc: database space exceeded”. Patroni then cannot write heartbeats or configuration changes to etcd, so cluster management fails.

Resolution

Pigsty v2.0.0 through v2.5.1 are affected by default. Pigsty v2.6.0 added automatic compaction for deployed etcd. If etcd stores only PostgreSQL high-availability leases, the problem no longer occurs in regular use.


Defective Storage Engine

TimescaleDB’s experimental Hypercore storage engine has confirmed defects. Reported cases include VACUUM failing to reclaim space and causing XID wraparound. If you use Hypercore, promptly migrate to native PostgreSQL tables or the default TimescaleDB engine.

Detailed introduction: PG New Storage Engine Failure Case (Chinese)

7.2 - Manual PITR Drill

Run, validate, and complete PostgreSQL point-in-time recovery in explicit stages inside an isolated sandbox.

This tutorial performs a PostgreSQL PITR drill in Pigsty v4.5.0’s four-node sandbox. It runs pgsql-pitr.yml in the sequence down → pitr → up. Separate validation steps precede data overwrite, timeline promotion, and HA reconstruction.

For one current node, use pig pitr. For direct pgBackRest control, see the low-level pg-pitr utility.

Follow this literally only in a disposable sandbox

Recovery stops Patroni/PostgreSQL and overwrites the target PGDATA with pgbackrest --force restore. The up stage deletes the target cluster’s etcd prefix and rebuilds Patroni state. The playbook prints a plan but has no interactive confirmation. Before production use, the operator must state and confirm the exact cluster and recovery point. Verify a recent usable backup that has been independently tested. Run --check with exactly the same -l, variables, and tags, and schedule a maintenance window. This tutorial does not authorize these commands in a production environment.


Prepare an Isolated Sandbox

Use Vagrant or another disposable four-node lab and select the ha/full template, which includes a Silo backup repository:

curl -fsSL https://repo.pigsty.io/get | bash
cd ~/pigsty
./configure -c ha/full
./deploy.yml

ha/full defines a single-node pg-meta, a three-node pg-test, and a Silo/pgBackRest repository. This tutorial uses the exact target pg-meta. Before you copy that selector to another environment, resolve its inventory.

Initial deployment and backup both change sandbox state. Production environments require their own deployment and backup approval process.


Establish Recovery Evidence

Start with read-only topology, backup-chain, and WAL-range checks:

pig pg list pg-meta
sudo -iu postgres pig pb info
sudo -iu postgres pig pb check

info must show at least one usable backup with status: ok. Archived WAL must cover the intended target. check validates the current stanza and archive path. It does not replace a real restore drill or validation of an independent copy.

In the sandbox, run Pigsty’s heartbeat helper to create an easy-to-verify time series:

sudo -iu postgres /pg/bin/pg-heartbeat

Record, then stop the workload:

  • the timezone-qualified timestamp you intend to recover to;
  • heartbeat, LSN, and transaction boundaries around that point;
  • current primary, timeline, and backup label;
  • target cluster pg-meta and target node.

Inspecting real application tables requires separate authorization. This tutorial uses only sandbox heartbeat data.


Declare the Recovery Task

Declare the target under pg-meta.vars in the sandbox inventory:

pg_pitr:
  cluster: pg-meta
  time: "2026-08-13 10:00:00+08"
  action: pause
  archive: true
  backup: false
  • cluster is the source backup stanza and defaults to the target pg_cluster.
  • action: pause pauses PostgreSQL at the target for a human validation gate.
  • archive: true preserves archive settings.
  • backup: true is not a safe-backup substitute: it deletes an existing <pg_data>-backup before moving current PGDATA, so this drill keeps it false.

You can also pass the object temporarily with -e. Preflight and all three stages must use exactly the same valid JSON to prevent variable differences.


Full Preflight

Before any stop or write action, check the complete workflow against the same target:

./pgsql-pitr.yml -l pg-meta --check

Confirm that Ansible resolves exactly pg-meta, then review the output for:

  • source stanza, recovery type, time, timeline, and action;
  • destination pg_data, port, and repository;
  • tablespace/link mappings;
  • archive and backup behavior.

--check validates inventory, variables, and task selection. It cannot prove that a pgBackRest backup is restorable. If the target, backup, or variables change, repeat preflight.


Stage One: Stop

Only after the operator reconfirms exact target pg-meta, recovery point, and maintenance window, run:

./pgsql-pitr.yml -l pg-meta -t down

down attempts to pause Patroni automatic failover and stops Patroni on every target member. If PostgreSQL remains running, it uses immediate shutdown. Then verify every target node independently of the playbook result:

sudo systemctl is-active patroni
sudo -iu postgres pg_ctl -D /pg/data status

The expected results are inactive and “server is not running.” If any member remains active, stop and diagnose the problem. Do not start the restore.


Stage Two: Restore and Validate

Recheck pg_pitr and the target nodes before running the destructive stage:

./pgsql-pitr.yml -l pg-meta -t pitr

This stage:

  1. renders /pg/conf/pitr.conf and /pg/bin/pg-restore;
  2. optionally moves old PGDATA according to backup;
  3. creates the destination and runs pgBackRest restore with --force and delta=y;
  4. starts PostgreSQL directly and waits for a consistent recovery state in the log;
  5. prints a pg_controldata summary.

Control data proves only that the directory contains readable control state. It does not verify the time, XID, or application boundary. With action: pause, confirm that WAL replay reached and paused near the target:

sudo -iu postgres psql -p 5432 -Atqc \
  'SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp()'

Then inspect only the smallest authorized data scope. In the sandbox, check heartbeat rows. If the target is wrong:

  1. keep every Patroni member stopped;
  2. stop the manually started PostgreSQL;
  3. adjust the target and rerun the complete --check;
  4. rerun the pitr stage.

Do not run up or allow replicas from the old timeline to rejoin.


Promote and Stage Three: Rebuild HA

Promote only after the operator accepts the recovered result and the creation of a new timeline:

sudo -iu postgres pg_ctl -D /pg/data promote
sudo -iu postgres psql -p 5432 -Atqc 'SELECT pg_is_in_recovery()'

The expected result is f. Promotion changes state, and you cannot undo it without loss.

With every Patroni member still stopped and exact target still pg-meta, run:

./pgsql-pitr.yml -l pg-meta -t up

For the primary, up deletes the /pg/pg-meta/ prefix from etcd. The effective prefix also depends on pg_namespace and Citus settings. It stops the manually started PostgreSQL and starts Patroni on the primary. It then starts replicas one at a time and resumes HA. The etcd deletion task tolerates errors. Thus, a successful playbook result does not prove that it removed stale DCS state correctly.


Post-Recovery Acceptance

Verify each layer. Service startup alone does not complete recovery:

pig pg list pg-meta
sudo -iu postgres psql -Atqc \
  "SELECT pg_is_in_recovery(), pg_current_wal_lsn(), current_setting('archive_mode')"
sudo -iu postgres pig pb check

Also confirm:

  • exactly the intended member is primary and replicas stream from the new timeline;
  • HAProxy/VIP/DNS and application traffic point only to accepted instances;
  • data and event boundaries around the recovery target are correct;
  • archive_mode, archive_command, and new-WAL archiving work;
  • monitoring, alerts, and the backup repository contain no stale cluster state.

After the new timeline is stable, create and verify a new full backup under the applicable approval process:

sudo -iu postgres pg-backup full
sudo -iu postgres pig pb info

If recovery used archive: false, it wrote archive-mode=off. Validate recovery and confirm a maintenance window before you reset that override and perform a controlled restart. The default archive: true does not require this step.


Multi-Node and Cross-Cluster Recovery

  • Old-timeline replicas must not rejoin without validation. up starts replicas one at a time and waits for clone/recovery; monitor them to completion.
  • In cross-stanza recovery, pg_pitr.cluster is the source while -l is the destination being overwritten. Record and state both separately.
  • Cross-cluster recovery should normally use archive: false so a test destination cannot write WAL into the source stanza. Enable its own archiving only after acceptance and post-clone stanza cleanup.
  • link_map, data, port, and a temporary repo change the actual storage and data targets; include all of them in both --check and human review.

7.3 - Enabling HugePage for PostgreSQL

Enabling HugePage for PostgreSQL to reduce memory fragmentation and improve performance.

Use node_hugepage_count and node_hugepage_ratio or /pg/bin/pg-tune-hugepage

To enable HugePages, consider node_hugepage_count and node_hugepage_ratio. Apply these settings with ./node.yml -t node_tune.

HugePages have advantages and disadvantages for databases. Exclusive memory management prevents reallocation and reduces database OOM risk. However, HugePages can reduce performance in some conditions.

Before PostgreSQL starts, allocate enough huge pages. The pg-tune-hugepage script can reclaim the unused portion. This script is available only for PostgreSQL 15+.

If PostgreSQL is already running, use this method to enable huge pages. It requires PG15+:

sync; echo 3 > /proc/sys/vm/drop_caches   # Flush disk, release system cache (be prepared for database perf impact)
sudo /pg/bin/pg-tune-hugepage             # Write nr_hugepages to /etc/sysctl.d/hugepage.conf
pg restart <cls>                          # Restart postgres to use hugepage

7.4 - Clone and Side-Restore a PostgreSQL Instance

Create a local physical copy with pg-fork and run low-level pgBackRest recovery against a stopped data directory with pg-pitr.

Pigsty v4.5.0 provides two local shell utilities:

  • pg-fork copies a PostgreSQL data directory and gives the copy a separate port.
  • pg-pitr invokes pgBackRest to restore a stopped data directory to a selected target.

Use these utilities for sandbox drills, side-channel investigation, and temporary testing. They do not orchestrate complete Patroni-cluster recovery. Prefer pig pitr for a managed instance and staged pgsql-pitr.yml for a multi-node cluster.

Verify paths, backups, and stopped state first

pg-fork recursively removes an existing destination directory. pg-pitr overwrites the destination with backup data. Both can run without a prompt in a non-interactive environment. Before a real run, verify absolute source and destination paths, ports, tablespaces, and exact cluster/instance identity. Also verify an independent recent backup that has been tested. A new CoW clone is not an independent backup.


pg-fork

pg-fork copies a PostgreSQL data directory on the current node. Run it as the database OS user, normally postgres, or at least as a member of the postgres group:

pg-fork 1                         # /pg/data -> /pg/data1, destination port 15432
pg-fork 2 -d /pg/data1            # /pg/data1 -> /pg/data2, destination port 25432
pg-fork 3 -D /srv/pg-clone -P 55432

Parameters

pg-fork <FORK_ID> [options]
ParameterMeaningDefault
<FORK_ID>One digit from 1 to 9, used to derive the default directory and portRequired
-d, --data <path>Source data directory$PG_DATA or /pg/data
-D, --dst <path>Destination data directory/pg/data<FORK_ID>
-p, --port <port>Source instance port$PG_PORT or 5432
-P, --dst-port <port>Destination instance port<FORK_ID>5432
-s, --skipSkip the online-backup API and force cold-copy modeNo
-y, --yesSkip interactive confirmationNo

The script rejects identical normalized source and destination paths. It cannot determine whether a custom destination contains other important data. If the destination directory exists, the script removes it recursively before the copy.

Hot Backup and Cold Copy

By default, the script connects to the source port and uses one psql session to run:

  1. CHECKPOINT;
  2. pg_backup_start();
  3. rm -rf <destination> followed by cp -a --reflink=auto;
  4. pg_backup_stop(wait_for_archive => false).

If the source is unreachable on the selected port, the script automatically falls back to a cold copy. It does not abort. -s also forces a cold copy. A cold copy is safe only after you independently confirm that the source instance has fully stopped. A postmaster.pid warning alone does not prove the process state.

On the same filesystem, the script recognizes XFS with reflink enabled, Btrfs, Bcachefs, and OCFS2 as fast CoW modes. Other filesystems or cross-filesystem destinations still use cp --reflink=auto, but may require a full copy. The help text describes ZFS support more broadly than the detector. The v4.5.0 implementation does not classify ZFS as a confirmed fast-CoW mode.

Clone Configuration

After a successful copy, pg-fork:

  • removes postmaster.pid, postmaster.opts, and standby.signal from the destination;
  • clears physical replication-slot files in the destination;
  • writes a separate port, archive_mode=off, and a local log_directory to the destination’s postgresql.auto.conf;
  • removes primary_conninfo, primary_slot_name, and old recovery_target* overrides.

The script does not check destination port availability or adjust memory settings. Before you start the copy, inspect at least these settings:

postgres -D /pg/data1 -C port
postgres -D /pg/data1 -C archive_mode
postgres -D /pg/data1 -C shared_buffers
pg_ctl -D /pg/data1 status
External tablespaces are not isolated

cp -a preserves symlinks under pg_tblspc. pg-fork does not copy or remap tablespaces outside PGDATA. Starting such a clone can access or modify the source tablespaces. If external tablespaces exist, copy and remap each one independently. Otherwise, do not use this script to create a writable clone.

Interaction Boundary

The script asks Proceed with fork? [y/N] only if standard input is a terminal and you did not use -y. Pipes, CI, cron, and other non-interactive calls do not get this prompt. Automation must enforce a strict absolute-path allowlist and check whether the destination exists before it calls the script. Do not add -y by default only for convenience.


pg-pitr

pg-pitr is a low-level pgBackRest restore wrapper. It does not pause or start Patroni, stop or start PostgreSQL, clear DCS state, or rebuild replicas.

Recovery Targets

Understand and select a recovery target before execution. Without arguments, the command only shows help:

ParameterpgBackRest semantics
-d, --defaultSet no stop target and replay to the available end of WAL
-i, --immediateStop when the selected backup becomes consistent
-t, --time <timestamp>Recover to a timestamp
-n, --name <restore-point>Recover to a named restore point
-l, --lsn <lsn>Recover to an LSN
-x, --xid <xid>Recover to a transaction ID

-S/--set, with compatibility alias -b/--backup, selects which backup set recovery starts from. It does not select a stop target. For example, -S 20251225-120000F -d still replays to the end of WAL. Use -S ... -i to stop when that backup first becomes consistent.

For time, name, lsn, xid, and immediate, pgBackRest pauses at the target by default. -P/--promote changes this action to automatic promotion. Use -X/--exclusive only with a precise target such as time, lsn, or xid.

Other Options

ParameterMeaning
-D, --data <path>Absolute destination data directory; default /pg/data
-s, --stanza <name>pgBackRest stanza; defaults to the first non-global stanza in the config
-T, --timeline <value>latest, current, or a positive numeric timeline
-P, --promoteAutomatically promote recovery methods that have a stop target
-v, --verboseEnable pgBackRest info-level console logging
-c, --check, --dry-runPrint the command without executing it
-y, --yesSkip the five-second countdown
-- <args>Pass additional arguments directly to pgBackRest

-c checks the generated command. It does not prove that the backup/WAL is usable or that PostgreSQL and Patroni have stopped. The wrapper does not filter conflicting native arguments. If you pass repository, tablespace, or link-mapping options, carefully review the final command.

Safe Execution Sequence

This example shows the low-level procedure for one isolated destination. For production cluster recovery, use the complete runbook:

# 1. Read-only verification of backup sets and the recovery window
pig pb info

# 2. Confirm the exact target is stopped; stop Patroni first for a managed instance
pg_ctl -D /pg/data1 status

# 3. Render and review the exact command without writing data
pg-pitr -D /pg/data1 -t "2026-08-13 10:00:00+08" -c

# 4. Remove -c only after the operator reconfirms destination, backup, and stopped state
pg-pitr -D /pg/data1 -t "2026-08-13 10:00:00+08"

Real execution rejects root and aborts if postmaster.pid exists in the destination. Before removing even a stale PID file, the operator must confirm that PostgreSQL has stopped. There is no y/N question. An interactive terminal gets only an interruptible five-second countdown. A non-interactive call skips the countdown and starts the restore immediately.

After restore, the operator starts and validates the instance:

pg_ctl -D /pg/data1 start
psql -p 15432 -Atqc \
  'SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(), pg_last_xact_replay_timestamp()'

Promote only after you verify the recovery target, authorized application data, timeline, and archive settings. Promotion creates a new timeline and is not a reversible inspection operation. pg-pitr does not disable archiving itself. Before you follow its generic final “enable archive_mode” hint, inspect the effective setting. Correct only an override introduced by this recovery.

Additional Side-Restore Risks

For a custom destination such as /pg/data1, pgBackRest can restore postgresql.auto.conf and overwrite the separate port set by pg-fork. Before startup, recheck port, archive_mode, sockets, logging, and memory settings.

If the backup contains external tablespaces or links, a side restore can use the original paths. For isolation, pass reviewed pgBackRest --tablespace-map, --link-map, or related arguments after --. Inspect the generated command. Otherwise, do not start the restored copy on the production host.


  1. Verify the source instance, destination absolute path, destination port, tablespaces, and independent backup.
  2. Run pg-fork <id> in an interactive terminal and confirm that the plan shows hot backup rather than an unintended cold-copy fallback.
  3. Before starting the clone, run pg-pitr -D <clone> ... -c and inspect the recovery command.
  4. Execute restore only after explicit destination confirmation, then recheck the clone’s port and every external path.
  5. Start the clone on the isolated port and verify recovery state and only the application data you are authorized to inspect.
  6. Promote only if the clone is intentionally becoming a new primary; otherwise stop it and clean up only the exact, verified path.

Side validation reduces direct writes to the current PGDATA. It still uses the same backup repository, consumes host resources, and may access external tablespaces. The sandbox therefore still has risks.


7.5 - Accidental Deletion

Handling accidental data deletion, table deletion, and database deletion

Accidental Data Deletion

For a small accidental DELETE, consider pg_surgery or pg_dirtyread for selective recovery in place.

-- Immediately disable Auto Vacuum on this table and abort Auto Vacuum worker processes for this table
ALTER TABLE public.some_table SET (autovacuum_enabled = off, toast.autovacuum_enabled = off);

CREATE EXTENSION pg_dirtyread;
SELECT * FROM pg_dirtyread('tablename') AS t(col1 type1, col2 type2, ...);

If the deleted data has already been reclaimed by VACUUM, then use the general accidental deletion recovery process.

Accidental Object Deletion

For accidental DROP/DELETE operations, usually select a recovery method in this order:

  1. Check whether the application or another data system can supply the data. If so, recover it from that system.
  2. Check whether a delayed replica exists. If so, advance it to the time before deletion and query the data for recovery.
  3. If deletion is confirmed, check the backups and their recovery window. If the window covers the deletion time, start PITR.
  4. Select where to replay: an in-place cluster PITR rollback, a new server, or a replica. Then perform recovery.

Accidental Cluster Deletion

Pigsty management commands can delete an entire cluster, for example pgsql-rm.yml or bin/pgsql-rm. Unless pg_rm_backup is false, these commands also delete the cluster backups.

Warning: In this situation, you cannot recover the data. Review the target and the deletion consequences carefully before you proceed.

For production, we recommend that you set this parameter to false globally in the inventory. This preserves backups when you remove clusters.

7.6 - HA Drill: Handling 2-of-3 Node Failure

HA scenario response plan: When two of three nodes fail and auto-failover doesn’t work, how to recover from the emergency state?

In a typical 3-node HA deployment, simultaneous failure of two nodes removes the majority. The system usually cannot complete automatic failover and needs manual intervention.

First, assess the two failed servers. If they can restart quickly, prioritize their recovery. Otherwise, use the Emergency Recovery Procedure.

The Emergency Recovery Procedure assumes that the administration node has failed and only one regular database node survives. In this case, the fastest recovery procedure is:

  • Adjust HAProxy configuration to direct traffic to the primary.
  • Stop Patroni and manually promote the PostgreSQL replica to primary.

Adjust HAProxy Configuration

If connections bypass HAProxy, skip this step. If connections use HAProxy, change its configuration to direct read/write traffic to the primary manually.

  • Edit /etc/haproxy/conf.d/<pg_cluster>-primary.cfg, where <pg_cluster> is your PostgreSQL cluster name, e.g., pg-meta.
  • Comment out the health check configuration options to stop health checks.
  • Comment out the other two failed machines in the server list, keeping only the current primary server.
listen pg-meta-primary
    bind *:5433
    mode tcp
    maxconn 5000
    balance roundrobin

    # Comment out the following four health check lines
    #option httpchk                               # <---- remove this
    #option http-keep-alive                       # <---- remove this
    #http-check send meth OPTIONS uri /primary    # <---- remove this
    #http-check expect status 200                 # <---- remove this

    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    server pg-meta-1 10.10.10.10:6432 check port 8008 weight 100

    # Comment out the other two failed machines
    #server pg-meta-2 10.10.10.11:6432 check port 8008 weight 100 <---- comment this
    #server pg-meta-3 10.10.10.12:6432 check port 8008 weight 100 <---- comment this

After you change the configuration, wait until primary promotion before you run systemctl reload haproxy. The new configuration disables HAProxy primary health checks, which use Patroni by default. HAProxy then directs write traffic to the current primary.


Manually Promote Replica

Log in to the target server and switch to the dbsu user. Run CHECKPOINT to flush data to disk. Stop Patroni, restart PostgreSQL, then promote the replica:

sudo su - postgres                     # Switch to database dbsu user
psql -c 'checkpoint; checkpoint;'      # Two Checkpoints to flush dirty pages, avoid long PG restart
sudo systemctl stop patroni            # Stop Patroni
pg-restart                             # Restart PostgreSQL
pg-promote                             # Promote PostgreSQL replica to primary
psql -c 'SELECT pg_is_in_recovery();'  # If result is f, it has been promoted to primary

If you changed the HAProxy configuration, now run systemctl reload haproxy. This applies the configuration and directs traffic to the new primary.

systemctl reload haproxy                # Reload HAProxy configuration to direct write traffic to current instance

Avoid Split Brain

After emergency recovery, the next priority is to Avoid Split Brain. Prevent the other two servers from returning online and becoming competing primaries. Such a split brain can cause data inconsistency.

Simple approaches:

  • Power off/disconnect network the other two servers to ensure they don’t come online uncontrollably.
  • Adjust the database connection string used by applications to point directly to the surviving server’s primary.

Then decide the next steps based on the specific situation:

  • A: The two servers have temporary failures (e.g., network/power outage) and can be repaired in place to continue service.
  • B: The two failed servers have permanent failures (e.g., hardware damage) and will be removed and decommissioned.

Recovery After Temporary Failure

If the other two servers have temporary failures and can return to service, use these repair and rebuild steps:

  • Handle one failed server at a time, prioritize the admin node / INFRA node.
  • Start the failed server and stop Patroni after startup.

When ETCD regains quorum, it resumes operation. Start Patroni on the surviving server, which is the current primary. Patroni takes control of the existing PostgreSQL instance and regains cluster leadership. Then enter maintenance mode:

sudo systemctl restart patroni
pg pause <pg_cluster>

On each of the other two instances, use the postgres account to run touch /pg/data/standby.signal. The marker identifies the instance as a replica. Then start Patroni:

sudo -iu postgres touch /pg/data/standby.signal
sudo systemctl restart patroni

After confirming Patroni cluster identity/roles are correct, exit maintenance mode:

pg resume <pg_cluster>

Recovery After Permanent Failure

After permanent failure, first recover ~/pigsty on the administration node. The essential files are pigsty.yml and files/pki/ca/ca.key.

If you cannot retrieve these two files or their backups, deploy a new Pigsty installation. Migrate the existing cluster through Backup Cluster.

Back up the pigsty directory regularly, for example with Git version control.

Configuration Repair

You can use the surviving node as the new administration node. Copy ~/pigsty to it, then adjust the configuration. For example, replace the original administration node 10.10.10.10 with the surviving node 10.10.10.12:

all:
  vars:
    admin_ip: 10.10.10.12               # Use new admin node address
    node_etc_hosts: [10.10.10.12 h.pigsty a.pigsty p.pigsty g.pigsty sss.pigsty]
    infra_portal: {}                    # Also modify other configs referencing old admin IP (10.10.10.10)

  children:

    infra:                              # Adjust Infra cluster
      hosts:
        # 10.10.10.10: { infra_seq: 1 } # Old Infra node
        10.10.10.12: { infra_seq: 3 }   # New Infra node

    etcd:                               # Adjust ETCD cluster
      hosts:
        #10.10.10.10: { etcd_seq: 1 }   # Comment out this failed node
        #10.10.10.11: { etcd_seq: 2 }   # Comment out this failed node
        10.10.10.12: { etcd_seq: 3 }    # Keep surviving node
      vars:
        etcd_cluster: etcd

    pg-meta:                            # Adjust PGSQL cluster configuration
      hosts:
        #10.10.10.10: { pg_seq: 1, pg_role: primary }
        #10.10.10.11: { pg_seq: 2, pg_role: replica }
        #10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
        10.10.10.12: { pg_seq: 3, pg_role: primary , pg_offline_query: true }
      vars:
        pg_cluster: pg-meta

ETCD Repair

Then execute the following command to reset ETCD to a single-node cluster:

./etcd.yml -e etcd_safeguard=false -e etcd_clean=true

Follow the instructions in ETCD Reload Configuration to adjust ETCD Endpoint references.

INFRA Repair

If the surviving node does not have the INFRA module, configure and install it on that node. Run this command:

./infra.yml -l 10.10.10.12

Repair monitoring on the current node:

./node.yml -t monitor

PGSQL Repair

./pgsql.yml -t pg_conf                            # Regenerate PG configuration files
systemctl reload patroni                          # Reload Patroni configuration on surviving node

After you repair each module, use the standard expansion procedure to add nodes and restore cluster high availability.

7.7 - Bind a L2 VIP to PostgreSQL Primary with VIP-Manager

If all cluster nodes share one L2 network, you can optionally define an L2 VIP for the PostgreSQL cluster.

The VIP uses Master-Backup mode. It always points to the node that runs the cluster primary instance.

VIP-Manager manages the VIP. It reads the Leader Key that Patroni writes to DCS (etcd) to determine whether the local node is primary.


Enable VIP

To enable VIP for a cluster, set pg_vip_enabled to true at cluster level. You can also enable it globally.

# pgsql 3 node ha cluster: pg-test
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
    10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
    10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
  vars:
    pg_cluster: pg-test           # define pgsql cluster name
    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    pg_databases: [{ name: test }]

    # Enable L2 VIP
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.3/24
    #pg_vip_interface: auto

pg_vip_address must be an available, valid IP address with a subnet in the current L2 network.

pg_vip_interface defaults to auto. With this setting, Pigsty detects each instance interface from its inventory IPv4 address.

If automatic detection is unsuitable for a non-standard or policy-routing environment, specify a valid interface name for each instance. For example:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_vip_interface: eth0  }
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_vip_interface: eth1  }
    10.10.10.13: { pg_seq: 3, pg_role: replica , pg_vip_interface: ens33 }
  vars:
    pg_cluster: pg-test           # define pgsql cluster name
    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    pg_databases: [{ name: test }]

    # Enable L2 VIP
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.3/24

To refresh the VIP configuration and restart the VIP-Manager, use the following command:

./pgsql.yml -t pg_vip

7.8 - Deploy HA Citus Cluster

How to deploy a Citus high-availability distributed cluster?

Citus is a PostgreSQL extension that turns PostgreSQL into a distributed database. It scales horizontally across nodes to process large data volumes and query workloads.

Patroni v3.0+ provides native high-availability support for Citus. This simplifies Citus cluster setup. Pigsty also provides native support.

Note: Citus 13.x supports PostgreSQL 18, 17, 16, 15, and 14. Pigsty extension repo provides Citus ARM64 packages.


Citus Cluster

Pigsty natively supports Citus. The current complete configuration template is conf/ha/citus.yml.

The following four-node example shows the key parameters. It has a two-node coordinator cluster, pg-citus0, and two single-node Worker clusters, pg-citus1 and pg-citus2. It is not an exact excerpt from the current complete template.

pg-citus:
  hosts:
    10.10.10.10: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.2/24 ,pg_seq: 1, pg_role: primary }
    10.10.10.11: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.2/24 ,pg_seq: 2, pg_role: replica }
    10.10.10.12: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.3/24 ,pg_seq: 1, pg_role: primary }
    10.10.10.13: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.4/24 ,pg_seq: 1, pg_role: primary }
  vars:
    pg_mode: citus                            # pgsql cluster mode: citus
    pg_version: 18                            # citus 13.x supports PG 14-18
    pg_shard: pg-citus                        # citus shard name: pg-citus
    pg_primary_db: citus                      # primary database used by citus
    pg_vip_enabled: true                      # enable vip for citus cluster
    pg_vip_interface: auto                    # auto detect vip interface for every member
    pg_dbsu_password: DBUser.Postgres         # all dbsu password access for citus cluster
    pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]  # install these extensions
    pg_libs: 'citus, pg_cron, pg_stat_statements' # explicitly preload citus and keep it first
    pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ]    }]
    pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
    pg_parameters:
      cron.database_name: citus
      citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

Citus has additional configuration requirements beyond those for a standard PostgreSQL cluster. First, download, install, load, and enable the Citus extension. Use these four parameters:

  • repo_packages: Must include the citus extension, or you need to use a PostgreSQL offline package that includes Citus.
  • pg_extensions: Must include the citus extension, i.e., you must install the citus extension on each node.
  • pg_libs: Must explicitly include citus in the first position; current Patroni templates use this parameter directly for shared_preload_libraries.
  • pg_databases: Define a primary database that must have the citus extension installed.

Then configure the Citus cluster:

  • pg_mode: Must be set to citus to tell Patroni to use Citus mode.
  • pg_primary_db: Must specify the name of the primary database with citus extension, named citus here.
  • pg_shard: Must specify a unified name as the cluster name prefix for all horizontal shard PG clusters, pg-citus here.
  • pg_group: Set an integer shard number, starting from zero. 0 identifies the coordinator cluster. Other values identify Worker clusters.
  • pg_cluster: Must be unique among physical PostgreSQL clusters. The usual name combines pg_shard with a sequence number. The current role does not require the name to equal the concatenation of pg_shard and pg_group.
  • pg_dbsu_password: Must be set to a non-empty plaintext password, otherwise Citus will not work properly.
  • pg_parameters: Recommended to set citus.node_conninfo to enforce SSL access and require node-to-node client certificate verification.

After configuration, you can deploy the Citus cluster using pgsql.yml just like a regular PostgreSQL cluster.


Manage Citus Cluster

After defining the Citus cluster, deploy it using the pgsql.yml playbook:

./pgsql.yml -l pg-citus    # Deploy Citus cluster pg-citus

On any member, use the DBSU (postgres) account to show Citus cluster status with patronictl, whose alias is pg:

$ pg list
+ Citus cluster: pg-citus ----------+---------+-----------+----+-----------+--------------------+
| Group | Member      | Host        | Role    | State     | TL | Lag in MB | Tags               |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     0 | pg-citus0-1 | 10.10.10.10 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 20C.40G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     1 | pg-citus1-1 | 10.10.10.11 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     2 | pg-citus2-1 | 10.10.10.12 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     2 | pg-citus2-2 | 10.10.10.13 | Replica | streaming |  1 |         0 | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+

Manage each horizontal shard as an independent PGSQL cluster with pg (patronictl). For Citus, specify the shard number with --group:

pg list pg-citus --group 0   # Use --group 0 to specify cluster shard number

The Citus system table pg_dist_node records cluster node information. Patroni maintains this table automatically.

PGURL=postgres://postgres:[email protected]/citus

psql $PGURL -c 'SELECT * FROM pg_dist_node;'       # View node information
 nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole  | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-------------+----------+----------+-------------+----------+-----------+-------------+----------------+------------------
      1 |       0 | 10.10.10.10 |     5432 | default  | t           | t        | primary   | default     | t              | f
      4 |       1 | 10.10.10.12 |     5432 | default  | t           | t        | primary   | default     | t              | t
      5 |       2 | 10.10.10.13 |     5432 | default  | t           | t        | primary   | default     | t              | t
      6 |       0 | 10.10.10.11 |     5432 | default  | t           | t        | secondary | default     | t              | f

You can also view user authentication information (superuser access only):

$ psql $PGURL -c 'SELECT * FROM pg_dist_authinfo;'   # View node auth info (superuser only)

You can then connect with a regular application user, such as dbuser_citus with DDL privileges:

psql postgres://dbuser_citus:[email protected]/citus -c 'SELECT * FROM pg_dist_node;'

Using Citus Cluster

We strongly recommend the Citus official documentation for its architecture and core concepts.

Understand the characteristics and uses of the five Citus table types:

  • Distributed Table
  • Reference Table
  • Local Table
  • Local Management Table
  • Schema Table

On the coordinator, create distributed tables and reference tables. You can query them from any data node. From Citus 11.2, any Citus database node can act as coordinator.

Use pgbench to create tables. Distribute the main table, pgbench_accounts, across nodes. Use the other small tables as reference tables:

PGURL=postgres://dbuser_citus:[email protected]/citus
pgbench -i $PGURL

psql $PGURL <<-EOF
SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table('public.pgbench_accounts');
SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table('public.pgbench_branches');
SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_history');
SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_tellers');
EOF

Run read/write tests:

pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]/citus      # Direct connect to coordinator port 5432
pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]:6432/citus # Through connection pool, reduce client connection pressure
pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]/citus      # Any primary node can act as coordinator
pgbench --select-only -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]/citus # Read-only queries

Production Deployment

For production, you usually need physical streaming-replication replicas for the Coordinator and each Worker cluster.

The current conf/ha/citus.yml defines one pg-meta instance and 12 Citus instances across 13 hosts. The Citus instances form six two-node physical clusters with pg_group values 0-5. The following 10-node example shows a separate production topology. It is not the current template.

pg-citus: # citus group
  hosts:
    10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
  vars:
    pg_mode: citus                            # pgsql cluster mode: citus
    pg_version: 18                            # citus 13.x supports PG 14-18
    pg_shard: pg-citus                        # citus shard name: pg-citus
    pg_primary_db: citus                      # primary database used by citus
    pg_vip_enabled: true                      # enable vip for citus cluster
    pg_vip_interface: auto                    # auto detect vip interface for every member
    pg_dbsu_password: DBUser.Postgres         # enable dbsu password access for citus
    pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]  # install these extensions
    pg_libs: 'citus, pg_cron, pg_stat_statements' # citus will be added by patroni automatically
    pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ]    }]
    pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
    pg_parameters:
      cron.database_name: citus
      citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

We will cover a series of advanced Citus topics in subsequent tutorials:

  • Read/write separation
  • Failure handling
  • Consistent backup and recovery
  • Advanced monitoring and diagnostics
  • Connection pooling

8 - Monitoring

Overview of Pigsty’s monitoring system architecture and how to monitor existing PostgreSQL instances

This page describes the Pigsty monitoring architecture, including metrics, logs, and target management. It also explains how to monitor existing PostgreSQL clusters and remote RDS services.


Monitoring Overview

Pigsty uses these observability components for PostgreSQL monitoring:

  • Grafana displays metrics and provides PostgreSQL data sources.
  • VictoriaMetrics collects PostgreSQL, Pgbouncer, Patroni, HAProxy, and node metrics.
  • VictoriaLogs stores PostgreSQL, Pgbouncer, Patroni, pgBackRest, and host component logs.
  • Included Grafana dashboards cover all PostgreSQL operations. The set changes with each release. Use the live demo and this page’s dashboard links.

Metrics

The pg_exporter configuration defines all PostgreSQL monitoring metrics: roles/pg_monitor/templates/pg_exporter.yml. VictoriaMetrics and vmalert recording and alerting rules process them further: files/victoria/rules/pgsql.yml.

Pigsty attaches three identity labels to all metrics and logs: cls, ins, and ip. It also uses metrics from Pgbouncer, host nodes (NODE), and load balancers. Where possible, these metrics use the same labels so you can compare related data.

- { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
- { cls: pg-test, ins: pg-test-1, ip: 10.10.10.11 }
- { cls: pg-test, ins: pg-test-2, ip: 10.10.10.12 }
- { cls: pg-test, ins: pg-test-3, ip: 10.10.10.13 }

Logs

Vector collects PostgreSQL-related logs and sends them to VictoriaLogs on infrastructure nodes for storage and queries.

Target Management

Each VictoriaMetrics monitoring target instance has a static file in /infra/targets/pgsql/. This example defines pg-meta-1:

# pg-meta-1 [primary] @ 10.10.10.10
- labels: { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
  targets:
    - 10.10.10.10:9630    # <--- pg_exporter for PostgreSQL metrics
    - 10.10.10.10:9631    # <--- pgbouncer_exporter for PgBouncer metrics
    - 10.10.10.10:8008    # <--- Patroni metrics (when API SSL is not enabled)
    - 10.10.10.10:9854    # <--- pgbackrest_exporter for backup metrics

If patroni_ssl_enabled is enabled globally, Patroni uses HTTPS scrape endpoints. Pigsty writes these targets separately to /infra/targets/patroni/<ins>.yml. For RDS monitoring, targets are in /infra/targets/pgrds/ and are managed by cluster.

bin/pgsql-rm and pgsql-rm.yml remove the corresponding monitoring targets when they remove a cluster. You can also run:

bin/pgmon-rm <cls|ins>    # Remove monitoring targets from all infra nodes

pgsql-monitor.yml and bin/pgmon-add create remote RDS targets in /infra/targets/pgrds/<cls>.yml.


Monitoring Modes

Pigsty provides three monitoring modes to suit different monitoring needs.

Item \ LevelL1L2L3
NameBasicManagedStandard
AbbrRDSMANAGEDFULL
ScenarioConnection string only, e.g., RDSExisting DB, nodes manageableInstances created by Pigsty
PGCAT Features✅ Fully Available✅ Fully Available✅ Fully Available
PGSQL Features✅ PG metrics only✅ PG & node metrics only✅ Full Features
Connection Pool Metrics❌ Not Available⚠️ Optional✅ Pre-installed
Load Balancer Metrics❌ Not Available⚠️ Optional✅ Pre-installed
PGLOG Features❌ Not Available⚠️ Optional✅ Pre-installed
PG Exporter⚠️ On infra nodes✅ On DB nodes✅ On DB nodes
Node Exporter❌ Not deployed✅ On DB nodes✅ On DB nodes
Intrusiveness✅ Non-intrusive⚠️ Install Exporter⚠️ Fully managed by Pigsty
Monitor Existing Instances✅ Supported✅ Supported❌ For Pigsty-managed only
Monitoring Users & ViewsManual setupManual setupAuto-created by Pigsty
Deployment Playbookbin/pgmon-add <cls>Partial pgsql.yml/node.ymlpgsql.yml
Required PermissionsConnectable PGURL from infraSSH & sudo on DB nodesSSH & sudo on DB nodes
Feature SummaryPGCAT + PGRDSMost featuresFull features

Pigsty automatically monitors fully managed databases with all supported features. They usually need no extra configuration. For existing PostgreSQL clusters or RDS services, use managed deployment if Pigsty can access the nodes through SSH and sudo. This gives monitoring similar to a native Pigsty deployment. If you have only a PGURL connection string, use basic mode.


Monitor Existing Cluster

If Pigsty can manage the target database nodes, deploy monitoring components there with the pg_exporter task in pgsql.yml. This requires ssh access and sudo privileges. Use pgbouncer and pgbouncer_exporter from the same playbook to deploy connection pools and pool monitoring. Use node_exporter, haproxy, and vector from node.yml for host monitoring, load balancing, and log collection. These components provide the same monitoring features as native Pigsty database instances.

Define existing clusters in the same way as Pigsty-managed clusters. Run only the required pgsql.yml tasks, rather than the complete playbook.

./node.yml  -l <cls> -t node_repo,node_pkg           # Add YUM repos from INFRA nodes and install packages on host nodes
./node.yml  -l <cls> -t node_exporter,node_register  # Configure host monitoring and add to VictoriaMetrics
./node.yml  -l <cls> -t vector                       # Configure host log collection and send to VictoriaLogs
./pgsql.yml -l <cls> -t pg_exporter,pg_register      # Configure PostgreSQL monitoring and register with VictoriaMetrics/Grafana

For an existing cluster, manually create monitoring users, schemas, and extensions in the target database.


Monitor RDS

If you have only a PGURL connection string, follow this section. Pigsty deploys PG Exporters on INFRA nodes to collect remote database metrics:

------ infra ------
|                 |
| victoria-metrics|            v---- pg-foo-1 ----v
|       ^         |  metrics   |         ^        |
|   pg_exporter <-|------------|----  postgres    |
|   (port: 20001) |            | 10.10.10.10:5432 |
|       ^         |            ^------------------^
|       ^         |                      ^
|       ^         |            v---- pg-foo-2 ----v
|       ^         |  metrics   |         ^        |
|   pg_exporter <-|------------|----  postgres    |
|   (port: 20002) |            | 10.10.10.11:5433 |
-------------------            ^------------------^

This mode provides database metrics and current status from the data catalog. It does not provide host, connection pool, load balancer, or high-availability component metrics. Pigsty provides two dashboards for PostgreSQL metrics: PGRDS Cluster and PGRDS Instance. Overview and database monitoring use the existing dashboards. Because Pigsty cannot manage RDS, configure monitoring objects on the target database first.

Limitations when monitoring external Postgres instances
  • PgBouncer connection pool metrics are not available
  • Patroni high availability component metrics are not available
  • Host node monitoring metrics are not available, including node HAProxy and Keepalived metrics
  • Log collection and log-derived metrics are not available

The following example uses the sandbox. Treat pg-meta as the RDS instance pg-foo-1, and pg-test as the RDS cluster pg-bar.

  1. Create monitoring schemas, users, and permissions on the target. Refer to Monitor Setup for details

  2. Declare the cluster in the configuration inventory. For example, if we want to monitor “remote” pg-meta & pg-test clusters:

    infra:            # Infra cluster for proxies, monitoring, alerts, etc.
      hosts: { 10.10.10.10: { infra_seq: 1 } }
      vars:           # Install pg_exporter on group 'infra' for remote postgres RDS
        pg_exporters: # List all remote instances here, assign a unique unused local port for k
          20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 , pg_databases: [{ name: meta }] } # Register meta database as Grafana datasource
    
          20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.11 , pg_port: 5432 } # Different connection string methods
          20003: { pg_cluster: pg-bar, pg_seq: 2, pg_host: 10.10.10.12 , pg_exporter_url: 'postgres://dbuser_monitor:[email protected]:5432/postgres?sslmode=disable'}
          20004: { pg_cluster: pg-bar, pg_seq: 3, pg_host: 10.10.10.13 , pg_monitor_username: dbuser_monitor, pg_monitor_password: DBUser.Monitor }

    Pigsty registers databases from pg_databases as PostgreSQL data sources in Grafana for PGCAT dashboards. If you do not need PGCAT or Grafana registration, set pg_databases to an empty array or leave it blank.

    pigsty-monitor.jpg
  3. Execute the add monitoring command: bin/pgmon-add <clsname>

    bin/pgmon-add pg-foo  # Bring pg-foo cluster into monitoring
    bin/pgmon-add pg-bar  # Bring pg-bar cluster into monitoring
  4. To remove remote cluster monitoring targets, use bin/pgmon-rm <clsname>

    bin/pgmon-rm pg-foo  # Remove pg-foo from Pigsty monitoring
    bin/pgmon-rm pg-bar  # Remove pg-bar from Pigsty monitoring

Use additional parameters to override default pg_exporter options. This example monitors Aliyun RDS for PostgreSQL and PolarDB:

Example: Monitoring Aliyun RDS for PostgreSQL and PolarDB

For details, refer to: remote.yml

infra:            # Infra cluster for proxies, monitoring, alerts, etc.
  hosts: { 10.10.10.10: { infra_seq: 1 } }
  vars:
    pg_exporters:   # List all remote RDS PG instances to be monitored here

      20001:        # Assign a unique unused local port for local monitoring agent, this is a PolarDB primary
        pg_cluster: pg-polar                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pc-2ze379wb1d4irc18x.polardbpg.rds.aliyuncs.com # RDS host address
        pg_port: 1921                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'test'  # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username, overrides global config
        pg_monitor_password: DBUser_Monitor   # Monitoring password, overrides global config
        pg_databases: [{ name: test }]        # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20002:       # This is a PolarDB standby
        pg_cluster: pg-polar                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 2                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pe-2ze7tg620e317ufj4.polarpgmxs.rds.aliyuncs.com # RDS host address
        pg_port: 1521                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'test,postgres'  # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username
        pg_monitor_password: DBUser_Monitor   # Monitoring password
        pg_databases: [ { name: test } ]        # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20004: # This is a basic single-node RDS for PostgreSQL instance
        pg_cluster: pg-rds                    # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgm-2zern3d323fe9ewk.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username
        pg_monitor_password: DBUser_Monitor   # Monitoring password
        pg_databases: [ { name: rds } ]       # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20005: # This is a high-availability RDS for PostgreSQL cluster primary
        pg_cluster: pg-rdsha                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgm-2ze3d35d27bq08wu.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_databases: [ { name: rds }, {name : test} ]  # Include these two databases in PGCAT management, register as Grafana datasources

      20006: # This is a high-availability RDS for PostgreSQL cluster read-only instance (standby)
        pg_cluster: pg-rdsha                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 2                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgr-2zexqxalk7d37edt.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_databases: [ { name: rds }, {name : test} ]  # Include these two databases in PGCAT management, register as Grafana datasources

Monitor Setup

Before you monitor an existing RDS or self-managed PostgreSQL instance, configure the target database for Pigsty access.

To monitor an external PostgreSQL instance, provide a connection string that can access the instance or cluster. A business user or superuser connection string can work. However, use a dedicated monitoring user to avoid exposing other privileges.

  • Monitor User: The default username is dbuser_monitor, which should belong to the pg_monitor role group or have access to relevant views
  • Monitor Authentication: Default password authentication is used; ensure HBA policies allow the monitoring user to access databases from the admin node or DB node locally
  • Monitor Schema: Fixed schema name monitor is used for installing additional monitoring views and extension plugins; optional but recommended
  • Monitor Extension: Strongly recommended to enable the built-in monitoring extension pg_stat_statements
  • Monitor Views: Monitoring views are optional but can provide additional metric support

Monitor User

Using the default monitoring user dbuser_monitor as an example, create the following user on the target database cluster.

CREATE USER dbuser_monitor;                                       -- Create monitoring user
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';          -- Comment on monitoring user
GRANT pg_monitor TO dbuser_monitor;                               -- Grant pg_monitor privilege to monitoring user, otherwise some metrics cannot be collected

ALTER USER dbuser_monitor PASSWORD 'DBUser.Monitor';              -- Modify monitoring user password as needed (strongly recommended! but keep consistent with Pigsty config)
ALTER USER dbuser_monitor SET log_min_duration_statement = 1000;  -- Recommended to avoid logs filling up with monitoring slow queries
ALTER USER dbuser_monitor SET search_path = monitor,public;       -- Recommended to ensure pg_stat_statements extension works properly

The monitoring username and password must match pg_monitor_username and pg_monitor_password.


Monitor Authentication

Add these rules to pg_hba.conf. They allow the monitoring user to access all databases from localhost and the administration node with password authentication:

# allow local role monitor with password
local   all  dbuser_monitor                    md5
host    all  dbuser_monitor  127.0.0.1/32      md5
host    all  dbuser_monitor  <admin_machine_IP>/32 md5

If RDS does not allow HBA configuration, allow the internal IP address of the machine that runs Pigsty.


Monitor Schema

The monitoring schema is optional. Core Pigsty monitoring works without it. However, creating this schema is strongly recommended.

CREATE SCHEMA IF NOT EXISTS monitor;               -- Create dedicated monitoring schema
GRANT USAGE ON SCHEMA monitor TO dbuser_monitor;   -- Allow monitoring user to use it

Monitor Extension

The monitoring extension is optional. However, enabling pg_stat_statements is strongly recommended because it supplies important query performance data.

Add this extension to shared_preload_libraries to enable it. Changing this parameter requires a database restart.

CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

Install this extension in the default administration database, postgres. Some RDS services do not allow a monitoring schema in postgres. In this case, install pg_stat_statements in the default public schema. Configure the monitoring user search_path as shown above so it can find the pg_stat_statements view.

CREATE EXTENSION IF NOT EXISTS "pg_stat_statements";
ALTER USER dbuser_monitor SET search_path = monitor,public; -- Recommended to ensure pg_stat_statements extension works properly

Monitor Views

Monitoring views provide common preprocessed query results. They also control access to metrics that need high privileges, such as shared memory allocation. This makes these metrics easier to query and use. Creating the views in every monitored database is strongly recommended.

Monitoring schema and monitoring view definitions

The SQL below is provided to explain the monitored objects. The complete definitions rendered by current Pigsty are authoritative in roles/pgsql/templates/pg-init-template.sql, which also includes additional hardening for secure search paths and privilege boundaries.

----------------------------------------------------------------------
-- Table bloat estimate : monitor.pg_table_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, relname , tblid , bs * tblpages AS size,
       CASE WHEN tblpages - est_tblpages_ff > 0 THEN (tblpages - est_tblpages_ff)/tblpages::FLOAT ELSE 0 END AS ratio
FROM (
         SELECT ceil( reltuples / ( (bs-page_hdr)*fillfactor/(tpl_size*100) ) ) + ceil( toasttuples / 4 ) AS est_tblpages_ff,
                tblpages, fillfactor, bs, tblid, nspname, relname, is_na
         FROM (
                  SELECT
                      ( 4 + tpl_hdr_size + tpl_data_size + (2 * ma)
                          - CASE WHEN tpl_hdr_size % ma = 0 THEN ma ELSE tpl_hdr_size % ma END
                          - CASE WHEN ceil(tpl_data_size)::INT % ma = 0 THEN ma ELSE ceil(tpl_data_size)::INT % ma END
                          ) AS tpl_size, (heappages + toastpages) AS tblpages, heappages,
                      toastpages, reltuples, toasttuples, bs, page_hdr, tblid, nspname, relname, fillfactor, is_na
                  FROM (
                           SELECT
                               tbl.oid AS tblid, ns.nspname , tbl.relname, tbl.reltuples,
                               tbl.relpages AS heappages, coalesce(toast.relpages, 0) AS toastpages,
                               coalesce(toast.reltuples, 0) AS toasttuples,
                               coalesce(substring(array_to_string(tbl.reloptions, ' ') FROM 'fillfactor=([0-9]+)')::smallint, 100) AS fillfactor,
                               current_setting('block_size')::numeric AS bs,
                               CASE WHEN version()~'mingw32' OR version()~'64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END AS ma,
                               24 AS page_hdr,
                               23 + CASE WHEN MAX(coalesce(s.null_frac,0)) > 0 THEN ( 7 + count(s.attname) ) / 8 ELSE 0::int END
                                   + CASE WHEN bool_or(att.attname = 'oid' and att.attnum < 0) THEN 4 ELSE 0 END AS tpl_hdr_size,
                               sum( (1-coalesce(s.null_frac, 0)) * coalesce(s.avg_width, 0) ) AS tpl_data_size,
                               bool_or(att.atttypid = 'pg_catalog.name'::regtype)
                                   OR sum(CASE WHEN att.attnum > 0 THEN 1 ELSE 0 END) <> count(s.attname) AS is_na
                           FROM pg_attribute AS att
                                    JOIN pg_class AS tbl ON att.attrelid = tbl.oid
                                    JOIN pg_namespace AS ns ON ns.oid = tbl.relnamespace
                                    LEFT JOIN pg_stats AS s ON s.schemaname=ns.nspname AND s.tablename = tbl.relname AND s.inherited=false AND s.attname=att.attname
                                    LEFT JOIN pg_class AS toast ON tbl.reltoastrelid = toast.oid
                           WHERE NOT att.attisdropped AND tbl.relkind = 'r' AND nspname NOT IN ('pg_catalog','information_schema')
                           GROUP BY 1,2,3,4,5,6,7,8,9,10
                       ) AS s
              ) AS s2
     ) AS s3
WHERE NOT is_na;
COMMENT ON VIEW monitor.pg_table_bloat IS 'postgres table bloat estimate';

GRANT SELECT ON monitor.pg_table_bloat TO pg_monitor;

----------------------------------------------------------------------
-- Index bloat estimate : monitor.pg_index_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, idxname AS relname, tblid, idxid, relpages::BIGINT * bs AS size,
       COALESCE((relpages - ( reltuples * (6 + ma - (CASE WHEN index_tuple_hdr % ma = 0 THEN ma ELSE index_tuple_hdr % ma END)
                                               + nulldatawidth + ma - (CASE WHEN nulldatawidth % ma = 0 THEN ma ELSE nulldatawidth % ma END))
                                  / (bs - pagehdr)::FLOAT  + 1 )), 0) / relpages::FLOAT AS ratio
FROM (
         SELECT nspname,idxname,indrelid AS tblid,indexrelid AS idxid,
                reltuples,relpages,
                current_setting('block_size')::INTEGER                                                               AS bs,
                (CASE WHEN version() ~ 'mingw32' OR version() ~ '64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END)  AS ma,
                24                                                                                                   AS pagehdr,
                (CASE WHEN max(COALESCE(pg_stats.null_frac, 0)) = 0 THEN 2 ELSE 6 END)                               AS index_tuple_hdr,
                sum((1.0 - COALESCE(pg_stats.null_frac, 0.0)) *
                    COALESCE(pg_stats.avg_width, 1024))::INTEGER                                                     AS nulldatawidth
         FROM pg_attribute
                  JOIN (
             SELECT pg_namespace.nspname,
                    ic.relname                                                   AS idxname,
                    ic.reltuples,
                    ic.relpages,
                    pg_index.indrelid,
                    pg_index.indexrelid,
                    tc.relname                                                   AS tablename,
                    regexp_split_to_table(pg_index.indkey::TEXT, ' ') :: INTEGER AS attnum,
                    pg_index.indexrelid                                          AS index_oid
             FROM pg_index
                      JOIN pg_class ic ON pg_index.indexrelid = ic.oid
                      JOIN pg_class tc ON pg_index.indrelid = tc.oid
                      JOIN pg_namespace ON pg_namespace.oid = ic.relnamespace
                      JOIN pg_am ON ic.relam = pg_am.oid
             WHERE pg_am.amname = 'btree' AND ic.relpages > 0 AND nspname NOT IN ('pg_catalog', 'information_schema')
         ) ind_atts ON pg_attribute.attrelid = ind_atts.indexrelid AND pg_attribute.attnum = ind_atts.attnum
                  JOIN pg_stats ON pg_stats.schemaname = ind_atts.nspname
             AND ((pg_stats.tablename = ind_atts.tablename AND pg_stats.attname = pg_get_indexdef(pg_attribute.attrelid, pg_attribute.attnum, TRUE))
                 OR (pg_stats.tablename = ind_atts.idxname AND pg_stats.attname = pg_attribute.attname))
         WHERE pg_attribute.attnum > 0
         GROUP BY 1, 2, 3, 4, 5, 6
     ) est;
COMMENT ON VIEW monitor.pg_index_bloat IS 'postgres index bloat estimate (btree-only)';

GRANT SELECT ON monitor.pg_index_bloat TO pg_monitor;

----------------------------------------------------------------------
-- Relation Bloat : monitor.pg_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_bloat AS
SELECT coalesce(ib.datname, tb.datname)                                                   AS datname,
       coalesce(ib.nspname, tb.nspname)                                                   AS nspname,
       coalesce(ib.tblid, tb.tblid)                                                       AS tblid,
       coalesce(tb.nspname || '.' || tb.relname, ib.nspname || '.' || ib.tblid::RegClass) AS tblname,
       tb.size                                                                            AS tbl_size,
       CASE WHEN tb.ratio < 0 THEN 0 ELSE round(tb.ratio::NUMERIC, 6) END                 AS tbl_ratio,
       (tb.size * (CASE WHEN tb.ratio < 0 THEN 0 ELSE tb.ratio::NUMERIC END)) ::BIGINT    AS tbl_wasted,
       ib.idxid,
       ib.nspname || '.' || ib.relname                                                    AS idxname,
       ib.size                                                                            AS idx_size,
       CASE WHEN ib.ratio < 0 THEN 0 ELSE round(ib.ratio::NUMERIC, 5) END                 AS idx_ratio,
       (ib.size * (CASE WHEN ib.ratio < 0 THEN 0 ELSE ib.ratio::NUMERIC END)) ::BIGINT    AS idx_wasted
FROM monitor.pg_index_bloat ib
         FULL OUTER JOIN monitor.pg_table_bloat tb ON ib.tblid = tb.tblid;

COMMENT ON VIEW monitor.pg_bloat IS 'postgres relation bloat detail';
GRANT SELECT ON monitor.pg_bloat TO pg_monitor;

----------------------------------------------------------------------
-- monitor.pg_index_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat_human AS
SELECT idxname                            AS name,
       tblname,
       idx_wasted                         AS wasted,
       pg_size_pretty(idx_size)           AS idx_size,
       round(100 * idx_ratio::NUMERIC, 2) AS idx_ratio,
       pg_size_pretty(idx_wasted)         AS idx_wasted,
       pg_size_pretty(tbl_size)           AS tbl_size,
       round(100 * tbl_ratio::NUMERIC, 2) AS tbl_ratio,
       pg_size_pretty(tbl_wasted)         AS tbl_wasted
FROM monitor.pg_bloat
WHERE idxname IS NOT NULL;
COMMENT ON VIEW monitor.pg_index_bloat_human IS 'postgres index bloat info in human-readable format';
GRANT SELECT ON monitor.pg_index_bloat_human TO pg_monitor;


----------------------------------------------------------------------
-- monitor.pg_table_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat_human AS
SELECT tblname                                          AS name,
       idx_wasted + tbl_wasted                          AS wasted,
       pg_size_pretty(idx_wasted + tbl_wasted)          AS all_wasted,
       pg_size_pretty(tbl_wasted)                       AS tbl_wasted,
       pg_size_pretty(tbl_size)                         AS tbl_size,
       tbl_ratio,
       pg_size_pretty(idx_wasted)                       AS idx_wasted,
       pg_size_pretty(idx_size)                         AS idx_size,
       round(idx_wasted::NUMERIC * 100.0 / idx_size, 2) AS idx_ratio
FROM (SELECT datname,
             nspname,
             tblname,
             coalesce(max(tbl_wasted), 0)                         AS tbl_wasted,
             coalesce(max(tbl_size), 1)                           AS tbl_size,
             round(100 * coalesce(max(tbl_ratio), 0)::NUMERIC, 2) AS tbl_ratio,
             coalesce(sum(idx_wasted), 0)                         AS idx_wasted,
             coalesce(sum(idx_size), 1)                           AS idx_size
      FROM monitor.pg_bloat
      WHERE tblname IS NOT NULL
      GROUP BY 1, 2, 3
     ) d;
COMMENT ON VIEW monitor.pg_table_bloat_human IS 'postgres table bloat info in human-readable format';
GRANT SELECT ON monitor.pg_table_bloat_human TO pg_monitor;


----------------------------------------------------------------------
-- Activity Overview: monitor.pg_session
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_session CASCADE;
CREATE OR REPLACE VIEW monitor.pg_session AS
SELECT coalesce(datname, 'all') AS datname, numbackends, active, idle, ixact, max_duration, max_tx_duration, max_conn_duration
FROM (
         SELECT datname,
                count(*)                                         AS numbackends,
                count(*) FILTER ( WHERE state = 'active' )       AS active,
                count(*) FILTER ( WHERE state = 'idle' )         AS idle,
                count(*) FILTER ( WHERE state = 'idle in transaction'
                    OR state = 'idle in transaction (aborted)' ) AS ixact,
                max(extract(epoch from now() - state_change))
                FILTER ( WHERE state = 'active' )                AS max_duration,
                max(extract(epoch from now() - xact_start))      AS max_tx_duration,
                max(extract(epoch from now() - backend_start))   AS max_conn_duration
         FROM pg_stat_activity
         WHERE backend_type = 'client backend'
           AND pid <> pg_backend_pid()
         GROUP BY ROLLUP (1)
         ORDER BY 1 NULLS FIRST
     ) t;
COMMENT ON VIEW monitor.pg_session IS 'postgres activity group by session';
GRANT SELECT ON monitor.pg_session TO pg_monitor;


----------------------------------------------------------------------
-- Sequential Scan: monitor.pg_seq_scan
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_seq_scan CASCADE;
CREATE OR REPLACE VIEW monitor.pg_seq_scan AS
SELECT schemaname                                                        AS nspname,
       relname,
       seq_scan,
       seq_tup_read,
       seq_tup_read / seq_scan                                           AS seq_tup_avg,
       idx_scan,
       n_live_tup + n_dead_tup                                           AS tuples,
       round(n_live_tup * 100.0::NUMERIC / (n_live_tup + n_dead_tup), 2) AS live_ratio
FROM pg_stat_user_tables
WHERE seq_scan > 0
  and (n_live_tup + n_dead_tup) > 0
ORDER BY seq_scan DESC;
COMMENT ON VIEW monitor.pg_seq_scan IS 'table that have seq scan';
GRANT SELECT ON monitor.pg_seq_scan TO pg_monitor;
Function for viewing shared memory allocation (PG13 and above)
DROP FUNCTION IF EXISTS monitor.pg_shmem() CASCADE;
CREATE OR REPLACE FUNCTION monitor.pg_shmem() RETURNS SETOF
    pg_shmem_allocations SET search_path = '' AS $$ SELECT * FROM pg_shmem_allocations;$$ LANGUAGE SQL SECURITY DEFINER;
COMMENT ON FUNCTION monitor.pg_shmem() IS 'security wrapper for system view pg_shmem';
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM PUBLIC;
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM dbrole_readonly;
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM dbrole_offline;
GRANT EXECUTE ON FUNCTION monitor.pg_shmem() TO pg_monitor;

9 - Dashboard

Pigsty provides Grafana monitoring dashboards for PostgreSQL

Pigsty provides Grafana monitoring dashboards for PostgreSQL: Demo & Gallery.

The current source provides 31 PostgreSQL-related dashboards. It includes 29 PostgreSQL / PGCAT dashboards under files/grafana/pgsql and two PGLOG dashboards under files/grafana/app. The hierarchy has four categories: Overview, Cluster, Instance, and Database. The data source categories are PGSQL, PGCAT, and PGLOG.

pigsty-dashboard.jpg

Overview

Overview

  • pgsql-overview: Main dashboard for the PGSQL module
  • pgsql-alert: Global key metrics and alert events for PGSQL
  • pgsql-shard: Overview of horizontally sharded PGSQL clusters (e.g., Citus/GPSQL)

Cluster

  • pgsql-cluster: Main dashboard for a PGSQL cluster
  • pgrds-cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics
  • pgsql-activity: Session/load/QPS/TPS/locks for PGSQL cluster
  • pgsql-replication: Replication, slots, and pub/sub for PGSQL cluster
  • pgsql-service: Service, proxy, routing, and load balancing for PGSQL cluster
  • pgsql-databases: Database CRUD, slow queries, and table statistics across all instances
  • pgsql-patroni: HA status and Patroni component status for cluster
  • pgsql-pitr: PITR context for point-in-time recovery assistance

Instance

  • pgsql-instance: Main dashboard for a single PGSQL instance
  • pgrds-instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics
  • pgcat-instance: Instance info retrieved directly from database catalog
  • pgsql-proxy: Detailed metrics for a single HAProxy load balancer
  • pgsql-pgbouncer: Metrics overview for a single Pgbouncer connection pooler
  • pgsql-persist: Persistence metrics: WAL, XID, checkpoint, archive, IO
  • pgsql-session: Session and active/idle time metrics for a single instance
  • pgsql-xacts: Transaction, lock, TPS/QPS related metrics
  • pgsql-exporter: Self-monitoring metrics for Postgres and Pgbouncer exporters

Database

  • pgsql-database: Main dashboard for a single PGSQL database
  • pgcat-database: Database info retrieved directly from database catalog
  • pgsql-tables: Table/index access metrics within a single database
  • pgsql-table: Detailed info for a single table (QPS/RT/index/sequence…)
  • pgcat-table: Detailed table info from database catalog (stats/bloat…)
  • pgsql-query: Detailed info for a query type (QPS/RT)
  • pgcat-query: Query details from database catalog (SQL/stats)
  • pgcat-schema: Schema info from database catalog (tables/indexes/sequences…)
  • pgcat-locks: Activity and lock wait info from database catalog

Overview

PGSQL Overview: Main dashboard for the PGSQL module

PGSQL Overview

pgsql-overview.jpg

PGSQL Alert: Global core metrics overview and alert events

PGSQL Alert

pgsql-alert.jpg

PGSQL Shard: Cross-shard metric comparison for horizontally sharded PGSQL clusters (e.g., CITUS/GPSQL)

PGSQL Shard

pgsql-shard.jpg


Cluster

PGSQL Cluster: Main dashboard for a PGSQL cluster

PGSQL Cluster

pgsql-cluster.jpg

PGRDS Cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics

PGRDS Cluster

pgrds-cluster.jpg

PGSQL Service: Service, proxy, routing, and load balancing for PGSQL cluster

PGSQL Service

pgsql-service.jpg

PGSQL Activity: Session/load/QPS/TPS/locks for PGSQL cluster

PGSQL Activity

pgsql-activity.jpg

PGSQL Replication: Replication, slots, and pub/sub for PGSQL cluster

PGSQL Replication

pgsql-replication.jpg

PGSQL Databases: Database CRUD, slow queries, and table statistics across all instances

PGSQL Databases

pgsql-databases.jpg

PGSQL Patroni: HA status and Patroni component status for cluster

PGSQL Patroni

pgsql-patroni.jpg

PGSQL PITR: PITR context for point-in-time recovery assistance

PGSQL PITR

pgsql-patroni.jpg


Instance

PGSQL Instance: Main dashboard for a single PGSQL instance

PGSQL Instance

pgsql-instance.jpg

PGRDS Instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics

PGRDS Instance

pgrds-instance.jpg

PGSQL Proxy: Detailed metrics for a single HAProxy load balancer

PGSQL Proxy

pgsql-proxy.jpg

PGSQL Pgbouncer: Metrics overview for a single Pgbouncer connection pooler

PGSQL Pgbouncer

pgsql-pgbouncer.jpg

PGSQL Persist: Persistence metrics: WAL, XID, checkpoint, archive, IO

PGSQL Persist

pgsql-persist.jpg

PGSQL Xacts: Transaction, lock, TPS/QPS related metrics

PGSQL Xacts

pgsql-xacts.jpg

PGSQL Session: Session and active/idle time metrics for a single instance

PGSQL Session

pgsql-session.jpg

PGSQL Exporter: Self-monitoring metrics for Postgres/Pgbouncer exporters

PGSQL Exporter

pgsql-exporter.jpg


Database

PGSQL Database: Main dashboard for a single PGSQL database

PGSQL Database

pgsql-database.jpg

PGSQL Tables: Table/index access metrics within a single database

PGSQL Tables

pgsql-tables.jpg

PGSQL Table: Detailed info for a single table (QPS/RT/index/sequence…)

PGSQL Table

pgsql-table.jpg

PGSQL Query: Detailed info for a query type (QPS/RT)

PGSQL Query

pgsql-query.jpg


PGCAT

PGCAT Instance: Instance info retrieved directly from database catalog

PGCAT Instance

pgcat-instance.jpg

PGCAT Database: Database info retrieved directly from database catalog

PGCAT Database

pgcat-database.jpg

PGCAT Schema: Schema info from database catalog (tables/indexes/sequences…)

PGCAT Schema

pgcat-schema.jpg

PGCAT Table: Detailed table info from database catalog (stats/bloat…)

PGCAT Table

pgcat-table.jpg

PGCAT Query: Query details from database catalog (SQL/stats)

PGCAT Query

pgcat-query.jpg

PGCAT Locks: Activity and lock wait info from database catalog

PGCAT Locks

pgcat-locks.jpg


PGLOG

PGLOG Overview: Overview of CSV log samples in Pigsty CMDB

PGLOG Overview

pglog-overview.jpg

PGLOG Session: Log details for a single session in CSV log samples

PGLOG Session

pglog-session.jpg


See pigsty/wiki/gallery for details.

PGSQL Overview

pgsql-overview.jpg

PGSQL Shard

pgsql-shard.jpg

PGSQL Cluster

pgsql-cluster.jpg

PGSQL Service

pgsql-service.jpg

PGSQL Activity

pgsql-activity.jpg

PGSQL Replication

pgsql-replication.jpg

PGSQL Databases

pgsql-databases.jpg

PGSQL Instance

pgsql-instance.jpg

PGSQL Proxy

pgsql-proxy.jpg

PGSQL Pgbouncer

pgsql-pgbouncer.jpg

PGSQL Session

pgsql-session.jpg

PGSQL Xacts

pgsql-xacts.jpg

PGSQL Persist

pgsql-persist.jpg

PGSQL Database

pgsql-database.jpg

PGSQL Tables

pgsql-tables.jpg

PGSQL Table

pgsql-table.jpg

PGSQL Query

pgsql-query.jpg

PGCAT Instance

pgcat-instance.jpg

PGCAT Database

pgcat-database.jpg

PGCAT Schema

pgcat-schema.jpg

PGCAT Table

pgcat-table.jpg

PGCAT Lock

pgcat-locks.jpg

PGCAT Query

pgcat-query.jpg

PGLOG Overview

pglog-overview.jpg

PGLOG Session

pglog-session.jpg

9.1 - Overview

PostgreSQL module global overview monitoring dashboards

Use these dashboards for a global overview of the PostgreSQL module:

9.1.1 - PGSQL Overview

Main dashboard for the PGSQL module

Main dashboard for the PGSQL module: Demo

PGSQL Overview is the main dashboard for the PostgreSQL module. It shows a global overview of the entire PGSQL module.

pgsql-overview

9.1.2 - PGSQL Alert

Global key metrics and alert events for PGSQL

Global key metrics and alert events for PGSQL: Demo

PGSQL Alert provides a global overview of core metrics and alert events for PostgreSQL clusters.

pgsql-alert

9.1.3 - PGSQL Shard

Overview of horizontally sharded PGSQL clusters

Overview of horizontally sharded PGSQL clusters: Demo

PGSQL Shard provides cross-shard metric comparison for horizontally sharded PGSQL clusters such as CITUS or GPSQL.

pgsql-shard

9.2 - Cluster

PostgreSQL cluster-level monitoring dashboards

Use these dashboards to monitor PostgreSQL clusters:

  • PGSQL Cluster: Main dashboard for a PGSQL cluster
  • PGRDS Cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics
  • PGSQL Activity: Session/load/QPS/TPS/locks for PGSQL cluster
  • PGSQL Replication: Replication, slots, and pub/sub for PGSQL cluster
  • PGSQL Service: Service, proxy, routing, and load balancing for PGSQL cluster
  • PGSQL Databases: Database CRUD, slow queries, and table statistics across all instances
  • PGSQL Patroni: HA status and Patroni component status for cluster
  • PGSQL PITR: PITR context for point-in-time recovery assistance

9.2.1 - PGSQL Cluster

Main dashboard for a PGSQL cluster

Main dashboard for a PGSQL cluster: Demo

PGSQL Cluster is the main dashboard for a single PostgreSQL cluster. It shows the core metrics for the cluster.

pgsql-cluster

9.2.2 - PGRDS Cluster

RDS version of PGSQL Cluster focusing on PostgreSQL-native metrics

RDS version of PGSQL Cluster: Demo

PGRDS Cluster is the RDS version of PGSQL Cluster. It shows PostgreSQL-native metrics and excludes host-level metrics.

pgrds-cluster

9.2.3 - PGSQL Activity

Session/load/QPS/TPS/locks for PGSQL cluster

Session/load/QPS/TPS/locks for PGSQL cluster: Demo

PGSQL Activity focuses on session activity, load, QPS, TPS, and lock status for a PostgreSQL cluster.

pgsql-activity

9.2.4 - PGSQL Replication

Replication, slots, and pub/sub for PGSQL cluster

Replication, slots, and pub/sub for PGSQL cluster: Demo

PGSQL Replication focuses on replication status, replication slots, and logical replication (pub/sub) for a PostgreSQL cluster.

pgsql-replication

9.2.5 - PGSQL Service

Service, proxy, routing, and load balancing for PGSQL cluster

Service, proxy, routing, and load balancing for PGSQL cluster: Demo

PGSQL Service focuses on service endpoints, proxy routing, and load balancing status for a PostgreSQL cluster.

pgsql-service

9.2.6 - PGSQL Databases

Database CRUD, slow queries, and table statistics across all instances

Database CRUD, slow queries, and table statistics: Demo

PGSQL Databases focuses on database-level CRUD operations, slow queries, and table statistics across all instances in a cluster.

pgsql-databases

9.2.7 - PGSQL Patroni

HA status and Patroni component status for cluster

HA status and Patroni component status: Demo

PGSQL Patroni focuses on high-availability status and Patroni component health for a PostgreSQL cluster.

pgsql-patroni

9.2.8 - PGSQL PITR

PITR context for point-in-time recovery assistance

PITR context for point-in-time recovery: Demo

PGSQL PITR shows backup status and the WAL timeline. Use this information for point-in-time recovery operations.

pgsql-pitr

9.3 - Instance

PostgreSQL instance-level monitoring dashboards

Use these dashboards to monitor PostgreSQL instances:

  • PGSQL Instance: Main dashboard for a single PGSQL instance
  • PGRDS Instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics
  • PGCAT Instance: Instance info retrieved directly from database catalog
  • PGSQL Persist: Persistence metrics: WAL, XID, checkpoint, archive, IO
  • PGSQL Proxy: Detailed metrics for a single HAProxy load balancer
  • PGSQL Pgbouncer: Metrics overview for a single Pgbouncer connection pooler
  • PGSQL Session: Session and active/idle time metrics for a single instance
  • PGSQL Xacts: Transaction, lock, TPS/QPS related metrics
  • PGSQL Exporter: Self-monitoring metrics for Postgres and Pgbouncer exporters

9.3.1 - PGSQL Instance

Main dashboard for a single PGSQL instance

Main dashboard for a single PGSQL instance: Demo

PGSQL Instance is the main dashboard for a single PostgreSQL instance. It provides a full set of instance-level metrics.

pgsql-instance

9.3.2 - PGRDS Instance

RDS version of PGSQL Instance focusing on PostgreSQL-native metrics

RDS version of PGSQL Instance: Demo

PGRDS Instance is the RDS version of PGSQL Instance. It shows PostgreSQL-native metrics and excludes host-level metrics.

pgrds-instance

9.3.3 - PGCAT Instance

Instance info retrieved directly from database catalog

Instance info from database catalog: Demo

PGCAT Instance reads instance-level information directly from the PostgreSQL system catalog and shows it.

pgcat-instance

9.3.4 - PGSQL Persist

Persistence metrics - WAL, XID, checkpoint, archive, IO

Persistence metrics for PGSQL instance: Demo

PGSQL Persist focuses on persistence-related metrics: WAL generation, XID consumption, checkpoints, archiving, and I/O patterns.

pgsql-persist

9.3.5 - PGSQL Proxy

Detailed metrics for a single HAProxy load balancer

Detailed metrics for HAProxy: Demo

PGSQL Proxy shows detailed metrics for a single HAProxy load balancer instance serving PostgreSQL traffic.

pgsql-proxy

9.3.6 - PGSQL Pgbouncer

Metrics overview for a single Pgbouncer connection pooler

Metrics overview for Pgbouncer: Demo

PGSQL Pgbouncer shows connection pooling metrics for a single Pgbouncer instance.

pgsql-pgbouncer

9.3.7 - PGSQL Session

Session and active/idle time metrics for a single instance

Session and active/idle time metrics: Demo

PGSQL Session focuses on session statistics and active/idle time distribution for a single PostgreSQL instance.

pgsql-session

9.3.8 - PGSQL Xacts

Transaction, lock, TPS/QPS related metrics

Transaction, lock, TPS/QPS metrics: Demo

PGSQL Xacts focuses on transaction processing, lock activity, and TPS/QPS metrics for a single PostgreSQL instance.

pgsql-xacts

9.3.9 - PGSQL Exporter

Self-monitoring metrics for Postgres and Pgbouncer exporters

Self-monitoring metrics for exporters: Demo

PGSQL Exporter shows self-monitoring metrics for the Postgres exporter and Pgbouncer exporter components.

pgsql-exporter

9.4 - Database

PostgreSQL database-level monitoring dashboards

Use these dashboards to monitor PostgreSQL databases:

9.4.1 - PGSQL Database

Main dashboard for a single PGSQL database

Main dashboard for a single PGSQL database: Demo

PGSQL Database is the main dashboard for a single PostgreSQL database. It provides a full set of database-level metrics.

pgsql-database

9.4.2 - PGCAT Database

Database info retrieved directly from database catalog

Database info from database catalog: Demo

PGCAT Database reads database-level information directly from the PostgreSQL system catalog and shows it.

pgcat-database

9.4.3 - PGSQL Tables

Table/index access metrics within a single database

Table/index access metrics: Demo

PGSQL Tables shows table and index access metrics for all objects within a single PostgreSQL database.

pgsql-tables

9.4.4 - PGSQL Table

Detailed info for a single table (QPS/RT/index/sequence)

Detailed info for a single table: Demo

PGSQL Table shows detailed metrics for a single table. These metrics include QPS, response time, index usage, and sequence information.

pgsql-table

9.4.5 - PGCAT Table

Detailed table info from database catalog

Detailed table info from catalog: Demo

PGCAT Table shows detailed table information from the database catalog. This information includes statistics and bloat analysis.

pgcat-table

9.4.6 - PGSQL Query

Detailed info for a query type (QPS/RT)

Detailed info for a query type: Demo

PGSQL Query shows detailed metrics for a specific query type. These metrics include QPS and response time distribution.

pgsql-query

9.4.7 - PGCAT Query

Query details from database catalog

Query details from database catalog: Demo

PGCAT Query shows query details from the database catalog. These details include SQL text and execution statistics.

pgcat-query

9.4.8 - PGCAT Locks

Activity and lock wait info from database catalog

Activity and lock wait info: Demo

PGCAT Locks shows active sessions and lock wait information from the database catalog.

pgcat-locks

9.4.9 - PGCAT Schema

Schema info from database catalog

Schema info from database catalog: Demo

PGCAT Schema shows schema-level information from the database catalog. This information includes tables, indexes, and sequences.

pgcat-schema

10 - Metrics

Complete monitoring metrics reference for the Pigsty PGSQL module

The PGSQL module provides 638 available monitoring metrics.

Metric NameTypeLabelsDescription
ALERTSUnknowncategory, job, level, ins, severity, ip, alertname, alertstate, instance, clsN/A
ALERTS_FOR_STATEUnknowncategory, job, level, ins, severity, ip, alertname, instance, clsN/A
cls:pressure1Unknownjob, clsN/A
cls:pressure15Unknownjob, clsN/A
cls:pressure5Unknownjob, clsN/A
go_gc_duration_secondssummaryjob, ins, ip, instance, quantile, clsA summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_countUnknownjob, ins, ip, instance, clsN/A
go_gc_duration_seconds_sumUnknownjob, ins, ip, instance, clsN/A
go_goroutinesgaugejob, ins, ip, instance, clsNumber of goroutines that currently exist.
go_infogaugeversion, job, ins, ip, instance, clsInformation about the Go environment.
go_memstats_alloc_bytesgaugejob, ins, ip, instance, clsNumber of bytes allocated and still in use.
go_memstats_alloc_bytes_totalcounterjob, ins, ip, instance, clsTotal number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes used by the profiling bucket hash table.
go_memstats_frees_totalcounterjob, ins, ip, instance, clsTotal number of frees.
go_memstats_gc_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytesgaugejob, ins, ip, instance, clsNumber of heap bytes allocated and still in use.
go_memstats_heap_idle_bytesgaugejob, ins, ip, instance, clsNumber of heap bytes waiting to be used.
go_memstats_heap_inuse_bytesgaugejob, ins, ip, instance, clsNumber of heap bytes that are in use.
go_memstats_heap_objectsgaugejob, ins, ip, instance, clsNumber of allocated objects.
go_memstats_heap_released_bytesgaugejob, ins, ip, instance, clsNumber of heap bytes released to OS.
go_memstats_heap_sys_bytesgaugejob, ins, ip, instance, clsNumber of heap bytes obtained from system.
go_memstats_last_gc_time_secondsgaugejob, ins, ip, instance, clsNumber of seconds since 1970 of last garbage collection.
go_memstats_lookups_totalcounterjob, ins, ip, instance, clsTotal number of pointer lookups.
go_memstats_mallocs_totalcounterjob, ins, ip, instance, clsTotal number of mallocs.
go_memstats_mcache_inuse_bytesgaugejob, ins, ip, instance, clsNumber of bytes in use by mcache structures.
go_memstats_mcache_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytesgaugejob, ins, ip, instance, clsNumber of bytes in use by mspan structures.
go_memstats_mspan_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytesgaugejob, ins, ip, instance, clsNumber of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes used for other system allocations.
go_memstats_stack_inuse_bytesgaugejob, ins, ip, instance, clsNumber of bytes in use by the stack allocator.
go_memstats_stack_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes obtained from system for stack allocator.
go_memstats_sys_bytesgaugejob, ins, ip, instance, clsNumber of bytes obtained from system.
go_threadsgaugejob, ins, ip, instance, clsNumber of OS threads created.
ins:pressure1Unknownjob, ins, ip, clsN/A
ins:pressure15Unknownjob, ins, ip, clsN/A
ins:pressure5Unknownjob, ins, ip, clsN/A
patroni_cluster_unlockedgaugejob, ins, ip, instance, cls, scopeValue is 1 if the cluster is unlocked, 0 if locked.
patroni_dcs_last_seengaugejob, ins, ip, instance, cls, scopeEpoch timestamp when DCS was last contacted successfully by Patroni.
patroni_failsafe_mode_is_activegaugejob, ins, ip, instance, cls, scopeValue is 1 if failsafe mode is active, 0 if inactive.
patroni_is_pausedgaugejob, ins, ip, instance, cls, scopeValue is 1 if auto failover is disabled, 0 otherwise.
patroni_mastergaugejob, ins, ip, instance, cls, scopeValue is 1 if this node is the leader, 0 otherwise.
patroni_pending_restartgaugejob, ins, ip, instance, cls, scopeValue is 1 if the node needs a restart, 0 otherwise.
patroni_postgres_in_archive_recoverygaugejob, ins, ip, instance, cls, scopeValue is 1 if Postgres is replicating from archive, 0 otherwise.
patroni_postgres_runninggaugejob, ins, ip, instance, cls, scopeValue is 1 if Postgres is running, 0 otherwise.
patroni_postgres_server_versiongaugejob, ins, ip, instance, cls, scopeVersion of Postgres (if running), 0 otherwise.
patroni_postgres_streaminggaugejob, ins, ip, instance, cls, scopeValue is 1 if Postgres is streaming, 0 otherwise.
patroni_postgres_timelinecounterjob, ins, ip, instance, cls, scopePostgres timeline of this node (if running), 0 otherwise.
patroni_postmaster_start_timegaugejob, ins, ip, instance, cls, scopeEpoch seconds since Postgres started.
patroni_primarygaugejob, ins, ip, instance, cls, scopeValue is 1 if this node is the leader, 0 otherwise.
patroni_replicagaugejob, ins, ip, instance, cls, scopeValue is 1 if this node is a replica, 0 otherwise.
patroni_standby_leadergaugejob, ins, ip, instance, cls, scopeValue is 1 if this node is the standby_leader, 0 otherwise.
patroni_sync_standbygaugejob, ins, ip, instance, cls, scopeValue is 1 if this node is a sync standby replica, 0 otherwise.
patroni_upUnknownjob, ins, ip, instance, clsN/A
patroni_versiongaugejob, ins, ip, instance, cls, scopePatroni semver without periods.
patroni_xlog_locationcounterjob, ins, ip, instance, cls, scopeCurrent location of the Postgres transaction log, 0 if this node is not the leader.
patroni_xlog_pausedgaugejob, ins, ip, instance, cls, scopeValue is 1 if the Postgres xlog is paused, 0 otherwise.
patroni_xlog_received_locationcounterjob, ins, ip, instance, cls, scopeCurrent location of the received Postgres transaction log, 0 if this node is not a replica.
patroni_xlog_replayed_locationcounterjob, ins, ip, instance, cls, scopeCurrent location of the replayed Postgres transaction log, 0 if this node is not a replica.
patroni_xlog_replayed_timestampgaugejob, ins, ip, instance, cls, scopeCurrent timestamp of the replayed Postgres transaction log, 0 if null.
pg:cls:active_backendsUnknownjob, clsN/A
pg:cls:active_time_rate15mUnknownjob, clsN/A
pg:cls:active_time_rate1mUnknownjob, clsN/A
pg:cls:active_time_rate5mUnknownjob, clsN/A
pg:cls:ageUnknownjob, clsN/A
pg:cls:buf_alloc_rate1mUnknownjob, clsN/A
pg:cls:buf_clean_rate1mUnknownjob, clsN/A
pg:cls:buf_flush_backend_rate1mUnknownjob, clsN/A
pg:cls:buf_flush_checkpoint_rate1mUnknownjob, clsN/A
pg:cls:cpu_countUnknownjob, clsN/A
pg:cls:cpu_usageUnknownjob, clsN/A
pg:cls:cpu_usage_15mUnknownjob, clsN/A
pg:cls:cpu_usage_1mUnknownjob, clsN/A
pg:cls:cpu_usage_5mUnknownjob, clsN/A
pg:cls:db_sizeUnknownjob, clsN/A
pg:cls:file_sizeUnknownjob, clsN/A
pg:cls:ixact_backendsUnknownjob, clsN/A
pg:cls:ixact_time_rate1mUnknownjob, clsN/A
pg:cls:lag_bytesUnknownjob, clsN/A
pg:cls:lag_secondsUnknownjob, clsN/A
pg:cls:leaderUnknownjob, ins, ip, instance, clsN/A
pg:cls:load1Unknownjob, clsN/A
pg:cls:load15Unknownjob, clsN/A
pg:cls:load5Unknownjob, clsN/A
pg:cls:lock_countUnknownjob, clsN/A
pg:cls:locksUnknownjob, cls, modeN/A
pg:cls:log_sizeUnknownjob, clsN/A
pg:cls:lsn_rate1mUnknownjob, clsN/A
pg:cls:membersUnknownjob, ins, ip, clsN/A
pg:cls:num_backendsUnknownjob, clsN/A
pg:cls:partitionUnknownjob, clsN/A
pg:cls:receiverUnknownstate, slot_name, job, appname, ip, cls, sender_host, sender_portN/A
pg:cls:rlock_countUnknownjob, clsN/A
pg:cls:saturation1Unknownjob, clsN/A
pg:cls:saturation15Unknownjob, clsN/A
pg:cls:saturation5Unknownjob, clsN/A
pg:cls:senderUnknownpid, usename, address, job, ins, appname, ip, clsN/A
pg:cls:session_time_rate1mUnknownjob, clsN/A
pg:cls:sizeUnknownjob, clsN/A
pg:cls:slot_countUnknownjob, clsN/A
pg:cls:slot_retained_bytesUnknownjob, clsN/A
pg:cls:standby_countUnknownjob, clsN/A
pg:cls:sync_stateUnknownjob, clsN/A
pg:cls:timelineUnknownjob, clsN/A
pg:cls:tup_deleted_rate1mUnknownjob, clsN/A
pg:cls:tup_fetched_rate1mUnknownjob, clsN/A
pg:cls:tup_inserted_rate1mUnknownjob, clsN/A
pg:cls:tup_modified_rate1mUnknownjob, clsN/A
pg:cls:tup_returned_rate1mUnknownjob, clsN/A
pg:cls:wal_sizeUnknownjob, clsN/A
pg:cls:xact_commit_rate15mUnknownjob, clsN/A
pg:cls:xact_commit_rate1mUnknownjob, clsN/A
pg:cls:xact_commit_rate5mUnknownjob, clsN/A
pg:cls:xact_rollback_rate15mUnknownjob, clsN/A
pg:cls:xact_rollback_rate1mUnknownjob, clsN/A
pg:cls:xact_rollback_rate5mUnknownjob, clsN/A
pg:cls:xact_total_rate15mUnknownjob, clsN/A
pg:cls:xact_total_rate1mUnknownjob, clsN/A
pg:cls:xact_total_sigma15mUnknownjob, clsN/A
pg:cls:xlock_countUnknownjob, clsN/A
pg:db:active_backendsUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:active_time_rate15mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:active_time_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:active_time_rate5mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:ageUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:age_deriv1hUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:age_exhaustUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blk_io_time_seconds_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blk_read_time_seconds_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blk_write_time_seconds_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blks_access_1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blks_hit_1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blks_hit_ratio1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:blks_read_1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:conn_limitUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:conn_usageUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:db_sizeUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:ixact_backendsUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:ixact_time_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:lock_countUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:num_backendsUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:rlock_countUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:session_time_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:temp_bytes_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:temp_files_1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:tup_deleted_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:tup_fetched_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:tup_inserted_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:tup_modified_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:tup_returned_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:wlock_countUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_commit_rate15mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_commit_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_commit_rate5mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_rollback_rate15mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_rollback_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_rollback_rate5mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_total_rate15mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_total_rate1mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_total_rate5mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xact_total_sigma15mUnknowndatname, job, ins, ip, instance, clsN/A
pg:db:xlock_countUnknowndatname, job, ins, ip, instance, clsN/A
pg:env:active_backendsUnknownjobN/A
pg:env:active_time_rate15mUnknownjobN/A
pg:env:active_time_rate1mUnknownjobN/A
pg:env:active_time_rate5mUnknownjobN/A
pg:env:ageUnknownjobN/A
pg:env:cpu_countUnknownjobN/A
pg:env:cpu_usageUnknownjobN/A
pg:env:cpu_usage_15mUnknownjobN/A
pg:env:cpu_usage_1mUnknownjobN/A
pg:env:cpu_usage_5mUnknownjobN/A
pg:env:ixact_backendsUnknownjobN/A
pg:env:ixact_time_rate1mUnknownjobN/A
pg:env:lag_bytesUnknownjobN/A
pg:env:lag_secondsUnknownjobN/A
pg:env:lsn_rate1mUnknownjobN/A
pg:env:session_time_rate1mUnknownjobN/A
pg:env:tup_deleted_rate1mUnknownjobN/A
pg:env:tup_fetched_rate1mUnknownjobN/A
pg:env:tup_inserted_rate1mUnknownjobN/A
pg:env:tup_modified_rate1mUnknownjobN/A
pg:env:tup_returned_rate1mUnknownjobN/A
pg:env:xact_commit_rate15mUnknownjobN/A
pg:env:xact_commit_rate1mUnknownjobN/A
pg:env:xact_commit_rate5mUnknownjobN/A
pg:env:xact_rollback_rate15mUnknownjobN/A
pg:env:xact_rollback_rate1mUnknownjobN/A
pg:env:xact_rollback_rate5mUnknownjobN/A
pg:env:xact_total_rate15mUnknownjobN/A
pg:env:xact_total_rate1mUnknownjobN/A
pg:env:xact_total_sigma15mUnknownjobN/A
pg:ins:active_backendsUnknownjob, ins, ip, instance, clsN/A
pg:ins:active_time_rate15mUnknownjob, ins, ip, instance, clsN/A
pg:ins:active_time_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:active_time_rate5mUnknownjob, ins, ip, instance, clsN/A
pg:ins:ageUnknownjob, ins, ip, instance, clsN/A
pg:ins:blks_hit_ratio1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:buf_alloc_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:buf_clean_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:buf_flush_backend_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:buf_flush_checkpoint_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:ckpt_1hUnknownjob, ins, ip, instance, clsN/A
pg:ins:ckpt_req_1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:ckpt_timed_1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:conn_limitUnknownjob, ins, ip, instance, clsN/A
pg:ins:conn_usageUnknownjob, ins, ip, instance, clsN/A
pg:ins:cpu_countUnknownjob, ins, ip, instance, clsN/A
pg:ins:cpu_usageUnknownjob, ins, ip, instance, clsN/A
pg:ins:cpu_usage_15mUnknownjob, ins, ip, instance, clsN/A
pg:ins:cpu_usage_1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:cpu_usage_5mUnknownjob, ins, ip, instance, clsN/A
pg:ins:db_sizeUnknownjob, ins, ip, instance, clsN/A
pg:ins:file_sizeUnknownjob, ins, ip, instance, clsN/A
pg:ins:fs_sizeUnknownjob, ins, ip, instance, clsN/A
pg:ins:is_leaderUnknownjob, ins, ip, instance, clsN/A
pg:ins:ixact_backendsUnknownjob, ins, ip, instance, clsN/A
pg:ins:ixact_time_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:lag_bytesUnknownjob, ins, ip, instance, clsN/A
pg:ins:lag_secondsUnknownjob, ins, ip, instance, clsN/A
pg:ins:load1Unknownjob, ins, ip, instance, clsN/A
pg:ins:load15Unknownjob, ins, ip, instance, clsN/A
pg:ins:load5Unknownjob, ins, ip, instance, clsN/A
pg:ins:lock_countUnknownjob, ins, ip, instance, clsN/A
pg:ins:locksUnknownjob, ins, ip, mode, instance, clsN/A
pg:ins:log_sizeUnknownjob, ins, ip, instance, clsN/A
pg:ins:lsn_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:mem_sizeUnknownjob, ins, ip, instance, clsN/A
pg:ins:num_backendsUnknownjob, ins, ip, instance, clsN/A
pg:ins:rlock_countUnknownjob, ins, ip, instance, clsN/A
pg:ins:saturation1Unknownjob, ins, ip, clsN/A
pg:ins:saturation15Unknownjob, ins, ip, clsN/A
pg:ins:saturation5Unknownjob, ins, ip, clsN/A
pg:ins:session_time_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:slot_retained_bytesUnknownjob, ins, ip, instance, clsN/A
pg:ins:space_usageUnknownjob, ins, ip, instance, clsN/A
pg:ins:statusUnknownjob, ins, ip, instance, clsN/A
pg:ins:sync_stateUnknownjob, ins, instance, clsN/A
pg:ins:target_countUnknownjob, cls, insN/A
pg:ins:timelineUnknownjob, ins, ip, instance, clsN/A
pg:ins:tup_deleted_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:tup_fetched_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:tup_inserted_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:tup_modified_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:tup_returned_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:wal_sizeUnknownjob, ins, ip, instance, clsN/A
pg:ins:wlock_countUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_commit_rate15mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_commit_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_commit_rate5mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_rollback_rate15mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_rollback_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_rollback_rate5mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_total_rate15mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_total_rate1mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_total_rate5mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xact_total_sigma15mUnknownjob, ins, ip, instance, clsN/A
pg:ins:xlock_countUnknownjob, ins, ip, instance, clsN/A
pg:query:call_rate1mUnknowndatname, query, job, ins, ip, instance, clsN/A
pg:query:rt_1mUnknowndatname, query, job, ins, ip, instance, clsN/A
pg:table:scan_rate1mUnknowndatname, relname, job, ins, ip, instance, clsN/A
pg_activity_countgaugedatname, state, job, ins, ip, instance, clsCount of connection among (datname,state)
pg_activity_max_conn_durationgaugedatname, state, job, ins, ip, instance, clsMax backend session duration since state change among (datname, state)
pg_activity_max_durationgaugedatname, state, job, ins, ip, instance, clsMax duration since last state change among (datname, state)
pg_activity_max_tx_durationgaugedatname, state, job, ins, ip, instance, clsMax transaction duration since state change among (datname, state)
pg_archiver_failed_countcounterjob, ins, ip, instance, clsNumber of failed attempts for archiving WAL files
pg_archiver_finish_countcounterjob, ins, ip, instance, clsNumber of WAL files that have been successfully archived
pg_archiver_last_failed_timecounterjob, ins, ip, instance, clsTime of the last failed archival operation
pg_archiver_last_finish_timecounterjob, ins, ip, instance, clsTime of the last successful archive operation
pg_archiver_reset_timegaugejob, ins, ip, instance, clsTime at which archive statistics were last reset
pg_backend_countgaugetype, job, ins, ip, instance, clsDatabase backend process count by backend_type
pg_bgwriter_buffers_alloccounterjob, ins, ip, instance, clsNumber of buffers allocated
pg_bgwriter_buffers_backendcounterjob, ins, ip, instance, clsNumber of buffers written directly by a backend
pg_bgwriter_buffers_backend_fsynccounterjob, ins, ip, instance, clsNumber of times a backend had to execute its own fsync call
pg_bgwriter_buffers_checkpointcounterjob, ins, ip, instance, clsNumber of buffers written during checkpoints
pg_bgwriter_buffers_cleancounterjob, ins, ip, instance, clsNumber of buffers written by the background writer
pg_bgwriter_checkpoint_sync_timecounterjob, ins, ip, instance, clsTotal amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in seconds
pg_bgwriter_checkpoint_write_timecounterjob, ins, ip, instance, clsTotal amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in seconds
pg_bgwriter_checkpoints_reqcounterjob, ins, ip, instance, clsNumber of requested checkpoints that have been performed
pg_bgwriter_checkpoints_timedcounterjob, ins, ip, instance, clsNumber of scheduled checkpoints that have been performed
pg_bgwriter_maxwritten_cleancounterjob, ins, ip, instance, clsNumber of times the background writer stopped a cleaning scan because it had written too many buffers
pg_bgwriter_reset_timecounterjob, ins, ip, instance, clsTime at which bgwriter statistics were last reset
pg_boot_timegaugejob, ins, ip, instance, clsunix timestamp when postmaster boot
pg_checkpoint_checkpoint_lsncounterjob, ins, ip, instance, clsLatest checkpoint location
pg_checkpoint_elapsegaugejob, ins, ip, instance, clsSeconds elapsed since latest checkpoint in seconds
pg_checkpoint_full_page_writesgaugejob, ins, ip, instance, clsLatest checkpoint’s full_page_writes enabled
pg_checkpoint_newest_commit_ts_xidcounterjob, ins, ip, instance, clsLatest checkpoint’s newestCommitTsXid
pg_checkpoint_next_multi_offsetcounterjob, ins, ip, instance, clsLatest checkpoint’s NextMultiOffset
pg_checkpoint_next_multixact_idcounterjob, ins, ip, instance, clsLatest checkpoint’s NextMultiXactId
pg_checkpoint_next_oidcounterjob, ins, ip, instance, clsLatest checkpoint’s NextOID
pg_checkpoint_next_xidcounterjob, ins, ip, instance, clsLatest checkpoint’s NextXID xid
pg_checkpoint_next_xid_epochcounterjob, ins, ip, instance, clsLatest checkpoint’s NextXID epoch
pg_checkpoint_oldest_active_xidcounterjob, ins, ip, instance, clsLatest checkpoint’s oldestActiveXID
pg_checkpoint_oldest_commit_ts_xidcounterjob, ins, ip, instance, clsLatest checkpoint’s oldestCommitTsXid
pg_checkpoint_oldest_multi_dbidgaugejob, ins, ip, instance, clsLatest checkpoint’s oldestMulti’s DB OID
pg_checkpoint_oldest_multi_xidcounterjob, ins, ip, instance, clsLatest checkpoint’s oldestMultiXid
pg_checkpoint_oldest_xidcounterjob, ins, ip, instance, clsLatest checkpoint’s oldestXID
pg_checkpoint_oldest_xid_dbidgaugejob, ins, ip, instance, clsLatest checkpoint’s oldestXID’s DB OID
pg_checkpoint_prev_tlicounterjob, ins, ip, instance, clsLatest checkpoint’s PrevTimeLineID
pg_checkpoint_redo_lsncounterjob, ins, ip, instance, clsLatest checkpoint’s REDO location
pg_checkpoint_timecounterjob, ins, ip, instance, clsTime of latest checkpoint
pg_checkpoint_tlicounterjob, ins, ip, instance, clsLatest checkpoint’s TimeLineID
pg_conf_reload_timegaugejob, ins, ip, instance, clsseconds since last configuration reload
pg_db_active_timecounterdatname, job, ins, ip, instance, clsTime spent executing SQL statements in this database, in seconds
pg_db_agegaugedatname, job, ins, ip, instance, clsAge of database calculated from datfrozenxid
pg_db_allow_conngaugedatname, job, ins, ip, instance, clsIf false(0) then no one can connect to this database.
pg_db_blk_read_timecounterdatname, job, ins, ip, instance, clsTime spent reading data file blocks by backends in this database, in seconds
pg_db_blk_write_timecounterdatname, job, ins, ip, instance, clsTime spent writing data file blocks by backends in this database, in seconds
pg_db_blks_accesscounterdatname, job, ins, ip, instance, clsNumber of times disk blocks that accessed read+hit
pg_db_blks_hitcounterdatname, job, ins, ip, instance, clsNumber of times disk blocks were found already in the buffer cache
pg_db_blks_readcounterdatname, job, ins, ip, instance, clsNumber of disk blocks read in this database
pg_db_cks_fail_timegaugedatname, job, ins, ip, instance, clsTime at which the last data page checksum failure was detected in this database
pg_db_cks_failscounterdatname, job, ins, ip, instance, clsNumber of data page checksum failures detected in this database, -1 for not enabled
pg_db_confl_confl_bufferpincounterdatname, job, ins, ip, instance, clsNumber of queries in this database that have been canceled due to pinned buffers
pg_db_confl_confl_deadlockcounterdatname, job, ins, ip, instance, clsNumber of queries in this database that have been canceled due to deadlocks
pg_db_confl_confl_lockcounterdatname, job, ins, ip, instance, clsNumber of queries in this database that have been canceled due to lock timeouts
pg_db_confl_confl_snapshotcounterdatname, job, ins, ip, instance, clsNumber of queries in this database that have been canceled due to old snapshots
pg_db_confl_confl_tablespacecounterdatname, job, ins, ip, instance, clsNumber of queries in this database that have been canceled due to dropped tablespaces
pg_db_conflictscounterdatname, job, ins, ip, instance, clsNumber of queries canceled due to conflicts with recovery in this database
pg_db_conn_limitgaugedatname, job, ins, ip, instance, clsSets maximum number of concurrent connections that can be made to this database. -1 means no limit.
pg_db_datidgaugedatname, job, ins, ip, instance, clsOID of the database
pg_db_deadlockscounterdatname, job, ins, ip, instance, clsNumber of deadlocks detected in this database
pg_db_frozen_xidgaugedatname, job, ins, ip, instance, clsAll transaction IDs before this one have been frozened
pg_db_is_templategaugedatname, job, ins, ip, instance, clsIf true(1), then this database can be cloned by any user with CREATEDB privileges
pg_db_ixact_timecounterdatname, job, ins, ip, instance, clsTime spent idling while in a transaction in this database, in seconds
pg_db_numbackendsgaugedatname, job, ins, ip, instance, clsNumber of backends currently connected to this database
pg_db_reset_timecounterdatname, job, ins, ip, instance, clsTime at which database statistics were last reset
pg_db_session_timecounterdatname, job, ins, ip, instance, clsTime spent by database sessions in this database, in seconds
pg_db_sessionscounterdatname, job, ins, ip, instance, clsTotal number of sessions established to this database
pg_db_sessions_abandonedcounterdatname, job, ins, ip, instance, clsNumber of database sessions to this database that were terminated because connection to the client was lost
pg_db_sessions_fatalcounterdatname, job, ins, ip, instance, clsNumber of database sessions to this database that were terminated by fatal errors
pg_db_sessions_killedcounterdatname, job, ins, ip, instance, clsNumber of database sessions to this database that were terminated by operator intervention
pg_db_temp_bytescounterdatname, job, ins, ip, instance, clsTotal amount of data written to temporary files by queries in this database.
pg_db_temp_filescounterdatname, job, ins, ip, instance, clsNumber of temporary files created by queries in this database
pg_db_tup_deletedcounterdatname, job, ins, ip, instance, clsNumber of rows deleted by queries in this database
pg_db_tup_fetchedcounterdatname, job, ins, ip, instance, clsNumber of rows fetched by queries in this database
pg_db_tup_insertedcounterdatname, job, ins, ip, instance, clsNumber of rows inserted by queries in this database
pg_db_tup_modifiedcounterdatname, job, ins, ip, instance, clsNumber of rows modified by queries in this database
pg_db_tup_returnedcounterdatname, job, ins, ip, instance, clsNumber of rows returned by queries in this database
pg_db_tup_updatedcounterdatname, job, ins, ip, instance, clsNumber of rows updated by queries in this database
pg_db_xact_commitcounterdatname, job, ins, ip, instance, clsNumber of transactions in this database that have been committed
pg_db_xact_rollbackcounterdatname, job, ins, ip, instance, clsNumber of transactions in this database that have been rolled back
pg_db_xact_totalcounterdatname, job, ins, ip, instance, clsNumber of transactions in this database
pg_downstream_countgaugestate, job, ins, ip, instance, clsCount of corresponding state
pg_exporter_agent_upUnknownjob, ins, ip, instance, clsN/A
pg_exporter_last_scrape_timegaugejob, ins, ip, instance, clsseconds exporter spending on scrapping
pg_exporter_query_cache_ttlgaugedatname, query, job, ins, ip, instance, clstimes to live of query cache
pg_exporter_query_scrape_durationgaugedatname, query, job, ins, ip, instance, clsseconds query spending on scrapping
pg_exporter_query_scrape_error_countgaugedatname, query, job, ins, ip, instance, clstimes the query failed
pg_exporter_query_scrape_hit_countgaugedatname, query, job, ins, ip, instance, clsnumbers been scrapped from this query
pg_exporter_query_scrape_metric_countgaugedatname, query, job, ins, ip, instance, clsnumbers of metrics been scrapped from this query
pg_exporter_query_scrape_total_countgaugedatname, query, job, ins, ip, instance, clstimes exporter server was scraped for metrics
pg_exporter_scrape_durationgaugejob, ins, ip, instance, clsseconds exporter spending on scrapping
pg_exporter_scrape_error_countcounterjob, ins, ip, instance, clstimes exporter was scraped for metrics and failed
pg_exporter_scrape_total_countcounterjob, ins, ip, instance, clstimes exporter was scraped for metrics
pg_exporter_server_scrape_durationgaugedatname, job, ins, ip, instance, clsseconds exporter server spending on scrapping
pg_exporter_server_scrape_error_countUnknowndatname, job, ins, ip, instance, clsN/A
pg_exporter_server_scrape_total_countgaugedatname, job, ins, ip, instance, clstimes exporter server was scraped for metrics
pg_exporter_server_scrape_total_secondsgaugedatname, job, ins, ip, instance, clsseconds exporter server spending on scrapping
pg_exporter_upgaugejob, ins, ip, instance, clsalways be 1 if your could retrieve metrics
pg_exporter_uptimegaugejob, ins, ip, instance, clsseconds since exporter primary server inited
pg_flush_lsncounterjob, ins, ip, instance, clsprimary only, location of current wal syncing
pg_func_callscounterdatname, funcname, job, ins, ip, instance, clsNumber of times this function has been called
pg_func_self_timecounterdatname, funcname, job, ins, ip, instance, clsTotal time spent in this function itself, not including other functions called by it, in ms
pg_func_total_timecounterdatname, funcname, job, ins, ip, instance, clsTotal time spent in this function and all other functions called by it, in ms
pg_in_recoverygaugejob, ins, ip, instance, clsserver is in recovery mode? 1 for yes 0 for no
pg_index_idx_blks_hitcounterdatname, relname, job, ins, relid, ip, instance, cls, idxnameNumber of buffer hits in this index
pg_index_idx_blks_readcounterdatname, relname, job, ins, relid, ip, instance, cls, idxnameNumber of disk blocks read from this index
pg_index_idx_scancounterdatname, relname, job, ins, relid, ip, instance, cls, idxnameNumber of index scans initiated on this index
pg_index_idx_tup_fetchcounterdatname, relname, job, ins, relid, ip, instance, cls, idxnameNumber of live table rows fetched by simple index scans using this index
pg_index_idx_tup_readcounterdatname, relname, job, ins, relid, ip, instance, cls, idxnameNumber of index entries returned by scans on this index
pg_index_relpagesgaugedatname, relname, job, ins, relid, ip, instance, cls, idxnameSize of the on-disk representation of this index in pages
pg_index_reltuplesgaugedatname, relname, job, ins, relid, ip, instance, cls, idxnameEstimate relation tuples
pg_insert_lsncounterjob, ins, ip, instance, clsprimary only, location of current wal inserting
pg_io_evictionscountertype, job, ins, object, ip, context, instance, clsNumber of times a block has been written out from a shared or local buffer
pg_io_extend_timecountertype, job, ins, object, ip, context, instance, clsTime spent in extend operations in seconds
pg_io_extendscountertype, job, ins, object, ip, context, instance, clsNumber of relation extend operations, each of the size specified in op_bytes.
pg_io_fsync_timecountertype, job, ins, object, ip, context, instance, clsTime spent in fsync operations in seconds
pg_io_fsyncscountertype, job, ins, object, ip, context, instance, clsNumber of fsync calls. These are only tracked in context normal
pg_io_hitscountertype, job, ins, object, ip, context, instance, clsThe number of times a desired block was found in a shared buffer.
pg_io_op_bytesgaugetype, job, ins, object, ip, context, instance, clsThe number of bytes per unit of I/O read, written, or extended. 8192 by default
pg_io_read_timecountertype, job, ins, object, ip, context, instance, clsTime spent in read operations in seconds
pg_io_readscountertype, job, ins, object, ip, context, instance, clsNumber of read operations, each of the size specified in op_bytes.
pg_io_reset_timegaugetype, job, ins, object, ip, context, instance, clsTimestamp at which these statistics were last reset
pg_io_reusescountertype, job, ins, object, ip, context, instance, clsThe number of times an existing buffer in reused
pg_io_write_timecountertype, job, ins, object, ip, context, instance, clsTime spent in write operations in seconds
pg_io_writeback_timecountertype, job, ins, object, ip, context, instance, clsTime spent in writeback operations in seconds
pg_io_writebackscountertype, job, ins, object, ip, context, instance, clsNumber of units of size op_bytes which the process requested the kernel write out to permanent storage.
pg_io_writescountertype, job, ins, object, ip, context, instance, clsNumber of write operations, each of the size specified in op_bytes.
pg_is_in_recoverygaugejob, ins, ip, instance, cls1 if in recovery mode
pg_is_wal_replay_pausedgaugejob, ins, ip, instance, cls1 if wal play paused
pg_laggaugejob, ins, ip, instance, clsreplica only, replication lag in seconds
pg_last_replay_timegaugejob, ins, ip, instance, clstime when last transaction been replayed
pg_lock_countgaugedatname, job, ins, ip, mode, instance, clsNumber of locks of corresponding mode and database
pg_lsncounterjob, ins, ip, instance, clslog sequence number, current write location
pg_meta_infogaugecls, extensions, version, job, ins, primary_conninfo, conf_path, hba_path, ip, cluster_id, instance, listen_port, wal_level, ver_num, cluster_name, data_dirconstant 1
pg_query_callscounterdatname, query, job, ins, ip, instance, clsNumber of times the statement was executed
pg_query_exec_timecounterdatname, query, job, ins, ip, instance, clsTotal time spent executing the statement, in seconds
pg_query_io_timecounterdatname, query, job, ins, ip, instance, clsTotal time the statement spent reading and writing blocks, in seconds
pg_query_rowscounterdatname, query, job, ins, ip, instance, clsTotal number of rows retrieved or affected by the statement
pg_query_sblk_dirtiedcounterdatname, query, job, ins, ip, instance, clsTotal number of shared blocks dirtied by the statement
pg_query_sblk_hitcounterdatname, query, job, ins, ip, instance, clsTotal number of shared block cache hits by the statement
pg_query_sblk_readcounterdatname, query, job, ins, ip, instance, clsTotal number of shared blocks read by the statement
pg_query_sblk_writtencounterdatname, query, job, ins, ip, instance, clsTotal number of shared blocks written by the statement
pg_query_wal_bytescounterdatname, query, job, ins, ip, instance, clsTotal amount of WAL bytes generated by the statement
pg_receive_lsncounterjob, ins, ip, instance, clsreplica only, location of wal synced to disk
pg_recovery_backup_end_lsncounterjob, ins, ip, instance, clsBackup end location
pg_recovery_backup_start_lsncounterjob, ins, ip, instance, clsBackup start location
pg_recovery_min_lsncounterjob, ins, ip, instance, clsMinimum recovery ending location
pg_recovery_min_timelinecounterjob, ins, ip, instance, clsMin recovery ending loc’s timeline
pg_recovery_prefetch_block_distancegaugejob, ins, ip, instance, clsHow many blocks ahead the prefetcher is looking
pg_recovery_prefetch_hitcounterjob, ins, ip, instance, clsNumber of blocks not prefetched because they were already in the buffer pool
pg_recovery_prefetch_io_depthgaugejob, ins, ip, instance, clsHow many prefetches have been initiated but are not yet known to have completed
pg_recovery_prefetch_prefetchcounterjob, ins, ip, instance, clsNumber of blocks prefetched because they were not in the buffer pool
pg_recovery_prefetch_reset_timecounterjob, ins, ip, instance, clsTime at which these recovery prefetch statistics were last reset
pg_recovery_prefetch_skip_fpwgaugejob, ins, ip, instance, clsNumber of blocks not prefetched because a full page image was included in the WAL
pg_recovery_prefetch_skip_initcounterjob, ins, ip, instance, clsNumber of blocks not prefetched because they would be zero-initialized
pg_recovery_prefetch_skip_newcounterjob, ins, ip, instance, clsNumber of blocks not prefetched because they didn’t exist yet
pg_recovery_prefetch_skip_repcounterjob, ins, ip, instance, clsNumber of blocks not prefetched because they were already recently prefetched
pg_recovery_prefetch_wal_distancegaugejob, ins, ip, instance, clsHow many bytes ahead the prefetcher is looking
pg_recovery_require_recordgaugejob, ins, ip, instance, clsEnd-of-backup record required
pg_recv_flush_lsncounterstate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portLast write-ahead log location already received and flushed to disk
pg_recv_flush_tlicounterstate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portTimeline number of last write-ahead log location received and flushed to disk
pg_recv_init_lsncounterstate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portFirst write-ahead log location used when WAL receiver is started
pg_recv_init_tlicounterstate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portFirst timeline number used when WAL receiver is started
pg_recv_msg_recv_timegaugestate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portReceipt time of last message received from origin WAL sender
pg_recv_msg_send_timegaugestate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portSend time of last message received from origin WAL sender
pg_recv_pidgaugestate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portProcess ID of the WAL receiver process
pg_recv_reported_lsncounterstate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portLast write-ahead log location reported to origin WAL sender
pg_recv_reported_timegaugestate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portTime of last write-ahead log location reported to origin WAL sender
pg_recv_timegaugestate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portTime of current snapshot
pg_recv_write_lsncounterstate, slot_name, job, ins, ip, instance, cls, sender_host, sender_portLast write-ahead log location already received and written to disk, but not flushed.
pg_relkind_countgaugedatname, job, ins, ip, instance, cls, relkindNumber of relations of corresponding relkind
pg_repl_backend_xmincounterpid, usename, address, job, ins, appname, ip, instance, clsThis standby’s xmin horizon reported by hot_standby_feedback.
pg_repl_client_portgaugepid, usename, address, job, ins, appname, ip, instance, clsTCP port number that the client is using for communication with this WAL sender, or -1 if a Unix socket is used
pg_repl_flush_diffgaugepid, usename, address, job, ins, appname, ip, instance, clsLast log position flushed to disk by this standby server diff with current lsn
pg_repl_flush_laggaugepid, usename, address, job, ins, appname, ip, instance, clsTime elapsed between flushing recent WAL locally and receiving notification that this standby server has written and flushed it
pg_repl_flush_lsncounterpid, usename, address, job, ins, appname, ip, instance, clsLast write-ahead log location flushed to disk by this standby server
pg_repl_launch_timecounterpid, usename, address, job, ins, appname, ip, instance, clsTime when this process was started, i.e., when the client connected to this WAL sender
pg_repl_lsncounterpid, usename, address, job, ins, appname, ip, instance, clsCurrent log position on this server
pg_repl_replay_diffgaugepid, usename, address, job, ins, appname, ip, instance, clsLast log position replayed into the database on this standby server diff with current lsn
pg_repl_replay_laggaugepid, usename, address, job, ins, appname, ip, instance, clsTime elapsed between flushing recent WAL locally and receiving notification that this standby server has written, flushed and applied it
pg_repl_replay_lsncounterpid, usename, address, job, ins, appname, ip, instance, clsLast write-ahead log location replayed into the database on this standby server
pg_repl_reply_timegaugepid, usename, address, job, ins, appname, ip, instance, clsSend time of last reply message received from standby server
pg_repl_sent_diffgaugepid, usename, address, job, ins, appname, ip, instance, clsLast log position sent to this standby server diff with current lsn
pg_repl_sent_lsncounterpid, usename, address, job, ins, appname, ip, instance, clsLast write-ahead log location sent on this connection
pg_repl_stategaugepid, usename, address, job, ins, appname, ip, instance, clsCurrent WAL sender encoded state 0-4 for streaming startup catchup backup stopping
pg_repl_sync_prioritygaugepid, usename, address, job, ins, appname, ip, instance, clsPriority of this standby server for being chosen as the synchronous standby
pg_repl_sync_stategaugepid, usename, address, job, ins, appname, ip, instance, clsEncoded synchronous state of this standby server, 0-3 for async potential sync quorum
pg_repl_timecounterpid, usename, address, job, ins, appname, ip, instance, clsCurrent timestamp in unix epoch
pg_repl_write_diffgaugepid, usename, address, job, ins, appname, ip, instance, clsLast log position written to disk by this standby server diff with current lsn
pg_repl_write_laggaugepid, usename, address, job, ins, appname, ip, instance, clsTime elapsed between flushing recent WAL locally and receiving notification that this standby server has written it
pg_repl_write_lsncounterpid, usename, address, job, ins, appname, ip, instance, clsLast write-ahead log location written to disk by this standby server
pg_replay_lsncounterjob, ins, ip, instance, clsreplica only, location of wal applied
pg_seq_blks_hitcounterdatname, job, ins, ip, instance, cls, seqnameNumber of buffer hits in this sequence
pg_seq_blks_readcounterdatname, job, ins, ip, instance, cls, seqnameNumber of disk blocks read from this sequence
pg_seq_last_valuecounterdatname, job, ins, ip, instance, cls, seqnameThe last sequence value written to disk
pg_setting_block_sizegaugejob, ins, ip, instance, clspg page block size, 8192 by default
pg_setting_data_checksumsgaugejob, ins, ip, instance, clswhether data checksum is enabled, 1 enabled 0 disabled
pg_setting_max_connectionsgaugejob, ins, ip, instance, clsnumber of concurrent connections to the database server
pg_setting_max_locks_per_transactiongaugejob, ins, ip, instance, clsno more than this many distinct objects can be locked at any one time
pg_setting_max_prepared_transactionsgaugejob, ins, ip, instance, clsmaximum number of transactions that can be in the prepared state simultaneously
pg_setting_max_replication_slotsgaugejob, ins, ip, instance, clsmaximum number of replication slots
pg_setting_max_wal_sendersgaugejob, ins, ip, instance, clsmaximum number of concurrent connections from standby servers
pg_setting_max_worker_processesgaugejob, ins, ip, instance, clsmaximum number of background processes that the system can support
pg_setting_wal_log_hintsgaugejob, ins, ip, instance, clswhether wal_log_hints is enabled, 1 enabled 0 disabled
pg_size_bytesgaugedatname, job, ins, ip, instance, clsFile size in bytes
pg_slot_activegaugeslot_name, job, ins, ip, instance, clsTrue(1) if this slot is currently actively being used
pg_slot_catalog_xmincounterslot_name, job, ins, ip, instance, clsThe oldest transaction affecting the system catalogs that this slot needs the database to retain.
pg_slot_confirm_lsncounterslot_name, job, ins, ip, instance, clsThe address (LSN) up to which the logical slot’s consumer has confirmed receiving data.
pg_slot_reset_timecounterslot_name, job, ins, ip, instance, clsWhen statistics were last reset
pg_slot_restart_lsncounterslot_name, job, ins, ip, instance, clsThe address (LSN) of oldest WAL which still might be required by the consumer of this slot
pg_slot_retained_bytesgaugeslot_name, job, ins, ip, instance, clsSize of bytes that retained for this slot
pg_slot_safe_wal_sizegaugeslot_name, job, ins, ip, instance, clsbytes that can be written to WAL which will not make slot into lost
pg_slot_spill_bytescounterslot_name, job, ins, ip, instance, clsBytes that spilled to disk due to logical decode mem exceeding
pg_slot_spill_countcounterslot_name, job, ins, ip, instance, clsXacts that spilled to disk due to logical decode mem exceeding (a xact can be spilled multiple times)
pg_slot_spill_txnscounterslot_name, job, ins, ip, instance, clsXacts that spilled to disk due to logical decode mem exceeding (subtrans included)
pg_slot_stream_bytescounterslot_name, job, ins, ip, instance, clsBytes that streamed to decoding output plugin after mem exceed
pg_slot_stream_countcounterslot_name, job, ins, ip, instance, clsXacts that streamed to decoding output plugin after mem exceed (a xact can be streamed multiple times)
pg_slot_stream_txnscounterslot_name, job, ins, ip, instance, clsXacts that streamed to decoding output plugin after mem exceed
pg_slot_temporarygaugeslot_name, job, ins, ip, instance, clsTrue(1) if this is a temporary replication slot.
pg_slot_total_bytescounterslot_name, job, ins, ip, instance, clsNumber of decoded bytes sent to the decoding output plugin for this slot
pg_slot_total_txnscounterslot_name, job, ins, ip, instance, clsNumber of decoded xacts sent to the decoding output plugin for this slot
pg_slot_wal_statusgaugeslot_name, job, ins, ip, instance, clsWAL reserve status 0-3 means reserved,extended,unreserved,lost, -1 means other
pg_slot_xmincounterslot_name, job, ins, ip, instance, clsThe oldest transaction that this slot needs the database to retain.
pg_slru_blks_existscounterjob, ins, ip, instance, clsNumber of blocks checked for existence for this SLRU
pg_slru_blks_hitcounterjob, ins, ip, instance, clsNumber of times disk blocks were found already in the SLRU, so that a read was not necessary
pg_slru_blks_readcounterjob, ins, ip, instance, clsNumber of disk blocks read for this SLRU
pg_slru_blks_writtencounterjob, ins, ip, instance, clsNumber of disk blocks written for this SLRU
pg_slru_blks_zeroedcounterjob, ins, ip, instance, clsNumber of blocks zeroed during initializations
pg_slru_flushescounterjob, ins, ip, instance, clsNumber of flushes of dirty data for this SLRU
pg_slru_reset_timecounterjob, ins, ip, instance, clsTime at which these statistics were last reset
pg_slru_truncatescounterjob, ins, ip, instance, clsNumber of truncates for this SLRU
pg_ssl_disabledgaugejob, ins, ip, instance, clsNumber of client connection that does not use ssl
pg_ssl_enabledgaugejob, ins, ip, instance, clsNumber of client connection that use ssl
pg_sync_standby_enabledgaugejob, ins, ip, names, instance, clsSynchronous commit enabled, 1 if enabled, 0 if disabled
pg_table_agegaugedatname, relname, job, ins, ip, instance, clsAge of this table in vacuum cycles
pg_table_analyze_countcounterdatname, relname, job, ins, ip, instance, clsNumber of times this table has been manually analyzed
pg_table_autoanalyze_countcounterdatname, relname, job, ins, ip, instance, clsNumber of times this table has been analyzed by the autovacuum daemon
pg_table_autovacuum_countcounterdatname, relname, job, ins, ip, instance, clsNumber of times this table has been vacuumed by the autovacuum daemon
pg_table_frozenxidcounterdatname, relname, job, ins, ip, instance, clsAll txid before this have been frozen on this table
pg_table_heap_blks_hitcounterdatname, relname, job, ins, ip, instance, clsNumber of buffer hits in this table
pg_table_heap_blks_readcounterdatname, relname, job, ins, ip, instance, clsNumber of disk blocks read from this table
pg_table_idx_blks_hitcounterdatname, relname, job, ins, ip, instance, clsNumber of buffer hits in all indexes on this table
pg_table_idx_blks_readcounterdatname, relname, job, ins, ip, instance, clsNumber of disk blocks read from all indexes on this table
pg_table_idx_scancounterdatname, relname, job, ins, ip, instance, clsNumber of index scans initiated on this table
pg_table_idx_tup_fetchcounterdatname, relname, job, ins, ip, instance, clsNumber of live rows fetched by index scans
pg_table_kindgaugedatname, relname, job, ins, ip, instance, clsRelation kind r/table/114
pg_table_n_dead_tupgaugedatname, relname, job, ins, ip, instance, clsEstimated number of dead rows
pg_table_n_ins_since_vacuumgaugedatname, relname, job, ins, ip, instance, clsEstimated number of rows inserted since this table was last vacuumed
pg_table_n_live_tupgaugedatname, relname, job, ins, ip, instance, clsEstimated number of live rows
pg_table_n_mod_since_analyzegaugedatname, relname, job, ins, ip, instance, clsEstimated number of rows modified since this table was last analyzed
pg_table_n_tup_delcounterdatname, relname, job, ins, ip, instance, clsNumber of rows deleted
pg_table_n_tup_hot_updcounterdatname, relname, job, ins, ip, instance, clsNumber of rows HOT updated (i.e with no separate index update required)
pg_table_n_tup_inscounterdatname, relname, job, ins, ip, instance, clsNumber of rows inserted
pg_table_n_tup_modcounterdatname, relname, job, ins, ip, instance, clsNumber of rows modified (insert + update + delete)
pg_table_n_tup_newpage_updcounterdatname, relname, job, ins, ip, instance, clsNumber of rows updated where the successor version goes onto a new heap page
pg_table_n_tup_updcounterdatname, relname, job, ins, ip, instance, clsNumber of rows updated (includes HOT updated rows)
pg_table_ncolsgaugedatname, relname, job, ins, ip, instance, clsNumber of columns in the table
pg_table_pagesgaugedatname, relname, job, ins, ip, instance, clsSize of the on-disk representation of this table in pages
pg_table_relidgaugedatname, relname, job, ins, ip, instance, clsRelation oid of this table
pg_table_seq_scancounterdatname, relname, job, ins, ip, instance, clsNumber of sequential scans initiated on this table
pg_table_seq_tup_readcounterdatname, relname, job, ins, ip, instance, clsNumber of live rows fetched by sequential scans
pg_table_size_bytesgaugedatname, relname, job, ins, ip, instance, clsTotal bytes of this table (including toast, index, toast index)
pg_table_size_indexsizegaugedatname, relname, job, ins, ip, instance, clsBytes of all related indexes of this table
pg_table_size_relsizegaugedatname, relname, job, ins, ip, instance, clsBytes of this table itself (main, vm, fsm)
pg_table_size_toastsizegaugedatname, relname, job, ins, ip, instance, clsBytes of toast tables of this table
pg_table_tbl_scancounterdatname, relname, job, ins, ip, instance, clsNumber of scans initiated on this table
pg_table_tup_readcounterdatname, relname, job, ins, ip, instance, clsNumber of live rows fetched by scans
pg_table_tuplescounterdatname, relname, job, ins, ip, instance, clsAll txid before this have been frozen on this table
pg_table_vacuum_countcounterdatname, relname, job, ins, ip, instance, clsNumber of times this table has been manually vacuumed (not counting VACUUM FULL)
pg_timestampgaugejob, ins, ip, instance, clsdatabase current timestamp
pg_upgaugejob, ins, ip, instance, clslast scrape was able to connect to the server: 1 for yes, 0 for no
pg_uptimegaugejob, ins, ip, instance, clsseconds since postmaster start
pg_versiongaugejob, ins, ip, instance, clsserver version number
pg_wait_countgaugedatname, job, ins, event, ip, instance, clsCount of WaitEvent on target database
pg_wal_buffers_fullcounterjob, ins, ip, instance, clsNumber of times WAL data was written to disk because WAL buffers became full
pg_wal_bytescounterjob, ins, ip, instance, clsTotal amount of WAL generated in bytes
pg_wal_fpicounterjob, ins, ip, instance, clsTotal number of WAL full page images generated
pg_wal_recordscounterjob, ins, ip, instance, clsTotal number of WAL records generated
pg_wal_reset_timecounterjob, ins, ip, instance, clsWhen statistics were last reset
pg_wal_synccounterjob, ins, ip, instance, clsNumber of times WAL files were synced to disk via issue_xlog_fsync request
pg_wal_sync_timecounterjob, ins, ip, instance, clsTotal amount of time spent syncing WAL files to disk via issue_xlog_fsync request, in seconds
pg_wal_writecounterjob, ins, ip, instance, clsNumber of times WAL buffers were written out to disk via XLogWrite request.
pg_wal_write_timecounterjob, ins, ip, instance, clsTotal amount of time spent writing WAL buffers to disk via XLogWrite request in seconds
pg_write_lsncounterjob, ins, ip, instance, clsprimary only, location of current wal writing
pg_xact_xmaxcounterjob, ins, ip, instance, clsFirst as-yet-unassigned txid. txid >= this are invisible.
pg_xact_xmincounterjob, ins, ip, instance, clsEarliest txid that is still active
pg_xact_xnumgaugejob, ins, ip, instance, clsCurrent active transaction count
pgbouncer:cls:load1Unknownjob, clsN/A
pgbouncer:cls:load15Unknownjob, clsN/A
pgbouncer:cls:load5Unknownjob, clsN/A
pgbouncer:db:conn_usageUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:conn_usage_reserveUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:pool_current_connUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:pool_disabledUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:pool_max_connUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:pool_pausedUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:pool_reserve_sizeUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:db:pool_sizeUnknowndatname, job, ins, ip, instance, host, cls, real_datname, portN/A
pgbouncer:ins:free_clientsUnknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:free_serversUnknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:load1Unknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:load15Unknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:load5Unknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:login_clientsUnknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:pool_databasesUnknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:pool_usersUnknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:poolsUnknownjob, ins, ip, instance, clsN/A
pgbouncer:ins:used_clientsUnknownjob, ins, ip, instance, clsN/A
pgbouncer_database_current_connectionsgaugedatname, job, ins, ip, instance, host, cls, real_datname, portCurrent number of connections for this database
pgbouncer_database_disabledgaugedatname, job, ins, ip, instance, host, cls, real_datname, portTrue(1) if this database is currently disabled, else 0
pgbouncer_database_max_connectionsgaugedatname, job, ins, ip, instance, host, cls, real_datname, portMaximum number of allowed connections for this database
pgbouncer_database_min_pool_sizegaugedatname, job, ins, ip, instance, host, cls, real_datname, portMinimum number of server connections
pgbouncer_database_pausedgaugedatname, job, ins, ip, instance, host, cls, real_datname, portTrue(1) if this database is currently paused, else 0
pgbouncer_database_pool_sizegaugedatname, job, ins, ip, instance, host, cls, real_datname, portMaximum number of server connections
pgbouncer_database_reserve_poolgaugedatname, job, ins, ip, instance, host, cls, real_datname, portMaximum number of additional connections for this database
pgbouncer_exporter_agent_upUnknownjob, ins, ip, instance, clsN/A
pgbouncer_exporter_last_scrape_timegaugejob, ins, ip, instance, clsseconds exporter spending on scrapping
pgbouncer_exporter_query_cache_ttlgaugedatname, query, job, ins, ip, instance, clstimes to live of query cache
pgbouncer_exporter_query_scrape_durationgaugedatname, query, job, ins, ip, instance, clsseconds query spending on scrapping
pgbouncer_exporter_query_scrape_error_countgaugedatname, query, job, ins, ip, instance, clstimes the query failed
pgbouncer_exporter_query_scrape_hit_countgaugedatname, query, job, ins, ip, instance, clsnumbers been scrapped from this query
pgbouncer_exporter_query_scrape_metric_countgaugedatname, query, job, ins, ip, instance, clsnumbers of metrics been scrapped from this query
pgbouncer_exporter_query_scrape_total_countgaugedatname, query, job, ins, ip, instance, clstimes exporter server was scraped for metrics
pgbouncer_exporter_scrape_durationgaugejob, ins, ip, instance, clsseconds exporter spending on scrapping
pgbouncer_exporter_scrape_error_countcounterjob, ins, ip, instance, clstimes exporter was scraped for metrics and failed
pgbouncer_exporter_scrape_total_countcounterjob, ins, ip, instance, clstimes exporter was scraped for metrics
pgbouncer_exporter_server_scrape_durationgaugedatname, job, ins, ip, instance, clsseconds exporter server spending on scrapping
pgbouncer_exporter_server_scrape_total_countgaugedatname, job, ins, ip, instance, clstimes exporter server was scraped for metrics
pgbouncer_exporter_server_scrape_total_secondsgaugedatname, job, ins, ip, instance, clsseconds exporter server spending on scrapping
pgbouncer_exporter_upgaugejob, ins, ip, instance, clsalways be 1 if your could retrieve metrics
pgbouncer_exporter_uptimegaugejob, ins, ip, instance, clsseconds since exporter primary server inited
pgbouncer_in_recoverygaugejob, ins, ip, instance, clsserver is in recovery mode? 1 for yes 0 for no
pgbouncer_list_itemsgaugejob, ins, ip, instance, list, clsNumber of corresponding pgbouncer object
pgbouncer_pool_active_cancel_clientsgaugedatname, job, ins, ip, instance, user, cls, pool_modeClient connections that have forwarded query cancellations to the server and are waiting for the server response.
pgbouncer_pool_active_cancel_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modeServer connections that are currently forwarding a cancel request
pgbouncer_pool_active_clientsgaugedatname, job, ins, ip, instance, user, cls, pool_modeClient connections that are linked to server connection and can process queries
pgbouncer_pool_active_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modeServer connections that are linked to a client
pgbouncer_pool_cancel_clientsgaugedatname, job, ins, ip, instance, user, cls, pool_modeClient connections that have not forwarded query cancellations to the server yet.
pgbouncer_pool_cancel_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modecancel requests have completed that were sent to cancel a query on this server
pgbouncer_pool_idle_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modeServer connections that are unused and immediately usable for client queries
pgbouncer_pool_login_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modeServer connections currently in the process of logging in
pgbouncer_pool_maxwaitgaugedatname, job, ins, ip, instance, user, cls, pool_modeHow long the first(oldest) client in the queue has waited, in seconds, key metric
pgbouncer_pool_maxwait_usgaugedatname, job, ins, ip, instance, user, cls, pool_modeMicrosecond part of the maximum waiting time.
pgbouncer_pool_tested_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modeServer connections that are currently running reset or check query
pgbouncer_pool_used_serversgaugedatname, job, ins, ip, instance, user, cls, pool_modeServer connections that have been idle for more than server_check_delay (means have to run check query)
pgbouncer_pool_waiting_clientsgaugedatname, job, ins, ip, instance, user, cls, pool_modeClient connections that have sent queries but have not yet got a server connection
pgbouncer_stat_avg_query_countgaugedatname, job, ins, ip, instance, clsAverage queries per second in last stat period
pgbouncer_stat_avg_query_timegaugedatname, job, ins, ip, instance, clsAverage query duration, in seconds
pgbouncer_stat_avg_recvgaugedatname, job, ins, ip, instance, clsAverage received (from clients) bytes per second
pgbouncer_stat_avg_sentgaugedatname, job, ins, ip, instance, clsAverage sent (to clients) bytes per second
pgbouncer_stat_avg_wait_timegaugedatname, job, ins, ip, instance, clsTime spent by clients waiting for a server, in seconds (average per second).
pgbouncer_stat_avg_xact_countgaugedatname, job, ins, ip, instance, clsAverage transactions per second in last stat period
pgbouncer_stat_avg_xact_timegaugedatname, job, ins, ip, instance, clsAverage transaction duration, in seconds
pgbouncer_stat_total_query_countgaugedatname, job, ins, ip, instance, clsTotal number of SQL queries pooled by pgbouncer
pgbouncer_stat_total_query_timecounterdatname, job, ins, ip, instance, clsTotal number of seconds spent when executing queries
pgbouncer_stat_total_receivedcounterdatname, job, ins, ip, instance, clsTotal volume in bytes of network traffic received by pgbouncer
pgbouncer_stat_total_sentcounterdatname, job, ins, ip, instance, clsTotal volume in bytes of network traffic sent by pgbouncer
pgbouncer_stat_total_wait_timecounterdatname, job, ins, ip, instance, clsTime spent by clients waiting for a server, in seconds
pgbouncer_stat_total_xact_countgaugedatname, job, ins, ip, instance, clsTotal number of SQL transactions pooled by pgbouncer
pgbouncer_stat_total_xact_timecounterdatname, job, ins, ip, instance, clsTotal number of seconds spent when in a transaction
pgbouncer_upgaugejob, ins, ip, instance, clslast scrape was able to connect to the server: 1 for yes, 0 for no
pgbouncer_versiongaugejob, ins, ip, instance, clsserver version number
process_cpu_seconds_totalcounterjob, ins, ip, instance, clsTotal user and system CPU time spent in seconds.
process_max_fdsgaugejob, ins, ip, instance, clsMaximum number of open file descriptors.
process_open_fdsgaugejob, ins, ip, instance, clsNumber of open file descriptors.
process_resident_memory_bytesgaugejob, ins, ip, instance, clsResident memory size in bytes.
process_start_time_secondsgaugejob, ins, ip, instance, clsStart time of the process since unix epoch in seconds.
process_virtual_memory_bytesgaugejob, ins, ip, instance, clsVirtual memory size in bytes.
process_virtual_memory_max_bytesgaugejob, ins, ip, instance, clsMaximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flightgaugejob, ins, ip, instance, clsCurrent number of scrapes being served.
promhttp_metric_handler_requests_totalcountercode, job, ins, ip, instance, clsTotal number of scrapes by HTTP status code.
scrape_duration_secondsUnknownjob, ins, ip, instance, clsN/A
scrape_samples_post_metric_relabelingUnknownjob, ins, ip, instance, clsN/A
scrape_samples_scrapedUnknownjob, ins, ip, instance, clsN/A
scrape_series_addedUnknownjob, ins, ip, instance, clsN/A
upUnknownjob, ins, ip, instance, clsN/A

11 - Parameters

Customize PostgreSQL clusters with 120 parameters in the PGSQL module

Install PGSQL only on nodes that Pigsty manages through the NODE module. The deployment also needs an available ETCD cluster to store cluster metadata.

Install PGSQL on one node to create a standalone server, which is a primary instance. Install it on additional nodes to create read replicas that provide standby capacity and read-only access. You can create offline instances for ETL, OLAP, and interactive queries. Use synchronous standby and quorum commit for stronger data consistency. Use standby clusters and delayed clusters to respond quickly to data loss from human error or software defects.

You can organize multiple PGSQL clusters into a horizontally sharded cluster. Pigsty supports Citus cluster groups natively. You can upgrade a standard PGSQL cluster to a distributed database cluster in place.


SectionDescription
PG_IDPostgreSQL cluster and instance identity parameters
PG_BUSINESSBusiness users, databases, services and access control rule definition
PG_INSTALLPostgreSQL installation: version, paths, packages
PG_BOOTSTRAPPostgreSQL cluster initialization: Patroni high availability
PG_PROVISIONPostgreSQL cluster template provisioning: roles, privileges, extensions
PG_BACKUPpgBackRest backup and recovery configuration
PG_ACCESSService exposure, connection pooling, VIP, DNS client access config
PG_MONITORPostgreSQL monitoring exporter configuration
PG_REMOVEPostgreSQL instance cleanup and uninstall configuration

Parameter Overview


PG_ID defines cluster and instance identity, including cluster name, instance sequence number, role, shard, and other core identifiers.

ParameterTypeLevelDescription
pg_modeenumCpgsql cluster mode: pgsql,citus,mssql,mysql,ivory,pgtde,polar,gpsql,agens,oriole,pgedge
pg_clusterstringCpgsql cluster name, REQUIRED identity parameter
pg_seqintIpgsql instance seq number, REQUIRED identity parameter
pg_roleenumIpgsql instance role: primary, replica, standby, offline, or delayed
pg_instancesdictIdefine multiple pg instances on node in {port:ins_vars} format
pg_upstreamipIreplication upstream IP for a standby cluster or cascade replica
pg_shardstringCpgsql shard name; specify explicitly for horizontal sharding
pg_groupintCnon-negative pgsql shard index; specify explicitly for horizontal sharding
gp_roleenumCgreenplum role of this cluster, could be master or segment
pg_exportersdictCadditional pg_exporters to monitor remote postgres instances
pg_offline_queryboolIset to true to mark this replica as offline instance for offline queries

PG_BUSINESS defines business users, databases, services, access control rules, and default system user credentials.

ParameterTypeLevelDescription
pg_usersuser[]Cpostgres business users
pg_databasesdatabase[]Cpostgres business databases
pg_servicesservice[]Cpostgres business services
pg_hba_ruleshba[]Cbusiness hba rules for postgres
pgb_hba_ruleshba[]Cbusiness hba rules for pgbouncer
pg_crontabstring[]Ccrontab entries for postgres dbsu
pg_replication_usernameusernameGpostgres replication username, replicator by default
pg_replication_passwordpasswordGpostgres replication password, DBUser.Replicator by default
pg_admin_usernameusernameGpostgres admin username, dbuser_dba by default
pg_admin_passwordpasswordGpostgres admin password in plain text, DBUser.DBA by default
pg_monitor_usernameusernameGpostgres monitor username, dbuser_monitor by default
pg_monitor_passwordpasswordGpostgres monitor password, DBUser.Monitor by default
pg_dbsu_passwordpasswordG/Cdbsu password, empty string disables it by default, best not set

PG_INSTALL configures the PostgreSQL version, installation paths, packages, and extensions.

ParameterTypeLevelDescription
pg_dbsuusernameCos dbsu name, postgres by default, better not change it
pg_dbsu_uidintCos dbsu uid and gid, 26 for default postgres user and group
pg_dbsu_sudoenumCdbsu sudo privilege, none,limit,all,nopass. limit by default
pg_dbsu_homepathCpostgresql home directory, /var/lib/pgsql by default
pg_dbsu_ssh_exchangeboolCexchange postgres dbsu ssh key among same pgsql cluster
pg_versionenumCpostgres major version to be installed, 18 by default
pg_bin_dirpathCpostgres binary dir, /usr/pgsql/bin by default
pg_log_dirpathCpostgres log dir, /pg/log/postgres by default
pg_packagesstring[]Cpg packages to be installed, ${pg_version} will be replaced
pg_extensionsstring[]Cpg extensions to be installed, ${pg_version} will be replaced

PG_BOOTSTRAP configures cluster initialization, including Patroni high availability, storage paths, networking, encoding, and other core settings.

ParameterTypeLevelDescription
pg_datapathCPostgreSQL data directory, /pg/data by default
pg_fs_mainpathCmountpoint/path for pg main data, /data/postgres by default
pg_fs_backuppathCmountpoint/path for pg backup data, /data/backups by default
pg_storage_typeenumCstorage type for pg main data, SSD,HDD. SSD by default
pg_dummy_filesizesizeCsize of /pg/dummy, hold 64MB disk space for emergency use
pg_listenip(s)C/Ipostgres/pgbouncer listen addr, comma separated list, 0.0.0.0
pg_portportCpostgres listen port, 5432 by default
pg_localhostpathCpostgres unix socket dir for localhost connection
pg_namespacepathCtop level key namespace in etcd, used by patroni & vip
patroni_enabledboolCif disabled, no postgres cluster will be created during init
patroni_modeenumCpatroni working mode: default,pause,remove
patroni_portportCpatroni listen port, 8008 by default
patroni_log_dirpathCpatroni log dir, /pg/log/patroni by default
patroni_ssl_enabledboolGsecure patroni RestAPI communications with SSL?
patroni_watchdog_modeenumCpatroni watchdog mode: automatic,required,off. off by default
patroni_usernameusernameCpatroni restapi username, postgres by default
patroni_passwordpasswordCpatroni restapi password, Patroni.API by default
pg_primary_dbstringCprimary database name, used by citus,etc. postgres by default
pg_parametersdictCextra parameters in postgresql.auto.conf
pg_filespath[]Cextra files to be copied to PGDATA (e.g. license files)
pg_confenumCconfig template: oltp,olap,crit,tiny. oltp.yml by default
pg_max_connintCpostgres max connections, auto will use recommended value
pg_shared_buffer_ratiofloatCpostgres shared buffer memory ratio, 0.25 by default, 0.1~0.4
pg_rtoenumCRTO mode: fast, norm, safe, or wide; default norm
pg_rto_plandictGRTO presets for Patroni HA and HAProxy health-check timeouts
pg_rpointCsampled lag threshold for Patroni failover candidates; default 1MiB
pg_libsstringCpreloaded libraries, pg_stat_statements,auto_explain by default
pg_delayintervalIWAL replay apply delay for standby cluster, for delayed replica
pg_checksumboolCenable data checksum for postgres cluster?
pg_pwd_encenumCpassword encryption algorithm: fixed to scram-sha-256
pg_encodingenumCdatabase cluster encoding, UTF8 by default
pg_localeenumCdatabase cluster locale, C by default
pg_lc_collateenumCdatabase cluster collate, C by default
pg_lc_ctypeenumCdatabase character type, C by default
pg_io_methodenumCPostgreSQL IO method: auto, sync, worker, io_uring
pg_etcd_passwordpasswordCetcd password for this PostgreSQL cluster, cluster name by default
pgsodium_keystringCpgsodium encryption master key, 64 hex digits, sha256(pg_cluster)
pgsodium_getkey_scriptpathCpgsodium getkey script path, uses template pgsodium_getkey

PG_PROVISION configures cluster templates, including default roles, privileges, schemas, extensions, and HBA rules.

ParameterTypeLevelDescription
pg_provisionboolCprovision postgres cluster content after bootstrap?
pg_initstringG/Cinit script for cluster template, pg-init by default
pg_default_rolesrole[]G/Cdefault predefined roles and system users in postgres
pg_default_privilegesstring[]G/Cdefault privileges when created by admin user
pg_default_schemasstring[]G/Cdefault schemas to be created
pg_default_extensionsextension[]G/Cdefault extensions to be created
pg_reloadboolAreload postgres config after hba changes?
pg_default_hba_ruleshba[]G/Cpostgres default host-based auth rules, global default HBA
pgb_default_hba_ruleshba[]G/Cpgbouncer default host-based auth rules, global default HBA

PG_BACKUP configures pgBackRest backup and recovery, including repository types, paths, and retention policies.

ParameterTypeLevelDescription
pgbackrest_enabledboolCenable pgbackrest on pgsql host?
pgbackrest_log_dirpathCpgbackrest log dir, /pg/log/pgbackrest by default
pgbackrest_methodenumCpgbackrest repo method: local,minio,etc…
pgbackrest_init_backupboolCperform full backup after init? true by default
pgbackrest_repodictG/Cpgbackrest repo definition

PG_ACCESS configures services, connection pooling, VIP, DNS, and other client access options.

ParameterTypeLevelDescription
pgbouncer_enabledboolCif disabled, pgbouncer will not be configured
pgbouncer_portportCpgbouncer listen port, 6432 by default
pgbouncer_log_dirpathCpgbouncer log dir, /pg/log/pgbouncer by default
pgbouncer_auth_queryboolCuse AuthQuery to get unlisted business users from postgres?
pgbouncer_poolmodeenumCpool mode: transaction,session,statement. transaction by default
pgbouncer_sslmodeenumCpgbouncer client ssl mode, disabled by default
pgbouncer_ignore_paramstring[]Cpgbouncer ignore startup parameters list
pg_weightintIrelative load balancing weight in service, 0-255, 100 by default
pg_service_providerstringG/Cdedicated haproxy node group name, or use local haproxy
pg_default_service_destenumG/Cdefault service dest if svc.dest=‘default’: postgres or pgbouncer
pg_default_servicesservice[]G/Cpostgres default service definition list, shared globally
pg_vip_enabledboolCenable L2 VIP for pgsql primary? disabled by default
pg_vip_addresscidr4Cvip address in <ipv4>/<mask> format, required if vip enabled
pg_vip_interfacestringC/Ivip network interface to bind, auto by default
pg_dns_suffixstringCpgsql dns suffix, empty by default
pg_dns_targetenumCPG DNS resolves to: auto, primary, vip, none, or specific IP

PG_MONITOR configures PostgreSQL monitoring exporters: pg_exporter, pgbouncer_exporter, and pgbackrest_exporter.

ParameterTypeLevelDescription
pg_exporter_enabledboolCenable pg_exporter on pgsql host?
pg_exporter_configstringCpg_exporter config file/template name
pg_exporter_cache_ttlsstringCpg_exporter collector ttl stages, ‘1,10,60,300’ by default
pg_exporter_portportCpg_exporter listen port, 9630 by default
pg_exporter_paramsstringCextra URL parameters for pg_exporter dsn
pg_exporter_urlpgurlCoverwrite auto-generated postgres DSN connection string
pg_exporter_auto_discoveryboolCenable auto database discovery for monitoring? enabled
pg_exporter_exclude_databasestringCexcluded database list when auto-discovery, comma separated
pg_exporter_include_databasestringConly monitor these databases when auto-discovery enabled
pg_exporter_connect_timeoutintCpg_exporter connect timeout in ms, 200 by default
pg_exporter_optionsargCextra command line options for pg_exporter
pgbouncer_exporter_enabledboolCenable pgbouncer_exporter on pgsql host?
pgbouncer_exporter_portportCpgbouncer_exporter listen port, 9631 by default
pgbouncer_exporter_urlpgurlCoverwrite auto-generated pgbouncer dsn connection string
pgbouncer_exporter_optionsargCextra command line options for pgbouncer_exporter
pgbackrest_exporter_enabledboolCenable pgbackrest_exporter on pgsql host?
pgbackrest_exporter_portportCpgbackrest_exporter listen port, 9854 by default
pgbackrest_exporter_optionsargCextra command line options for pgbackrest_exporter

PG_REMOVE controls instance removal, including data directories, backups, and packages.

ParameterTypeLevelDescription
pg_rm_databoolG/C/Aremove postgres data directory when removing instance?
pg_rm_backupboolG/C/Aremove pgbackrest backup when removing primary?
pg_rm_pkgboolG/C/Auninstall related packages when removing pgsql instance?
pg_safeguardboolG/C/Aprevent accidental pgsql cleanup operations? false

PG_ID

Use these parameters to identify PGSQL entities, such as clusters, instances, and services:

# pg_cluster:           #CLUSTER  # pgsql cluster name, required identity parameter
# pg_seq: 0             #INSTANCE # pgsql instance seq number, required identity parameter
# pg_role: replica      #INSTANCE # pgsql role, required, could be primary,replica,offline
# pg_instances: {}      #INSTANCE # define multiple pg instances on node in `{port:ins_vars}` format
# pg_upstream:          #INSTANCE # repl upstream ip addr for standby cluster or cascade replica
# pg_shard:             #CLUSTER  # pgsql shard name, optional identity for sharding clusters
# pg_group: 0           #CLUSTER  # pgsql shard index number, optional identity for sharding clusters
# gp_role: master       #CLUSTER  # greenplum role of this cluster, could be master or segment
pg_offline_query: false #INSTANCE # set to true to enable offline query on this instance

You must explicitly set these identity parameters. They have no defaults:

NameTypeLevelDescription
pg_clusterstringCPG cluster name
pg_seqnumberIPG instance ID
pg_roleenumIPG instance role
pg_shardstringCShard name
pg_groupnumberCShard index
  • pg_cluster: Identifies the cluster name, configured at cluster level.
  • pg_role: Configured at instance level, identifies the role of the instance. Only primary role is treated specially. If not specified, defaults to replica role, with special delayed and offline roles.
  • pg_seq: Used to identify instances within a cluster, typically an integer starting from 0 or 1, once assigned it doesn’t change.
  • {{ pg_cluster }}-{{ pg_seq }} uniquely identifies an instance, i.e., pg_instance.
  • {{ pg_cluster }}-{{ pg_role }} identifies services within the cluster, i.e., pg_service.
  • pg_shard and pg_group are used for horizontal sharding clusters, only for citus, greenplum, and matrixdb.

Every PostgreSQL cluster requires these core identity parameters: pg_cluster, pg_role, and pg_seq. Set them explicitly, as in this example:

pg-test:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: replica}
    10.10.10.12: {pg_seq: 2, pg_role: primary}
    10.10.10.13: {pg_seq: 3, pg_role: replica}
  vars:
    pg_cluster: pg-test

Other parameters can inherit global or default values. You must manually assign and explicitly set identity parameters.

pg_mode

Parameter Name: pg_mode, Type: enum, Level: C

PostgreSQL cluster mode, default value is pgsql, i.e., standard PostgreSQL cluster.

Available mode options include:

  • pgsql: Standard PostgreSQL cluster
  • citus: Citus distributed database cluster
  • mssql: Babelfish MSSQL wire protocol compatible kernel
  • mysql: OpenHalo/HaloDB MySQL wire protocol compatible kernel
  • ivory: IvorySQL Oracle compatible kernel
  • pgtde: Percona PostgreSQL with pg_tde
  • polar: PolarDB for PostgreSQL kernel
  • gpsql: Greenplum parallel database cluster (monitoring)
  • agens: AgensGraph graph database kernel
  • oriole: OrioleDB storage-engine kernel
  • pgedge: pgEdge distributed-replication kernel

pg_shard defaults to pg_cluster, and pg_group defaults to 0. If pg_mode is citus or gpsql and the sharded system contains multiple physical clusters, set both explicitly. They define the horizontal-sharding identity.

In both cases, each PostgreSQL cluster is part of a larger business unit.

pg_cluster

Parameter Name: pg_cluster, Type: string, Level: C

PostgreSQL cluster name, required identity parameter, no default value.

The cluster name is used as the namespace for resources.

Current role validation accepts names matching ^[A-Za-z0-9-]+$ and rejects root. Lowercase names that start with a letter are still recommended for consistency with DNS names, service names, and operation scripts. Use only lowercase letters, digits, and hyphens in these names.

pg_seq

Parameter Name: pg_seq, Type: int, Level: I

PostgreSQL instance sequence number, required identity parameter, no default value.

Assign a unique sequence number to each instance within its cluster. Usually, use natural numbers starting at 0 or 1. Usually, do not recycle or reuse them.

pg_role

Parameter Name: pg_role, Type: enum, Level: I

PostgreSQL instance role, required identity parameter with no default. Current validation accepts primary, replica, standby, offline, and delayed.

The commonly used service-membership labels are:

  • primary: Primary instance, there is one and only one in a cluster.
  • replica: Replica for serving online read-only traffic, may have slight replication delay under high load (10ms~100ms, 100KB).
  • offline: Offline replica for handling offline read-only traffic, such as analytics/ETL/personal queries.

standby and delayed are valid inventory roles. However, these values alone do not create a standby cluster or delayed replication. Configure the topology with pg_upstream and pg_delay. HBA role filters use the inventory labels directly.

pg_instances

Parameter Name: pg_instances, Type: dict, Level: I

Define multiple PostgreSQL instances on a single host using {port:ins_vars} format.

This parameter reserves support for multiple instances on one node. Pigsty has not implemented this feature yet. A dedicated node per instance is strongly recommended.

pg_upstream

Parameter Name: pg_upstream, Type: ip, Level: I

Upstream instance IP address for standby cluster or cascade replica.

Set pg_upstream on the cluster primary to create a standby cluster. This instance becomes the standby leader and receives and applies changes from the upstream cluster.

On a non-primary instance, pg_upstream selects the upstream for physical replication. If this address differs from the primary IP, the instance becomes a cascade replica. You must select another instance in the same cluster.

pg_shard

Parameter Name: pg_shard, Type: string, Level: C

PostgreSQL horizontal shard name, defaulting to pg_cluster. Specify it explicitly for horizontal-sharding systems made up of multiple physical clusters, such as Citus.

A horizontal sharding cluster is a group of standard PostgreSQL clusters that jointly serve one business with horizontal sharding.

pg_shard is the shard group name. It is typically a prefix of pg_cluster.

For example, if we have a shard group pg-citus with 4 clusters, their identity parameters would be:

cls pg_shard: pg-citus
cls pg_group = 0:   pg-citus0
cls pg_group = 1:   pg-citus1
cls pg_group = 2:   pg-citus2
cls pg_group = 3:   pg-citus3

pg_group

Parameter Name: pg_group, Type: int, Level: C

PostgreSQL horizontal-sharding cluster index, defaulting to 0. Specify it explicitly for horizontal-sharding systems made up of multiple physical clusters, such as Citus.

Use this parameter with pg_shard. The index is usually a non-negative integer.

gp_role

Parameter Name: gp_role, Type: enum, Level: C

Greenplum/Matrixdb role of the PostgreSQL cluster, can be master or segment.

  • master: Marks the postgres cluster as a greenplum master instance (coordinator node), this is the default value.
  • segment: Marks the postgres cluster as a greenplum segment cluster (data node).

This parameter applies only to Greenplum/MatrixDB when pg_mode is gpsql. It has no effect on a regular PostgreSQL cluster.

pg_exporters

Parameter Name: pg_exporters, Type: dict, Level: C

Additional exporter definitions for monitoring remote PostgreSQL instances, default value: {}

To monitor remote PostgreSQL instances, define them in pg_exporters on the infrastructure cluster that runs monitoring. Then deploy with pgsql-monitor.yml.

pg_exporters: # list all remote instances here, alloc a unique unused local port as k
    20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
    20004: { pg_cluster: pg-foo, pg_seq: 2, pg_host: 10.10.10.11 }
    20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.12 }
    20003: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.13 }

pg_offline_query

Parameter Name: pg_offline_query, Type: bool, Level: I

Set to true to mark this instance as eligible for offline queries. The default is false.

The flag adds the instance to the default offline service candidates. It also enables HBA rules with role: offline on that instance. The flag alone does not grant connection access. Access still depends on generated HBA rules, database CONNECT privileges, and role attributes.

This flag has an effect similar to pg_role = offline. However, dedicated offline instances do not serve replica requests by default. They serve only offline and analytical workloads.

If no dedicated offline instance is available, enable this parameter on a regular replica. To restrict dbrole_offline to these instances, explicitly set role: offline on its HBA rule.


PG_BUSINESS

Customize cluster templates: users, databases, services, and permission rules.

Review these parameters carefully. They declare the database objects that your business needs.

Default database users and their credentials. Their passwords must be changed in production.

# postgres business object definition, overwrite in group vars
pg_users: []                      # postgres business users
pg_databases: []                  # postgres business databases
pg_services: []                   # postgres business services
pg_hba_rules: []                  # business hba rules for postgres
pgb_hba_rules: []                 # business hba rules for pgbouncer
pg_crontab: []                    # crontab entries for postgres dbsu
# global credentials, overwrite in global vars
pg_dbsu_password: ''              # dbsu password, empty string means no dbsu password by default
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor

pg_users

Parameter Name: pg_users, Type: user[], Level: C

This list defines PostgreSQL business users at cluster level. The default is the empty list [].

Each array element is a user/role definition, for example:

- name: dbuser_meta               # required, `name` is the only required field for user definition
  password: DBUser.Meta           # optional, password, can be scram-sha-256 hash string or plaintext
  login: true                     # optional, can login by default
  superuser: false                # optional, default false, is superuser?
  createdb: false                 # optional, default false, can create database?
  createrole: false               # optional, default false, can create role?
  inherit: true                   # optional, by default, can this role use inherited privileges?
  replication: false              # optional, default false, can this role do replication?
  bypassrls: false                # optional, default false, can this role bypass row-level security?
  pgbouncer: true                 # optional, default false, add this user to pgbouncer user list? (production users using connection pool should explicitly set to true)
  connlimit: -1                   # optional, user connection limit, default -1 disables limit
  expire_in: 3650                 # optional, this role expires: calculated from creation + n days (higher priority than expire_at)
  expire_at: '2030-12-31'         # optional, when this role expires, use YYYY-MM-DD format string to specify a specific date (lower priority than expire_in)
  comment: pigsty admin user      # optional, description and comment string for this user/role
  roles: [dbrole_admin]           # optional, default roles are: dbrole_{admin,readonly,readwrite,offline}
  parameters: {}                  # optional, use `ALTER ROLE SET` for this role, configure role-level database parameters
  pool_mode: transaction          # optional, pgbouncer pool mode at user level, default transaction
  pool_connlimit: 100             # optional, user-level pool connection limit; omitted values inherit Pigsty's global default of 100
  search_path: public             # optional, key-value config parameter per postgresql docs (e.g., use pigsty as default search_path)

User-level pool quota is consistently defined by pool_connlimit (mapped to Pgbouncer max_user_connections).

pg_databases

Parameter Name: pg_databases, Type: database[], Level: C

This list defines PostgreSQL business databases at cluster level. The default is the empty list [].

Each array element is a business database definition, for example:

- name: meta                      # required, `name` is the only required field for database definition
  baseline: cmdb.sql              # optional, database sql baseline file path (relative path in ansible search path, e.g., files/)
  pgbouncer: true                 # optional, add this database to pgbouncer database list? default true
  schemas: [pigsty]               # optional, additional schemas to create, array of schema name strings
  extensions:                     # optional, additional extensions to install: array of extension objects
    - { name: postgis , schema: public }  # can specify which schema to install extension into, or not (if not specified, installs to first schema in search_path)
    - { name: timescaledb }               # some extensions create and use fixed schemas, so no need to specify schema
  comment: pigsty meta database   # optional, description and comment for the database
  owner: postgres                 # optional, database owner, default is postgres
  template: template1             # optional, template to use, default is template1, target must be a template database
  encoding: UTF8                  # optional, database encoding, default UTF8 (must match template database)
  locale: C                       # optional, database locale setting, default C (must match template database)
  lc_collate: C                   # optional, database collate rule, default C (must match template database), no reason to change
  lc_ctype: C                     # optional, database ctype character set, default C (must match template database)
  tablespace: pg_default          # optional, default tablespace, default is 'pg_default'
  allowconn: true                 # optional, allow connections, default true. Explicitly set false to completely forbid connections
  revokeconn: false               # optional, revoke public connect privileges. default false, when true, CONNECT privilege revoked from users other than owner and admin
  register_datasource: true       # optional, register this database to grafana datasource? default true, explicitly false skips registration
  connlimit: -1                   # optional, database connection limit, default -1 means no limit, positive integer limits connections
  pool_auth_user: dbuser_meta     # optional, all connections to this pgbouncer database will authenticate using this user (useful when pgbouncer_auth_query enabled)
  pool_mode: transaction          # optional, database-level pgbouncer pooling mode, default transaction
  pool_size: 50                   # optional, database-level pgbouncer default pool size, default 50
  pool_reserve: 30                # optional, database-level pgbouncer pool reserve, default 30, max additional burst connections when default pool insufficient
  pool_size_min: 0                # optional, database-level pgbouncer pool minimum size, default 0
  pool_connlimit: 100             # optional, database-level max database connections, default 100

Since Pigsty v4.1.0, use pool_reserve and pool_connlimit for database pool fields. These names replace the legacy aliases pool_size_reserve and pool_max_db_conn.

In a database definition, only name is required. All other fields are optional.

pg_services

Parameter Name: pg_services, Type: service[], Level: C

This list defines PostgreSQL services at cluster level. The default is the empty list [].

Use this parameter for additional cluster-level services. Each array object defines one service. This example shows a complete definition:

- name: standby                   # required, service name, final svc name will use `pg_cluster` as prefix, e.g., pg-meta-standby
  port: 5435                      # required, exposed service port (as kubernetes service node port mode)
  ip: "*"                         # optional, IP address to bind service, default is all IP addresses
  selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
  backup: "[? pg_role == `primary`]"  # optional, service member selector (backup), service is handled by these instances when default selector instances are all down
  dest: default                   # optional, target port, default|postgres|pgbouncer|<port_number>, default is 'default', Default means use pg_default_service_dest value to decide
  check: /sync                    # optional, health check URL path, default is /, here uses Patroni API: /sync, only sync standby and primary return 200 health status
  maxconn: 5000                   # optional, max frontend connections allowed, default 5000
  balance: roundrobin             # optional, haproxy load balancing algorithm (default roundrobin, other option: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

This parameter adds cluster-level services. For services that all PostgreSQL clusters must provide, use pg_default_services.

pg_hba_rules

Parameter Name: pg_hba_rules, Type: hba[], Level: C

Client IP whitelist/blacklist rules for database cluster/instance. Default: [] empty list.

Array of objects, each object represents a rule. HBA rule object definition:

- title: allow intranet password access
  role: common
  rules:
    - host   all  all  10.0.0.0/8      md5
    - host   all  all  172.16.0.0/12   md5
    - host   all  all  192.168.0.0/16  md5
  • title: Rule title name, rendered as comment in HBA file.
  • rules: Rule array, each element is a standard HBA rule string.
  • role: Rule application scope, which instance roles will enable this rule?
    • common: Applies to all instances
    • primary, replica, offline: Only applies to instances with specific pg_role.
    • Special case: role: 'offline' rules apply to instances with pg_role : offline, and also to instances with pg_offline_query flag.

In addition to the native HBA rule definition above, Pigsty also provides a more convenient alias form:

- addr: 'intra'    # world|intra|infra|admin|local|localhost|cluster|<cidr>
  auth: 'pwd'      # trust|pwd|ssl|cert|deny|<official auth method>
  user: 'all'      # all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>
  db: 'all'        # all|replication|....
  rules: []        # raw hba string precedence over above all
  title: allow intranet password access

pg_default_hba_rules defines global HBA rules. Use this parameter for rules specific to a cluster or instance.

pgb_hba_rules

Parameter Name: pgb_hba_rules, Type: hba[], Level: C

Pgbouncer business HBA rules, default value: [], empty array.

Like pg_hba_rules, this parameter is an array of HBA rule objects. It applies to Pgbouncer.

pgb_default_hba_rules defines global pool HBA rules. Use this parameter for rules specific to a pool cluster or instance.

pg_crontab

Parameter Name: pg_crontab, Type: string[], Level: C

This list defines cron jobs for the PostgreSQL database superuser (dbsu), which defaults to postgres. The default list is empty: [].

Each array element is one user crontab entry: minute hour day month weekday command. Do not include a username.

pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'      # Full backup at 1 AM daily
  - '00 13 * * * /pg/bin/pg-backup'           # Incremental backup at 1 PM daily

This parameter writes cron jobs to the postgres user’s personal crontab file:

  • EL systems: /var/spool/cron/postgres
  • Debian systems: /var/spool/cron/crontabs/postgres

Note: This parameter replaces the former use of node_crontab for postgres user tasks. NODE initialization writes node_crontab to /etc/crontab. At that time, the postgres user may not exist, which causes cron errors.

pg_replication_username

Parameter Name: pg_replication_username, Type: username, Level: G

PostgreSQL physical replication username, default is replicator, not recommended to change this parameter.

pg_replication_password

Parameter Name: pg_replication_password, Type: password, Level: G

PostgreSQL physical replication user password, default value: DBUser.Replicator.

Warning: Change this password in production environments.

pg_admin_username

Parameter Name: pg_admin_username, Type: username, Level: G

PostgreSQL / Pgbouncer admin name, default: dbuser_dba.

This global database administrator has superuser privileges and permission to manage connection pool traffic. Restrict where you use it.

pg_admin_password

Parameter Name: pg_admin_password, Type: password, Level: G

PostgreSQL / Pgbouncer admin password, default: DBUser.DBA.

Warning: Change this password in production environments.

pg_monitor_username

Parameter Name: pg_monitor_username, Type: username, Level: G

PostgreSQL/Pgbouncer monitor username, default: dbuser_monitor.

This user monitors databases and connection pools. Changing the username is not recommended.

If an existing database has a different monitoring user, set this parameter when you define the monitoring target.

pg_monitor_password

Parameter Name: pg_monitor_password, Type: password, Level: G

Password used by PostgreSQL/Pgbouncer monitor user, default: DBUser.Monitor.

Avoid password characters such as @:/ that can be confused with URL delimiters.

Warning: Change this password in production environments.

pg_dbsu_password

Parameter Name: pg_dbsu_password, Type: password, Level: G/C

PostgreSQL pg_dbsu superuser password, default is empty string, meaning no password is set.

Password login for dbsu is not recommended because it increases the attack surface. The exception is pg_mode = citus. Citus requires a dbsu password for each shard cluster to allow connections within the shard cluster.


PG_INSTALL

These parameters control PostgreSQL and extension installation. Change pg_version and pg_extensions to select major versions and extensions. Some extensions are unavailable for some major versions.

pg_dbsu: postgres                 # os dbsu name, default is postgres, better not change it
pg_dbsu_uid: 26                   # os dbsu uid and gid, default is 26, for default postgres user and group
pg_dbsu_sudo: limit               # dbsu sudo privilege, none,limit,all,nopass. default is limit
pg_dbsu_home: /var/lib/pgsql      # postgresql home directory, default is `/var/lib/pgsql`
pg_dbsu_ssh_exchange: true        # exchange postgres dbsu ssh key among same pgsql cluster
pg_version: 18                    # postgres major version to be installed, default is 18
pg_bin_dir: /usr/pgsql/bin        # postgres binary dir, default is `/usr/pgsql/bin`
pg_log_dir: /pg/log/postgres      # postgres log dir, default is `/pg/log/postgres`
pg_packages:                      # pg packages to be installed, alias can be used
  - pgsql-main pgsql-common
pg_extensions: []                 # pg extensions to be installed, alias can be used

pg_dbsu

Parameter Name: pg_dbsu, Type: username, Level: C

OS dbsu username used by PostgreSQL, default is postgres, changing this username is not recommended.

Some deployments require a dbsu name other than postgres. For Greenplum, use gpadmin. For MatrixDB, use mxadmin.

pg_dbsu_uid

Parameter Name: pg_dbsu_uid, Type: int, Level: C

OS database superuser uid and gid, 26 is the default postgres user UID/GID from PGDG RPM.

Debian and Ubuntu have no default UID/GID, and 26 is often already in use. If Pigsty detects a Debian-based system and UID 26, it substitutes pg_dbsu_uid = 543 automatically.

pg_dbsu_sudo

Parameter Name: pg_dbsu_sudo, Type: enum, Level: C

Database superuser sudo privilege, can be none, limit, all, or nopass. Default is limit

  • none: No sudo privilege

  • limit: Limited sudo privilege for executing systemctl commands for database-related components (default option).

  • all: Full sudo privilege, requires password.

  • nopass: Full sudo privilege without password (not recommended).

  • Default value is limit, only allows executing sudo systemctl <start|stop|reload> <postgres|patroni|pgbouncer|...>.

pg_dbsu_home

Parameter Name: pg_dbsu_home, Type: path, Level: C

PostgreSQL home directory, default is /var/lib/pgsql, consistent with official pgdg RPM.

pg_dbsu_ssh_exchange

Parameter Name: pg_dbsu_ssh_exchange, Type: bool, Level: C

Whether to exchange OS dbsu ssh keys within the same PostgreSQL cluster?

Default is true, meaning database superusers in the same cluster can ssh to each other.

pg_cluster_members selects hosts with the same pg_cluster across the current inventory for key exchange. It does not require an Ansible group with the cluster name. The runtime -l limit still restricts playbook targets. Include every member that needs configuration.

pg_version

Parameter Name: pg_version, Type: enum, Level: C

PostgreSQL major version to install, default is 18.

Physical streaming replication cannot cross PostgreSQL major versions. Thus, avoid setting this parameter at instance level.

You can use parameters in pg_packages and pg_extensions to install different packages and extensions for specific PG major versions.

pg_bin_dir

Parameter Name: pg_bin_dir, Type: path, Level: C

PostgreSQL binary directory, default is /usr/pgsql/bin.

Installation creates this symlink manually. It points to the directory for the installed PostgreSQL version.

For example /usr/pgsql -> /usr/pgsql-15. On Ubuntu/Debian it points to /usr/lib/postgresql/15/bin.

For more details, see PGSQL File Structure.

pg_log_dir

Parameter Name: pg_log_dir, Type: path, Level: C

PostgreSQL log directory, default: /pg/log/postgres. The Vector log agent uses this variable to collect PostgreSQL logs.

If pg_log_dir is under pg_data, Pigsty does not create it separately. Data directory initialization creates it automatically.

pg_packages

Parameter Name: pg_packages, Type: string[], Level: C

This array specifies PostgreSQL RPM or DEB packages to install. Each element can contain package aliases separated by spaces or commas.

Pigsty v4 converges the default value to two aliases:

pg_packages:
  - pgsql-main pgsql-common
  • pgsql-main: Maps to PostgreSQL kernel, client, PL languages, and core extensions like pg_repack, wal2json, pgvector on the current platform.
  • pgsql-common: Maps to companion components required for running the database, such as Patroni, Pgbouncer, pgBackRest, pg_exporter, vip-manager, and other daemons.

Find aliases in pg_package_map under roles/node_id/vars/. Pigsty first resolves aliases for the operating system and architecture. It then replaces $v or ${pg_version} with pg_version and installs the packages. This hides package-name differences between distributions.

For additional packages, such as FDWs or extensions, append aliases or package names to pg_packages. Keep pgsql-main pgsql-common to ensure that core components remain installed.

pg_extensions

Parameter Name: pg_extensions, Type: string[], Level: G/C

This array specifies PostgreSQL extension RPM or DEB packages to install. Use package names or aliases.

Since v4, the default is an empty list, []. Pigsty no longer installs large extensions automatically. Select the extensions you need to avoid unnecessary disk use and dependencies.

To install extensions, fill in like this:

pg_extensions:
  - postgis timescaledb pgvector
  - pgsql-fdw     # use alias to install common FDWs at once

pg_package_map aliases hide package-name differences between distributions. These examples use extension aliases and category groups from the current v4.5 EL9 map. Availability can differ by platform and PostgreSQL major version:

pg_extensions: # extensions to be installed on this cluster
  - timescaledb postgis pgvector pg_search pg_duckdb pg_repack wal2json
  # You can also select current category groups; install only the groups you need
  # - pgsql-time pgsql-gis pgsql-rag pgsql-fts pgsql-olap
  # - pgsql-feat pgsql-lang pgsql-type pgsql-util pgsql-func
  # - pgsql-admin pgsql-stat pgsql-sec pgsql-fdw pgsql-sim pgsql-etl

For exact mappings, check the target platform in roles/node_id/vars/<os>.<arch>.yml and the current extension catalog. The v4.5 catalog and current mainstream-platform maps removed pg_analytics and spat. Do not copy them from older examples.


PG_BOOTSTRAP

Initialize the PostgreSQL cluster with Patroni and configure one Pgbouncer connection pool per instance.

It also initializes the database cluster with default roles, users, privileges, schemas, and extensions defined in PG_PROVISION.

These parameters configure PGSQL initialization. The internal variable pg_data always refers to the /pg/data symlink. Do not override it in the inventory. To change the physical data directory location, set pg_fs_main.

pg_fs_main: /data/postgres        # postgres main data directory, `/data/postgres` by default
pg_fs_backup: /data/backups       # postgres backup data directory, `/data/backups` by default
pg_storage_type: SSD              # storage type for pg main data, SSD,HDD, SSD by default
pg_dummy_filesize: 64MiB          # size of `/pg/dummy`, hold 64MB disk space for emergency use
pg_listen: '0.0.0.0'              # postgres/pgbouncer listen addresses, comma separated list
pg_port: 5432                     # postgres listen port, 5432 by default
pg_localhost: /var/run/postgresql # postgres unix socket dir for localhost connection
patroni_enabled: true             # if disabled, no postgres cluster will be created during init
patroni_mode: default             # patroni working mode: default,pause,remove
pg_namespace: /pg                 # top level key namespace in etcd, used by patroni & vip
patroni_port: 8008                # patroni listen port, 8008 by default
patroni_log_dir: /pg/log/patroni  # patroni log dir, `/pg/log/patroni` by default
patroni_ssl_enabled: false        # secure patroni RestAPI communications with SSL?
patroni_watchdog_mode: off        # patroni watchdog mode: automatic,required,off. off by default
patroni_username: postgres        # patroni restapi username, `postgres` by default
patroni_password: Patroni.API     # patroni restapi password, `Patroni.API` by default
pg_etcd_password: ''              # etcd password for this pg cluster, '' to use pg_cluster
pg_primary_db: postgres           # primary database name, used by citus,etc... ,postgres by default
pg_parameters: {}                 # extra parameters in postgresql.auto.conf
pg_files: []                      # extra files to be copied to postgres data directory (e.g. license)
pg_conf: oltp.yml                 # config template: oltp,olap,crit,tiny. `oltp.yml` by default
pg_max_conn: auto                 # postgres max connections, `auto` will use recommended value
pg_shared_buffer_ratio: 0.25      # postgres shared buffers ratio, 0.25 by default, 0.1~0.4
pg_io_method: worker              # io method for postgres, auto,sync,worker,io_uring, worker by default
pg_rto: norm                      # shared RTO mode: fast,norm,safe,wide
pg_rpo: 1048576                   # sampled failover-candidate lag threshold in bytes, `1MiB` by default
pg_libs: 'pg_stat_statements, auto_explain'  # preloaded libraries, `pg_stat_statements,auto_explain` by default
pg_delay: 0                       # replication apply delay for standby cluster leader
pg_checksum: true                 # enable data checksum for postgres cluster?
pg_pwd_enc: scram-sha-256         # passwords encryption algorithm: fixed to scram-sha-256
pg_encoding: UTF8                 # database cluster encoding, `UTF8` by default
pg_locale: C                      # database cluster local, `C` by default
pg_lc_collate: C                  # database cluster collate, `C` by default
pg_lc_ctype: C                    # database character type, `C` by default
#pgsodium_key: ""                 # pgsodium key, 64 hex digit, default to sha256(pg_cluster)
#pgsodium_getkey_script: ""       # pgsodium getkey script path, pgsodium_getkey by default

pg_data

Internal Variable: pg_data, Type: path

pg_data is an internal Pigsty variable. It always refers to the PostgreSQL data directory symlink, /pg/data. It is not a user configuration parameter.

Patroni templates, maintenance scripts, and cleanup procedures depend on this symlink. Do not override or change it in pigsty.yml. Set pg_fs_main to change the physical data location. See PGSQL File Structure.

pg_fs_main

Parameter Name: pg_fs_main, Type: path, Level: C

Mount point/file system path for PostgreSQL main data disk, default is /data/postgres.

Default value: /data/postgres, which will be used directly as the parent directory of PostgreSQL main data directory.

NVMe SSD storage is recommended for PostgreSQL main data. Pigsty optimizes for SSDs by default and also supports HDDs.

You can change pg_storage_type to HDD for HDD storage optimization.

pg_fs_backup

Parameter Name: pg_fs_backup, Type: path, Level: C

Mount point/file system path for PostgreSQL backup data disk, default is /data/backups.

If you’re using the default pgbackrest_method = local, it’s recommended to use a separate disk for backup storage.

The backup disk must hold all required backups. It should hold at least 3 base backups and 2 days of WAL archives. Large, inexpensive HDDs usually provide sufficient capacity.

Use a separate backup disk when possible. Otherwise, Pigsty uses the main data disk and consumes its capacity and I/O.

pg_storage_type

Parameter Name: pg_storage_type, Type: enum, Level: C

Type of PostgreSQL data storage media: SSD or HDD, default is SSD.

Default value: SSD, which affects some tuning parameters like random_page_cost and effective_io_concurrency.

pg_dummy_filesize

Parameter Name: pg_dummy_filesize, Type: size, Level: C

Size of /pg/dummy, default is 64MiB, 64MB disk space for emergency use.

If the disk is full, delete the placeholder file to release emergency space. For production, use at least 8GiB.

pg_listen

Parameter Name: pg_listen, Type: ip, Level: C

PostgreSQL / Pgbouncer listen address, default is 0.0.0.0 (all ipv4 addresses).

You can use placeholders in this variable, for example: '${ip},${lo}' or '${ip},${vip},${lo}':

  • ${ip}: Translates to inventory_hostname, which is the primary internal IP address defined in the inventory.
  • ${vip}: If pg_vip_enabled is enabled, will use the host part of pg_vip_address.
  • ${lo}: Will be replaced with 127.0.0.1

For production environments with high security requirements, restrict the listen IP addresses.

pg_port

Parameter Name: pg_port, Type: port, Level: C

Port that PostgreSQL server listens on, default is 5432.

pg_localhost

Parameter Name: pg_localhost, Type: path, Level: C

Unix socket directory for localhost PostgreSQL connection, default is /var/run/postgresql.

This directory holds Unix sockets for local PostgreSQL and Pgbouncer connections. pg_exporter and Patroni prefer Unix sockets for PostgreSQL access.

pg_namespace

Parameter Name: pg_namespace, Type: path, Level: C

Top-level namespace used in etcd, used by patroni and vip-manager, default is: /pg, not recommended to change.

patroni_enabled

Parameter Name: patroni_enabled, Type: bool, Level: C

Enable Patroni? Default is: true.

If disabled, Pigsty does not create a PostgreSQL cluster during initialization. It skips Patroni startup. Use this setting when adding components to an existing PostgreSQL instance.

patroni_mode

Parameter Name: patroni_mode, Type: enum, Level: C

Patroni working mode: default, pause, remove. Default: default.

  • default: Normal use of Patroni to bootstrap PostgreSQL cluster
  • pause: Similar to default, but enters maintenance mode after bootstrap
  • remove: Use Patroni to initialize cluster, then remove Patroni and use raw PostgreSQL.

patroni_port

Parameter Name: patroni_port, Type: port, Level: C

Patroni listen port, default is 8008, not recommended to change.

Patroni API server listens on this port for health checks and API requests.

patroni_log_dir

Parameter Name: patroni_log_dir, Type: path, Level: C

Patroni log directory, default is /pg/log/patroni, collected by Vector log agent.

patroni_ssl_enabled

Parameter Name: patroni_ssl_enabled, Type: bool, Level: G

Secure patroni RestAPI communications with SSL? Default is false.

Set this global flag only before deployment. When Patroni SSL is enabled, health checks, metric collection, and API calls must use HTTPS instead of HTTP.

patroni_watchdog_mode

Parameter Name: patroni_watchdog_mode, Type: string, Level: C

Patroni watchdog mode: automatic, required, off, default is off.

If the primary fails, Patroni can use a watchdog to force the old primary node to stop. This prevents split brain.

  • off: Don’t use watchdog. No fencing at all (default behavior)
  • automatic: Enable watchdog if kernel has softdog module enabled and watchdog belongs to dbsu.
  • required: Force enable watchdog, refuse to start Patroni/PostgreSQL if softdog unavailable.

The default is off. Do not enable the watchdog on infrastructure nodes. Consider it for critical systems where data consistency takes priority over availability, especially financial applications.

Normally, split brain is not a risk when all traffic uses HAProxy health-check service access.

patroni_username

Parameter Name: patroni_username, Type: username, Level: C

Patroni REST API username, default is postgres, used with patroni_password.

Patroni protects dangerous REST API operations, such as cluster restart, with an additional username and password. See Configure Cluster and Patroni REST API.

patroni_password

Parameter Name: patroni_password, Type: password, Level: C

Patroni REST API password, default is Patroni.API.

Warning: Must change this parameter in production environments!

pg_primary_db

Parameter Name: pg_primary_db, Type: string, Level: C

This parameter selects the primary database for Citus and other business databases. The default is postgres.

For example, when using Patroni to manage HA Citus clusters, you must choose a “primary database”.

The connection string printed after PGSQL installation also uses this database name.

pg_parameters

Parameter Name: pg_parameters, Type: dict, Level: G/C/I

Used to specify and manage configuration parameters in postgresql.auto.conf.

After all instances initialize, pg_param writes this dictionary’s key-value pairs in order to /pg/data/postgresql.auto.conf.

Do not edit this file manually or change cluster parameters with ALTER SYSTEM. The next configuration synchronization overwrites those changes.

This variable takes priority over cluster configuration in Patroni and DCS, including changes made with Patroni edit-config. Use it to override cluster defaults for an instance.

Cluster members with different hardware specifications are not recommended. If you use them, this parameter provides detailed configuration for each instance.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_parameters: { shared_buffers: '5GB' } }
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_parameters: { shared_buffers: '4GB' } }
    10.10.10.13: { pg_seq: 3, pg_role: replica , pg_parameters: { shared_buffers: '3GB' } }

Patroni sets some important cluster parameters through command-line arguments. These have requirements for primary and replica values and take the highest priority. You cannot override them with this parameter. Manage them with Patroni edit-config.

These PostgreSQL parameters must be consistent on the primary and replicas. Inconsistent values cause replica startup failure:

  • wal_level
  • max_connections
  • max_locks_per_transaction
  • max_worker_processes
  • max_prepared_transactions
  • track_commit_timestamp

For failover, these parameters should also be consistent on the primary and replicas:

  • listen_addresses
  • port
  • cluster_name
  • hot_standby
  • wal_log_hints
  • max_wal_senders
  • max_replication_slots
  • wal_keep_segments
  • wal_keep_size

You can set parameters that do not yet exist, such as extension GUCs that ALTER SYSTEM cannot change. However, invalid values for existing parameters can prevent PostgreSQL startup. Configure them carefully.

pg_files

Parameter Name: pg_files, Type: path[], Level: C

Used to specify a list of files to be copied to the PGDATA directory, default is empty array: []

Pigsty copies these files to {{ pg_data }}. This is mainly used for license files required by commercial PostgreSQL kernels.

Currently, only the Oracle-compatible PolarDB kernel requires license files. For example, put license.lic in files/ and list it in pg_files:

pg_files: [ license.lic ]

pg_conf

Parameter Name: pg_conf, Type: enum, Level: C

Configuration template: {oltp,olap,crit,tiny}.yml, default is oltp.yml.

  • tiny.yml: Optimized for small nodes, VMs, small demos (1-8 cores, 1-16GB)
  • oltp.yml: Optimized for OLTP workloads and latency-sensitive applications (4C8GB+) (default template)
  • olap.yml: Optimized for OLAP workloads and throughput (4C8G+)
  • crit.yml: Optimized for data consistency and critical applications (4C8G+)

The default is oltp.yml. On a small node, the configure script selects tiny.yml.

For a custom template, put it in templates/<mode>.yml. Set this parameter to the template name.

pg_max_conn

Parameter Name: pg_max_conn, Type: int, Level: C

This parameter sets the maximum PostgreSQL connections. Select a value from 50 to 5000, or use auto for the recommended value.

Default is auto, which sets max connections based on pg_conf and pg_default_service_dest.

  • tiny: 100
  • olap: 200
  • oltp: 200 (pgbouncer) / 1000 (postgres)
    • pg_default_service_dest = pgbouncer : 200
    • pg_default_service_dest = postgres : 1000
  • crit: 200 (pgbouncer) / 1000 (postgres)
    • pg_default_service_dest = pgbouncer : 200
    • pg_default_service_dest = postgres : 1000

Values above 5000 are not recommended. For larger values, manually increase the HAProxy service connection limits.

Pgbouncer transaction pooling reduces problems from excessive OLTP connections. Thus, a large connection count is not recommended by default.

For OLAP scenarios, change pg_default_service_dest to postgres to bypass connection pooling.

pg_shared_buffer_ratio

Parameter Name: pg_shared_buffer_ratio, Type: float, Level: C

Postgres shared buffer memory ratio, default is 0.25, normal range is 0.1~0.4.

The default, 0.25, assigns 25% of node memory to PostgreSQL shared buffers. For huge pages, keep this value sufficiently below node_hugepage_ratio.

Values above 0.4 (40%) are usually not recommended, but may help in extreme cases.

Shared buffers are only part of PostgreSQL shared memory. Use show shared_memory_size_in_huge_pages; to calculate the total shared memory.

pg_rto

Parameter Name: pg_rto, Type: enum, Level: C

Recovery Time Objective (RTO) mode controlling Patroni and HAProxy timeout parameters. The default is norm.

Pigsty provides four presets tuned for different network conditions and deployment scenarios:

ModeScenarioNetwork conditionsSource target RTOPatroni TTLFalse-failover risk
fastSame rack or switch< 1 ms, highly stable< 30s20sHigher
normSame datacenter1-5 ms, normal< 45s30sMedium
safeCross-datacenter10-50 ms< 90s60sLower
wideCross-region/continent100-200 ms, public WAN< 150s120sLowest

A shorter RTO speeds recovery but increases false-failure detection from network jitter. Select a mode for your network conditions. See RTO Trade-offs.

The current template recognizes only these four string keys. Other values, including numbers, select norm; they do not specify seconds. For another timeout combination, add a key to pg_rto_plan and use it as pg_rto.

pg_rto: norm   # default, suitable for a single datacenter
pg_rto: safe   # recommended for cross-datacenter deployments
# pg_rto: 30   # invalid key; current templates fall back to norm

pg_rto_plan

Parameter Name: pg_rto_plan, Type: dict, Level: G

Dictionary of RTO presets defining Patroni HA and HAProxy health-check timeouts. The defaults contain four modes:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'500ms'  ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'     ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1500ms' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'     ,'4s' ,3 ,3 ]  # rto < 150s

Each mode is an array of ten values controlling Patroni and HAProxy together:

IndexNameComponentDescription
0ttlPatroniPrimary-lock TTL in seconds
1loop_waitPatroniMain-loop sleep interval
2retry_timeoutPatroniDCS/PostgreSQL retry timeout
3primary_start_timeoutPatroniTime allowed for primary recovery
4safety_marginPatroniWatchdog safety margin
5interHAProxyHealth-check interval
6fastinterHAProxyFast interval during state transitions
7downinterHAProxyCheck interval while a server is down
8riseHAProxyConsecutive successes required for UP
9fallHAProxyConsecutive failures required for DOWN

Override this dictionary to customize an existing mode or add a new one:

pg_rto_plan:
  ultra: [ 10, 2, 3, 8, 2, '500ms', '250ms', '500ms', 2, 2 ]  # aggressive; low-latency networks only

Caution: inappropriate timeout combinations can make the cluster unstable or cause frequent false failovers.

pg_rpo

Parameter Name: pg_rpo, Type: int, Level: C

This parameter sets the failover-candidate lag threshold in bytes. The default is 1048576 (1 MiB). Pigsty writes it to Patroni maximum_lag_on_failover to determine replica eligibility for failover. It does not set an absolute limit on data loss. With asynchronous replication, maximum loss also depends on write rate and when Patroni sampled the primary WAL position.

If the primary fails while all replicas lag, choose between availability and consistency:

  • Promote a lagging replica and restore service as soon as possible, accepting possible data loss.
  • Wait for primary to come back online (may never happen), or manual intervention to avoid any data loss.

You can use the crit.yml conf template to ensure no data loss during failover, but this sacrifices some performance.

pg_libs

Parameter Name: pg_libs, Type: string, Level: C

This parameter lists preloaded dynamic shared libraries. The default is pg_stat_statements,auto_explain. Enabling these two built-in PostgreSQL extensions is strongly recommended.

For existing clusters, you can directly configure cluster shared_preload_libraries parameter and apply.

For TimescaleDB or Citus, add timescaledb or citus at the start of this list. For example:

citus,timescaledb,pg_stat_statements,auto_explain

Add other extensions that require dynamic loading, such as pg_cron and pgml, to this list. Usually, citus and timescaledb take highest priority and must come first.

pg_delay

Parameter Name: pg_delay, Type: interval, Level: I

Delayed standby replication delay, default: 0.

With a positive value, the standby cluster leader waits that interval before applying WAL changes. For example, 1h keeps cluster data one hour behind the original cluster.

See Delayed Standby Cluster for details.

pg_checksum

Parameter Name: pg_checksum, Type: bool, Level: C

Enable data checksum for PostgreSQL cluster? Default is true, enabled.

Set this parameter only before PGSQL deployment. You can still enable checksums manually later.

Data checksums detect disk corruption and hardware failures. Pigsty has enabled them by default since v3.5 to protect data integrity.

pg_pwd_enc

Parameter Name: pg_pwd_enc, Type: enum, Level: C

Password encryption algorithm, fixed to scram-sha-256 since Pigsty v4.

All new users use SCRAM credentials. md5 is deprecated. For older clients, upgrade business connection pools or client drivers to support SCRAM.

pg_encoding

Parameter Name: pg_encoding, Type: enum, Level: C

Database cluster encoding, default is UTF8.

Using other non-UTF8 encodings is not recommended.

pg_locale

Parameter Name: pg_locale, Type: enum, Level: C

Database cluster locale, default is C.

This parameter controls the default database locale, including collation and character classification. C and POSIX give the best performance and predictable sorting.

For language-specific behavior, select a locale such as en_US.UTF-8 or zh_CN.UTF-8. Locale affects index sort order. You cannot change it after cluster initialization.

pg_lc_collate

Parameter Name: pg_lc_collate, Type: enum, Level: C

Database cluster collation, default is C.

Change cluster-level collation only if you understand its effects.

pg_lc_ctype

Parameter Name: pg_lc_ctype, Type: enum, Level: C

Database character set CTYPE, default is C.

Starting from Pigsty v3.5, to be consistent with pg_lc_collate, the default value changed to C.

pg_io_method

Parameter Name: pg_io_method, Type: enum, Level: C

PostgreSQL IO method, default is worker. Available options include:

  • auto: Automatically select based on operating system, uses io_uring on Debian-based systems or EL 10+, otherwise uses worker
  • sync: Use traditional synchronous IO method
  • worker: Use background worker processes to handle IO (default option)
  • io_uring: Use Linux’s io_uring asynchronous IO interface

Current tuning templates write this parameter only for PostgreSQL 18 and later, where it controls the asynchronous I/O execution method.

  • PostgreSQL 18’s actual GUC values are worker, io_uring, and sync; auto is Pigsty template-selection logic, not a PostgreSQL enum value.
  • PostgreSQL 18 defaults to worker, which uses background processes for asynchronous I/O.
  • If you’re using Debian 12/Ubuntu 22+ or EL 10+ systems and want optimal IO performance, consider setting this to io_uring.

If the system does not support io_uring, selecting it can prevent PostgreSQL startup. auto and worker are safer choices.

pg_etcd_password

Parameter Name: pg_etcd_password, Type: password, Level: C

The password used by this PostgreSQL cluster in etcd, default is empty string ''.

With an empty string, Pigsty uses pg_cluster as the password. For Citus clusters, it uses pg_shard.

This password is used for authentication when Patroni connects to etcd and when vip-manager accesses etcd.

pgsodium_key

Parameter Name: pgsodium_key, Type: string, Level: C

The encryption master key for the pgsodium extension, consisting of 64 hexadecimal digits.

This parameter has no default value. If you omit it, Pigsty generates a deterministic key from sha256(pg_cluster).

pgsodium uses libsodium to provide PostgreSQL encryption functions and transparent column encryption. For encryption, explicitly set a secure random key and store it safely.

Example command to generate a random key:

openssl rand -hex 32   # Generate 64-digit hexadecimal key

pgsodium_getkey_script

Parameter Name: pgsodium_getkey_script, Type: path, Level: C

Path to the pgsodium key retrieval script, default uses the pgsodium_getkey script from Pigsty templates.

PostgreSQL runs this script at startup to get the pgsodium master key. The default script reads the key from environment variables or configuration files.

For custom key management, such as HashiCorp Vault or AWS KMS, provide a custom script path.

PG_PROVISION

PG_BOOTSTRAP creates the cluster. PG_PROVISION creates default objects within it:

pg_provision: true                # provision postgres cluster after bootstrap
pg_init: pg-init                  # init script for cluster template, default is `pg-init`
pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
pg_default_privileges:            # default privileges when admin user creates objects
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin
pg_default_schemas: [ monitor ]   # default schemas
pg_default_extensions:            # default extensions
  - { name: pg_stat_statements ,schema: monitor }
  - { name: pgstattuple        ,schema: monitor }
  - { name: pg_buffercache     ,schema: monitor }
  - { name: pageinspect        ,schema: monitor }
  - { name: pg_prewarm         ,schema: monitor }
  - { name: pg_visibility      ,schema: monitor }
  - { name: pg_freespacemap    ,schema: monitor }
  - { name: postgres_fdw       ,schema: public  }
  - { name: file_fdw           ,schema: public  }
  - { name: btree_gist         ,schema: public  }
  - { name: btree_gin          ,schema: public  }
  - { name: pg_trgm            ,schema: public  }
  - { name: intagg             ,schema: public  }
  - { name: intarray           ,schema: public  }
  - { name: pg_repack }
pg_reload: true                   # reload config after HBA changes?
pg_default_hba_rules:             # postgres default HBA rules, ordered by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}
pgb_default_hba_rules:            # pgbouncer default HBA rules, ordered by `order`
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

pg_provision

Parameter Name: pg_provision, Type: bool, Level: C

This parameter enables the provisioning described in this section after cluster initialization. The default is true.

If disabled, Pigsty skips cluster provisioning. Use this for special PostgreSQL variants, such as Greenplum.

pg_init

Parameter Name: pg_init, Type: string, Level: G/C

This parameter selects the shell script for database template initialization. The default is pg-init. Pigsty copies it to /pg/bin/pg-init, then runs it.

This script is located at roles/pgsql/templates/pg-init

Add your own logic to this script, or place a new script in templates/ and set pg_init to its name. Keep the existing initialization logic in a custom script.

pg_default_roles

Parameter Name: pg_default_roles, Type: role[], Level: G/C

Default roles and users in Postgres cluster.

Pigsty has a built-in role system. See PGSQL Access Control: Role System for details.

pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly]               ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite]  ,comment: role for object creation }
  - { name: postgres     ,superuser: true                                          ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly]   ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 , comment: pgsql admin user }
  - { name: dbuser_monitor   ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

pg_default_privileges

Parameter Name: pg_default_privileges, Type: string[], Level: G/C

Default privileges (DEFAULT PRIVILEGE) settings in each database:

pg_default_privileges:            # default privileges when admin user creates objects
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

Pigsty provides matching default privileges for the built-in role system. See PGSQL Access Control: Default Privileges for details.

pg_default_schemas

Parameter Name: pg_default_schemas, Type: string[], Level: G/C

This parameter lists schemas to create. The default, [ monitor ], creates monitor in all databases for monitoring extensions, tables, views, and functions.

pg_default_extensions

Parameter Name: pg_default_extensions, Type: extension[], Level: G/C

List of extensions to be created and enabled by default in all databases, default value:

pg_default_extensions: # default extensions to be created
  - { name: pg_stat_statements ,schema: monitor }
  - { name: pgstattuple        ,schema: monitor }
  - { name: pg_buffercache     ,schema: monitor }
  - { name: pageinspect        ,schema: monitor }
  - { name: pg_prewarm         ,schema: monitor }
  - { name: pg_visibility      ,schema: monitor }
  - { name: pg_freespacemap    ,schema: monitor }
  - { name: postgres_fdw       ,schema: public  }
  - { name: file_fdw           ,schema: public  }
  - { name: btree_gist         ,schema: public  }
  - { name: btree_gin          ,schema: public  }
  - { name: pg_trgm            ,schema: public  }
  - { name: intagg             ,schema: public  }
  - { name: intarray           ,schema: public  }
  - { name: pg_repack }

pg_repack is the only third-party extension in this list. It supports important database maintenance tasks. All other entries are PostgreSQL Contrib extensions.

By default, Pigsty installs monitoring extensions in monitor. pg_default_schemas creates this schema.

pg_reload

Parameter Name: pg_reload, Type: bool, Level: A

Reload PostgreSQL after HBA changes, default value is true.

To inspect HBA changes before applying them, set this parameter to false. This disables automatic configuration reload.

pg_default_hba_rules

Parameter Name: pg_default_hba_rules, Type: hba[], Level: G/C

PostgreSQL host-based authentication rules, global default rules definition. Default value is:

pg_default_hba_rules:             # postgres default host-based authentication rules, ordered by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

The defaults apply to ordinary deployments on a trusted intranet. They are not a hardened baseline for public or regulated environments. The default +dbrole_offline rule has no role field, so it applies to every instance. For instance isolation, copy the complete default list and set this rule to role: offline. See Authentication and Offline Role and Instance Isolation.

This parameter is an array of HBA rule objects, with the same format as pg_hba_rules. Set common pg_default_hba_rules globally. Use pg_hba_rules for cluster-specific additions. Pigsty applies both sets in order, with the latter taking higher priority.

pgb_default_hba_rules

Parameter Name: pgb_default_hba_rules, Type: hba[], Level: G/C

Pgbouncer default host-based authentication rules, array of HBA rule objects.

Default value provides a fair security level for common scenarios. Check PGSQL Authentication for details.

pgb_default_hba_rules:            # pgbouncer default host-based authentication rules, ordered by `order`
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

The default Pgbouncer HBA rules are simple:

  1. Allow login from localhost with password
  2. Allow login from intranet with password

Users can customize according to their own needs.

This parameter uses the same format as pgb_hba_rules. Set common pgb_default_hba_rules globally. Use pgb_hba_rules for cluster-specific additions. Pigsty applies both sets in order, with the latter taking higher priority.


PG_BACKUP

This section defines variables for pgBackRest, which is used for PGSQL Point-in-Time Recovery (PITR).

Check PGSQL Backup & PITR for detailed information.

pgbackrest_enabled: true          # enable pgBackRest on pgsql host?
pgbackrest_log_dir: /pg/log/pgbackrest # pgbackrest log dir, default is `/pg/log/pgbackrest`
pgbackrest_method: local          # pgbackrest repo method: local, minio, [user defined...]
pgbackrest_init_backup: true      # perform a full backup immediately after pgbackrest init?
pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix filesystem
    path: /pg/backup              # local backup directory, default is `/pg/backup`
    retention_full_type: count    # retain full backup by count
    retention_full: 2             # keep at most 3 full backups when using local filesystem repo, at least 2
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so use s3
    s3_endpoint: sss.pigsty       # minio endpoint domain, default is `sss.pigsty`
    s3_region: us-east-1          # minio region, default is us-east-1, not effective for minio
    s3_bucket: pgsql              # minio bucket name, default is `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio, instead of host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, default is 9000
    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, default is `/etc/pki/ca.crt`
    block: y                      # enable block-level incremental backup (pgBackRest 2.46+)
    bundle: y                     # bundle small files into one file
    bundle_limit: 20MiB           # object storage file bundling threshold, default 20MiB
    bundle_size: 128MiB           # object storage file bundling target size, default 128MiB
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backup by time on minio repo
    retention_full: 14            # keep full backups from the past 14 days

pgbackrest_enabled

Parameter Name: pgbackrest_enabled, Type: bool, Level: C

Enable pgBackRest on PGSQL nodes? Default value is: true

When enabled, Pigsty configures pgBackRest on every node. With the local filesystem repository, each member creates its own local stanza. Initial and scheduled backups run only on the current primary. On a replica, pg-backup exits after checking its role. Pigsty initializes a non-local shared stanza only on a primary without pg_upstream.

pgbackrest_log_dir

Parameter Name: pgbackrest_log_dir, Type: path, Level: C

pgBackRest log directory, default is /pg/log/pgbackrest. The Vector log agent references this parameter for log collection.

pgbackrest_method

Parameter Name: pgbackrest_method, Type: enum, Level: C

pgBackRest repository method: default options are local, minio, or other user-defined methods, default is local.

This parameter determines which repository to use for pgBackRest. All available repository methods are defined in pgbackrest_repo.

By default, Pigsty creates a local backup repository at /pg/backup on the primary instance. pg_fs_backup sets its physical storage path.

pgbackrest_init_backup

Parameter Name: pgbackrest_init_backup, Type: bool, Level: C

Perform a full backup immediately after pgBackRest initialization completes? Default is true.

The task runs only on the primary and only when pg_upstream is undefined. It uses ignore_errors, so enabling it does not guarantee that a base backup exists. It writes /etc/pgbackrest/initial.done only after a successful backup. Afterward, verify the repository with pig pb info or pb info.

pgbackrest_repo

Parameter Name: pgbackrest_repo, Type: dict, Level: G/C

pgBackRest repository documentation: https://pgbackrest.org/configuration.html#section-repository

The default dictionary contains local and minio candidates. pgbackrest_method selects one. The v4.5.0 template renders only the selected entry as repo1. Listing both keys does not configure backups to two repositories:

pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix filesystem
    path: /pg/backup              # local backup directory, default is `/pg/backup`
    retention_full_type: count    # retain full backup by count
    retention_full: 2             # keep at most 3 full backups when using local filesystem repo, at least 2
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so use s3
    s3_endpoint: sss.pigsty       # minio endpoint domain, default is `sss.pigsty`
    s3_region: us-east-1          # minio region, default is us-east-1, not effective for minio
    s3_bucket: pgsql              # minio bucket name, default is `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio, instead of host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, default is 9000
    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, default is `/etc/pki/ca.crt`
    block: y                      # enable block-level incremental backup (pgBackRest 2.46+)
    bundle: y                     # bundle small files into one file
    bundle_limit: 20MiB           # object storage file bundling threshold, default 20MiB
    bundle_size: 128MiB           # object storage file bundling target size, default 128MiB
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backup by time on minio repo
    retention_full: 14            # keep full backups from the past 14 days

You can define new backup repositories, such as using AWS S3, GCP, or other cloud providers’ S3-compatible storage services.

Block Incremental Backup: Since pgBackRest 2.46, block: y enables incremental backups at block level. An incremental backup then copies changed blocks, rather than complete changed files. This substantially reduces backup size and duration, especially for large databases. Enable it for object storage repositories when possible.


PG_ACCESS

This section handles database access paths, including:

  • Deploy Pgbouncer connection pooler on each PGSQL node and set default behavior
  • Publish service ports through local or dedicated haproxy nodes
  • Bind optional L2 VIP and register DNS records
pgbouncer_enabled: true           # if disabled, pgbouncer will not be launched on pgsql host
pgbouncer_port: 6432              # pgbouncer listen port, 6432 by default
pgbouncer_log_dir: /pg/log/pgbouncer  # pgbouncer log dir, `/pg/log/pgbouncer` by default
pgbouncer_auth_query: false       # query postgres to retrieve unlisted business users?
pgbouncer_poolmode: transaction   # pooling mode: transaction,session,statement, transaction by default
pgbouncer_sslmode: disable        # pgbouncer client ssl mode, disable by default
pgbouncer_ignore_param: [ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]
pg_weight: 100          #INSTANCE # relative load balance weight in service, 100 by default, 0-255
pg_service_provider: ''           # dedicate haproxy node group name, or empty string for local nodes by default
pg_default_service_dest: pgbouncer # default service destination if svc.dest='default'
pg_default_services:              # postgres default service definitions
  - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}
pg_vip_enabled: false             # enable a l2 vip for pgsql primary? false by default
pg_vip_address: 127.0.0.1/24      # vip address in `<ipv4>/<mask>` format, require if vip is enabled
pg_vip_interface: auto            # vip network interface to listen, auto by default
pg_dns_suffix: ''                 # pgsql dns suffix, '' by default
pg_dns_target: auto               # auto, primary, vip, none, or ad hoc ip

pgbouncer_enabled

Parameter Name: pgbouncer_enabled, Type: bool, Level: C

Default value is true. If disabled, the Pgbouncer connection pooler will not be configured on PGSQL nodes.

pgbouncer_port

Parameter Name: pgbouncer_port, Type: port, Level: C

Pgbouncer listen port, default is 6432.

pgbouncer_log_dir

Parameter Name: pgbouncer_log_dir, Type: path, Level: C

Pgbouncer log directory, default is /pg/log/pgbouncer. The Vector log agent collects Pgbouncer logs based on this parameter.

pgbouncer_auth_query

Parameter Name: pgbouncer_auth_query, Type: bool, Level: C

This parameter lets Pgbouncer query PostgreSQL to authenticate users that are not explicitly listed for pool access. The default is false.

If enabled, Pgbouncer authenticates users through PostgreSQL with SELECT username, password FROM monitor.pgbouncer_auth($1). Otherwise, only business users with pgbouncer: true can connect through the pool.

pgbouncer_poolmode

Parameter Name: pgbouncer_poolmode, Type: enum, Level: C

Pgbouncer connection pool pooling mode: transaction, session, statement, default is transaction.

  • session: Session-level pooling with best feature compatibility.
  • transaction: Transaction-level pooling with better performance (many small connections), may break some session-level features like NOTIFY/LISTEN, etc.
  • statements: Statement-level pooling for simple read-only queries.

If your application has feature compatibility issues, consider changing this parameter to session.

pgbouncer_sslmode

Parameter Name: pgbouncer_sslmode, Type: enum, Level: C

Pgbouncer client SSL mode, default is disable.

SSL can significantly affect Pgbouncer performance.

  • disable: Ignore if client requests TLS (default)
  • allow: Use TLS if client requests it. Use plain TCP if not. Does not verify client certificate.
  • prefer: Same as allow.
  • require: Client must use TLS. Reject client connection if not. Does not verify client certificate.
  • verify-ca: Client must use TLS with a valid client certificate.
  • verify-full: Same as verify-ca.

pgbouncer_ignore_param

Parameter Name: pgbouncer_ignore_param, Type: string[], Level: C

List of startup parameters ignored by PgBouncer, default value is:

[ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]

Pigsty writes this list to PgBouncer ignore_startup_parameters. If clients set these parameters at connection time, PgBouncer does not create connections to resolve differences in their values.

Thus, clients can share a pool even with different values for these parameters. Pigsty v3.5 added this parameter.


pg_weight

Parameter Name: pg_weight, Type: int, Level: I

Relative load balancing weight in service, default is 100, range 0-255.

The default is 100. Set it in instance variables, then reload services to apply the value.

pg_service_provider

Parameter Name: pg_service_provider, Type: string, Level: G/C

Dedicated haproxy node group name, or empty string for local nodes by default.

If specified, Pigsty registers PostgreSQL services on the dedicated HAProxy group. Otherwise, it uses the current PGSQL cluster nodes.

Assign a unique port to each service on the dedicated HAProxy nodes.

For example, if we define the following parameters on a 3-node pg-test cluster:

pg_service_provider: infra       # use load balancer on group `infra`
pg_default_services:             # alloc port 10001 and 10002 for pg-test primary/replica service
  - { name: primary ,port: 10001 ,dest: postgres  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 10002 ,dest: postgres  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }

pg_default_service_dest

Parameter Name: pg_default_service_dest, Type: enum, Level: G/C

If a service definition uses svc.dest='default', this parameter selects its destination.

The default is pgbouncer. Thus, the primary service on port 5433 and the replica service on port 5434 route traffic to Pgbouncer.

To bypass Pgbouncer, set this parameter to postgres. Traffic then goes directly to PostgreSQL.

pg_default_services

Parameter Name: pg_default_services, Type: service[], Level: G/C

Postgres default service definitions.

Default value is four default service definitions, as described in PGSQL Service.

pg_default_services:               # postgres default service definitions
  - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

pg_vip_enabled

Parameter Name: pg_vip_enabled, Type: bool, Level: C

This parameter enables an L2 VIP for the PGSQL cluster. The default is false, which creates no VIP.

When enabled, vip-manager uses etcd data to bind the VIP to the current primary node.

An L2 VIP works only within one L2 network. This can impose additional network topology requirements.

pg_vip_address

Parameter Name: pg_vip_address, Type: cidr4, Level: C

VIP address in <ipv4>/<mask> format is required if VIP is enabled.

Default value: 127.0.0.1/24. This value consists of two parts: ipv4 and mask, separated by /.

pg_vip_interface

Parameter Name: pg_vip_interface, Type: string, Level: C/I

This parameter selects the VIP network interface. The default, auto, detects the interface associated with the inventory instance IP.

For unusual networks where automatic detection is unsuitable, set the interface explicitly in instance variables. Examples include non-standard and policy routing:

pg-test:
    hosts:
        10.10.10.11: {pg_seq: 1, pg_role: replica ,pg_vip_interface: eth0 }
        10.10.10.12: {pg_seq: 2, pg_role: primary ,pg_vip_interface: eth1 }
        10.10.10.13: {pg_seq: 3, pg_role: replica ,pg_vip_interface: eth2 }
    vars:
      pg_vip_enabled: true          # enable L2 VIP for this cluster, binds to primary by default
      pg_vip_address: 10.10.10.3/24 # L2 network CIDR: 10.10.10.0/24, vip address: 10.10.10.3
      # pg_vip_interface: eth1      # if your nodes have a unified interface, you can define it here

pg_dns_suffix

Parameter Name: pg_dns_suffix, Type: string, Level: C

PostgreSQL DNS name suffix, default is empty string.

By default, the PostgreSQL cluster name is registered as a DNS domain in dnsmasq on Infra nodes for external resolution.

Set this parameter to add a domain suffix. The cluster DNS name becomes {{ pg_cluster }}{{ pg_dns_suffix }}.

For example, if you set pg_dns_suffix to .db.vip.company.tld, the pg-test cluster DNS name will be pg-test.db.vip.company.tld.

pg_dns_target

Parameter Name: pg_dns_target, Type: enum, Level: C

This parameter selects the target IP for the cluster DNS record. Use auto, primary, vip, none, or a specific IP address.

The default is auto. If pg_vip_enabled is enabled, it selects pg_vip_address. Otherwise, it selects the primary instance IP.

  • vip: bind to pg_vip_address
  • primary: resolve to cluster primary instance IP address
  • auto: resolve to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance IP address
  • none: do not bind to any IP address
  • <ipv4>: bind to the given IP address

PG_MONITOR

PG_MONITOR configures monitoring for PostgreSQL databases, Pgbouncer pools, and pgBackRest backups.

The group configures three exporters:

  • pg_exporter: PostgreSQL monitoring.
  • pgbouncer_exporter: Connection pool monitoring.
  • pgbackrest_exporter: Backup status monitoring.
pg_exporter_enabled: true              # enable pg_exporter on pgsql host?
pg_exporter_config: pg_exporter.yml    # pg_exporter config file name
pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector ttl stages (seconds), default is '1,10,60,300'
pg_exporter_port: 9630                 # pg_exporter listen port, default is 9630
pg_exporter_params: 'sslmode=disable'  # extra url parameters for pg_exporter dsn
pg_exporter_url: ''                    # if specified, will override auto-generated pg dsn
pg_exporter_auto_discovery: true       # enable auto database discovery? enabled by default
pg_exporter_exclude_database: 'template0,template1,postgres' # csv list of databases not monitored during auto-discovery
pg_exporter_include_database: ''       # csv list of databases monitored during auto-discovery
pg_exporter_connect_timeout: 200       # pg_exporter connection timeout (ms), default is 200
pg_exporter_options: ''                # extra options to override pg_exporter
pgbouncer_exporter_enabled: true       # enable pgbouncer_exporter on pgsql host?
pgbouncer_exporter_port: 9631          # pgbouncer_exporter listen port, default is 9631
pgbouncer_exporter_url: ''             # if specified, will override auto-generated pgbouncer dsn
pgbouncer_exporter_options: ''         # extra options to override pgbouncer_exporter
pgbackrest_exporter_enabled: true      # enable pgbackrest_exporter on pgsql host?
pgbackrest_exporter_port: 9854         # pgbackrest_exporter listen port, default is 9854
pgbackrest_exporter_options: >-        # extra options to override pgbackrest_exporter
  --collect.interval=120
  --log.level=info

pg_exporter_enabled

Parameter Name: pg_exporter_enabled, Type: bool, Level: C

Enable pg_exporter on PGSQL nodes? Default value is: true.

PG Exporter monitors PostgreSQL instances. Set this parameter to false to skip its installation.

pg_exporter_config

Parameter Name: pg_exporter_config, Type: string, Level: C

This parameter selects the pg_exporter collector template. The default is pg_exporter.yml. It does not affect Pgbouncer Exporter, which uses the fixed pgbouncer_exporter.yml template.

The default template is roles/pg_monitor/templates/pg_exporter.yml. To use a custom file, put it on the Ansible template search path and set its template name here.

pg_exporter_cache_ttls

Parameter Name: pg_exporter_cache_ttls, Type: string, Level: C

pg_exporter collector TTL stages (seconds), default is ‘1,10,60,300’.

The default, 1,10,60,300, assigns collector TTLs of 1s, 10s, 60s, and 300s.

PG Exporter caches results to reduce database load from repeated Prometheus scrapes. Collectors use four TTL categories:

ttl_fast: "{{ pg_exporter_cache_ttls.split(',')[0]|int }}"         # critical queries
ttl_norm: "{{ pg_exporter_cache_ttls.split(',')[1]|int }}"         # common queries
ttl_slow: "{{ pg_exporter_cache_ttls.split(',')[2]|int }}"         # slow queries (e.g table size)
ttl_slowest: "{{ pg_exporter_cache_ttls.split(',')[3]|int }}"      # ver slow queries (e.g bloat)

With the default configuration, liveness metrics remain cached for at most 1s. Most common metrics have a 10s TTL, which should match victoria_scrape_interval. A few queries with slowly changing results have a 60s TTL. A small number of expensive monitoring queries have a 300s TTL.

pg_exporter_port

Parameter Name: pg_exporter_port, Type: port, Level: C

pg_exporter listen port, default value is: 9630

pg_exporter_params

Parameter Name: pg_exporter_params, Type: string, Level: C

Extra URL path parameters in the DSN used by pg_exporter.

Default value: sslmode=disable, which disables SSL for monitoring connections (since local unix sockets are used by default).

pg_exporter_url

Parameter Name: pg_exporter_url, Type: pgurl, Level: C

This parameter overrides the generated PostgreSQL DSN. PG Exporter then uses the specified DSN to connect. The default is an empty string.

If not specified, PG Exporter will use the following connection string to access PostgreSQL by default:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@{{ pg_host }}:{{ pg_port }}/postgres{% if pg_exporter_params != '' %}?{{ pg_exporter_params }}{% endif %}

Use this parameter for a remote PostgreSQL instance, different monitoring credentials, or different connection options.

pg_exporter_auto_discovery

Parameter Name: pg_exporter_auto_discovery, Type: bool, Level: C

Enable auto database discovery? Enabled by default: true.

By default, PG Exporter collects global metrics from the DSN database, normally the postgres administration database. Enable this option to collect metrics from all business databases. PG Exporter then discovers databases in the instance and collects their database-level metrics.

pg_exporter_exclude_database

Parameter Name: pg_exporter_exclude_database, Type: string, Level: C

When automatic discovery is enabled, PG Exporter excludes databases in this list. Automatic discovery is enabled by default. The default list, template0,template1,postgres, excludes the template databases and the postgres administration database.

The DSN database is an exception. For example, if PG Exporter connects to postgres, it monitors that database even when postgres is in this list.

pg_exporter_include_database

Parameter Name: pg_exporter_include_database, Type: string, Level: C

When automatic discovery is enabled, PG Exporter monitors only databases in this list. Automatic discovery is enabled by default. The default is an empty string, which disables this restriction.

The parameter format is a comma-separated list of database names, e.g., db1,db2,db3.

This parameter takes priority over pg_exporter_exclude_database. Use it to allow monitoring only for specific databases.

pg_exporter_connect_timeout

Parameter Name: pg_exporter_connect_timeout, Type: int, Level: C

pg_exporter connection timeout (milliseconds), default is 200 (in milliseconds).

This value limits how long PG Exporter waits to connect to PostgreSQL. When the timeout expires, it stops the connection attempt and reports an error.

The default 200ms is sufficient for most uses, such as monitoring within one availability zone. For a remote instance on another continent, increase it if necessary to avoid connection timeouts.

pg_exporter_options

Parameter Name: pg_exporter_options, Type: arg, Level: C

Command line arguments passed to PG Exporter, default value is: "" empty string.

When using empty string, the default command arguments will be used:

{% if pg_exporter_options != '' %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pg_exporter_port }} {{ pg_exporter_options }}'
{% else %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pg_exporter_port }} --log.level=info'
{% endif %}

Note: Do not override the pg_exporter_port port configuration in this parameter.

pgbouncer_exporter_enabled

Parameter Name: pgbouncer_exporter_enabled, Type: bool, Level: C

Enable pgbouncer_exporter on PGSQL nodes? Default value is: true.

pgbouncer_exporter_port

Parameter Name: pgbouncer_exporter_port, Type: port, Level: C

pgbouncer_exporter listen port, default value is: 9631

pgbouncer_exporter_url

Parameter Name: pgbouncer_exporter_url, Type: pgurl, Level: C

This parameter overrides the generated Pgbouncer DSN. Pgbouncer Exporter then uses the specified DSN to connect. The default is an empty string.

If not specified, Pgbouncer Exporter will use the following connection string to access Pgbouncer by default:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@:{{ pgbouncer_port }}/pgbouncer?host={{ pg_localhost }}&sslmode=disable

Use this parameter for a remote Pgbouncer instance, different monitoring credentials, or different connection options.

pgbouncer_exporter_options

Parameter Name: pgbouncer_exporter_options, Type: arg, Level: C

Command line arguments passed to Pgbouncer Exporter, default value is: "" empty string.

When using empty string, the default command arguments will be used:

{% if pgbouncer_exporter_options != '' %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pgbouncer_exporter_port }} {{ pgbouncer_exporter_options }}'
{% else %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pgbouncer_exporter_port }} --log.level=info'
{% endif %}

Note: Do not override the pgbouncer_exporter_port port configuration in this parameter.

pgbackrest_exporter_enabled

Parameter Name: pgbackrest_exporter_enabled, Type: bool, Level: C

Enable pgbackrest_exporter on PGSQL nodes? Default value is: true.

pgbackrest_exporter monitors the pgBackRest backup system. Key metrics include backup size, time, type, and duration.

pgbackrest_exporter_port

Parameter Name: pgbackrest_exporter_port, Type: port, Level: C

pgbackrest_exporter listen port, default value is: 9854.

This port is registered in the VictoriaMetrics-compatible scrape targets for backup-related metrics.

pgbackrest_exporter_options

Parameter Name: pgbackrest_exporter_options, Type: arg, Level: C

Command-line arguments passed to pgbackrest_exporter. The default is:

pgbackrest_exporter_options: >-
  --collect.interval=120
  --log.level=info

The default collects metrics every 120 seconds with info logging. A custom value replaces the complete default argument set.


PG_REMOVE

pgsql-rm.yml uses the pg_remove role to remove PostgreSQL instances safely. These parameters control cleanup and help prevent accidental deletion.

pg_rm_data: true                  # remove postgres data during remove? true by default
pg_rm_backup: true                # remove pgbackrest backup during primary remove? true by default
pg_rm_pkg: true                   # uninstall postgres packages during remove? true by default
pg_safeguard: false               # stop pg_remove running if pg_safeguard is enabled, false by default

pg_rm_data

Parameter Name: pg_rm_data, Type: bool, Level: G/C/A

Whether to clean up pg_data and symlinks when removing PGSQL instances, default is true.

This switch applies to pgsql-rm.yml and other operations that call pg_remove. Set it to false to keep the data directory for manual inspection or remounting.

pg_rm_backup

Parameter Name: pg_rm_backup, Type: bool, Level: G/C/A

Whether to also clean up the pgBackRest repository and configuration when removing the primary, default is true.

This parameter applies only to instances with pg_role=primary. pg_remove first stops pgBackRest, then deletes the current cluster stanza. If pgbackrest_method == 'local', it also removes data in pg_fs_backup. It does not affect standby clusters or upstream backups.

pg_rm_pkg

Parameter Name: pg_rm_pkg, Type: bool, Level: G/C/A

Whether to uninstall all packages installed by pg_packages when cleaning up PGSQL instances, default is true.

To stop temporarily and keep the binaries, set this parameter to false. Otherwise, pg_remove uses the system package manager to uninstall PostgreSQL components completely.

pg_safeguard

Parameter Name: pg_safeguard, Type: bool, Level: G/C/A

This parameter protects against accidental deletion. The default is false. When it is true, pg_remove stops immediately and shows a message. To continue, use -e pg_safeguard=false or disable it in the variables.

Before batch cleanup in production, enable this switch. Verify the commands and target nodes, then disable it for removal. This helps prevent accidental instance deletion.

12 - Playbook

How to manage PostgreSQL clusters with Ansible playbooks

Pigsty provides playbooks for cluster provisioning, scaling, user and database management, monitoring, backup, recovery, and migration.

PlaybookFunction
pgsql.ymlInitialize PostgreSQL cluster or add new replicas
pgsql-rm.ymlRemove PostgreSQL cluster or specific instances
pgsql-user.ymlAdd new business user to existing PostgreSQL cluster
pgsql-db.ymlAdd new business database to existing PostgreSQL cluster
pgsql-monitor.ymlMonitor remote PostgreSQL instances
pgsql-migration.ymlGenerate migration manual and scripts for existing PostgreSQL
pgsql-pitr.ymlPerform Point-In-Time Recovery (PITR)

Safeguard

Use PGSQL playbooks with care. Incorrect use of pgsql.yml or pgsql-rm.yml can accidentally delete databases.

  • Always use -l to limit execution scope. Verify that the tasks and targets are correct.
  • Limit execution to one cluster when possible. Running pgsql.yml without parameters in production is a high-risk operation. Review it carefully.
  • Before removal, inspect pig pg list <cluster> and pig pb info. Verify a recent backup and have the operator enter the exact target.

The PGSQL module uses pg_safeguard to prevent accidental deletion. When pg_safeguard is true, pgsql-rm.yml stops immediately to protect the cluster.

# Will abort execution, protecting data
./pgsql-rm.yml -l pg-test -e pg_safeguard=true

# Override the safeguard switch from the command line
./pgsql-rm.yml -l pg-test -e pg_safeguard=false

In addition to pg_safeguard, pgsql-rm.yml provides finer-grained control parameters:

ParameterDefaultDescription
pg_safeguardfalseSafeguard switch; when true, playbook aborts
pg_rm_datatrueWhether to remove PostgreSQL data directory
pg_rm_backuptrueWhether to remove pgBackRest backup data (only when removing primary)
pg_rm_pkgtrueWhether to uninstall PostgreSQL packages

These parameters allow precise control over removal behavior:

# Remove the instance services, monitoring, and DCS registration, but keep its data directory
./pgsql-rm.yml -l pg-test -e pg_rm_data=false

# Remove cluster but keep backup data
./pgsql-rm.yml -l pg-test -e pg_rm_backup=false

# Remove cluster and uninstall packages
./pgsql-rm.yml -l pg-test -e pg_rm_pkg=true

pgsql.yml

Use pgsql.yml to initialize PostgreSQL clusters or add replicas.

Here’s a demo of initializing a PostgreSQL cluster in the sandbox environment:

asciicast

Basic Usage

./pgsql.yml -l pg-meta            # Initialize cluster pg-meta
./pgsql.yml -l 10.10.10.13        # Initialize/add instance 10.10.10.13
./pgsql.yml -l pg-test -t pg_service  # Refresh services for cluster pg-test
./pgsql.yml -l pg-test -t pg_hba,pgbouncer_hba,pgbouncer_reload -e pg_reload=true  # Reload HBA rules

Wrapper Scripts

Pigsty provides wrapper scripts for common operations:

bin/pgsql-add pg-meta             # Initialize pgsql cluster pg-meta
bin/pgsql-add 10.10.10.10         # Initialize pgsql instance 10.10.10.10
bin/pgsql-add pg-test 10.10.10.13 # Add 10.10.10.13 to cluster pg-test (auto refresh services)
bin/pgsql-svc pg-test             # Refresh haproxy services for pg-test (use after membership changes)
bin/pgsql-hba pg-test             # Reload pg/pgb HBA rules for pg-test

Subtasks

This playbook contains the following subtasks:

# pg_install              : install postgres packages & extensions
#   - pg_dbsu             : setup postgres superuser
#     - pg_dbsu_create    : create dbsu user
#     - pg_dbsu_sudo      : configure dbsu sudo privileges
#     - pg_ssh            : exchange dbsu SSH keys
#   - pg_pkg              : install postgres packages
#     - pg_pre            : pre-installation tasks
#     - pg_ext            : install postgres extension packages
#     - pg_post           : post-installation tasks
#   - pg_link             : link pgsql version bin to /usr/pgsql
#   - pg_path             : add pgsql bin to system path
#   - pg_dir              : create postgres directories and setup FHS
#   - pg_bin              : sync /pg/bin scripts
#   - pg_alias            : configure pgsql/psql aliases
#   - pg_dummy            : create dummy placeholder file
#
# pg_bootstrap            : bootstrap postgres cluster
#   - pg_config           : generate postgres config
#     - pg_conf           : generate patroni config
#     - pg_key            : generate pgsodium key
#   - pg_cert             : issue certificates for postgres
#     - pg_cert_private   : check pg private key existence
#     - pg_cert_issue     : sign pg server certificate
#     - pg_cert_copy      : copy key & certs to pg node
#   - pg_launch           : launch patroni primary & replicas
#     - pg_watchdog       : grant watchdog permission to postgres
#     - pg_primary        : launch patroni/postgres primary
#     - pg_init           : init pg cluster with roles/templates
#     - pg_pass           : write .pgpass file to pg home
#     - pg_replica        : launch patroni/postgres replicas
#     - pg_hba            : generate pg HBA rules
#     - patroni_reload    : reload patroni config
#     - pg_patroni        : pause or remove patroni if necessary
#
# pg_provision            : provision postgres business users & databases
#   - pg_user             : provision postgres business users
#     - pg_user_config    : render create user SQL
#     - pg_user_create    : create user on postgres
#   - pg_db               : provision postgres business databases
#     - pg_db_drop        : drop database on postgres (state=absent/recreate)
#     - pg_db_config      : render create database SQL
#     - pg_db_create      : create database on postgres
#
# pg_backup               : init postgres PITR backup
#   - pgbackrest          : setup pgbackrest for backup
#     - pgbackrest_config : generate pgbackrest config
#     - pgbackrest_init   : init pgbackrest repo
#     - pgbackrest_backup : make initial backup after bootstrap
#
# pg_access               : init postgres service access layer
#   - pgbouncer           : deploy pgbouncer connection pooler
#     - pgbouncer_dir     : create pgbouncer directories
#     - pgbouncer_config  : generate pgbouncer config
#       - pgbouncer_hba   : generate pgbouncer HBA config
#       - pgbouncer_user  : generate pgbouncer userlist
#     - pgbouncer_launch  : launch pgbouncer service
#     - pgbouncer_reload  : reload pgbouncer config
#   - pg_vip              : bind VIP to primary with vip-manager
#     - pg_vip_config     : generate vip-manager config
#     - pg_vip_launch     : launch vip-manager to bind VIP
#   - pg_dns              : register DNS name to infra dnsmasq
#     - pg_dns_ins        : register pg instance name
#     - pg_dns_cls        : register pg cluster name
#   - pg_service          : expose pgsql service with haproxy
#     - pg_service_config : generate local haproxy config for pg services
#     - pg_service_reload : expose postgres services with haproxy
#
# pg_monitor              : setup pgsql monitoring and register to infra
#   - pg_exporter         : configure and launch pg_exporter
#   - pgbouncer_exporter  : configure and launch pgbouncer_exporter
#   - pgbackrest_exporter : configure and launch pgbackrest_exporter
#   - pg_register         : register pgsql to monitoring/logging/datasource
#     - add_metrics       : register pg as VictoriaMetrics monitoring target
#     - add_logs          : register pg as Vector log source
#     - add_ds            : register pg database as Grafana datasource

Related Administration Tasks

Notes

  • Before you run this playbook on one replica, verify that the cluster primary is initialized.
  • After you scale out, reload services and reload HBA. The bin/pgsql-add wrapper does this automatically.

During cluster expansion, the Ansible playbook may time out while Patroni creates a replica.

  • The wait for postgres/patroni replica task typically runs for a long time before it stops.
  • Replica creation continues in the background. For creation that takes more than a day, see the replica creation failure entry in FAQ.

pgsql-rm.yml

Use pgsql-rm.yml to remove PostgreSQL clusters or specific instances.

Here’s a demo of removing a PostgreSQL cluster in the sandbox environment:

asciicast

Basic Usage

./pgsql-rm.yml -l pg-test          # Remove cluster pg-test
./pgsql-rm.yml -l 10.10.10.13      # Remove instance 10.10.10.13

Command Line Arguments

This playbook supports the following command line arguments:

./pgsql-rm.yml -l pg-test          # Remove cluster pg-test
    -e pg_safeguard=false          # Safeguard switch, disabled by default; override when enabled
    -e pg_rm_data=true             # Whether to remove PostgreSQL data directory, default: remove
    -e pg_rm_backup=true           # Whether to remove pgBackRest backup (primary only), default: remove
    -e pg_rm_pkg=true              # Whether to uninstall PostgreSQL packages, default: uninstall

Wrapper Scripts

bin/pgsql-rm pg-meta               # Remove pgsql cluster pg-meta
bin/pgsql-rm pg-test 10.10.10.13   # Remove instance 10.10.10.13 from cluster pg-test

Subtasks

This playbook contains the following subtasks:

# pg_safeguard           : abort if pg_safeguard is enabled
#
# pg_monitor             : remove registration from monitoring system
#   - pg_deregister      : remove pg monitoring targets from infra
#     - rm_metrics       : remove targets from the VictoriaMetrics target directory
#     - rm_ds            : remove datasource from grafana
#     - rm_logs          : remove log targets from vector
#   - pg_exporter        : remove pg_exporter
#   - pgbouncer_exporter : remove pgbouncer_exporter
#   - pgbackrest_exporter: remove pgbackrest_exporter
#
# pg_access              : remove pg service access layer
#   - dns                : remove pg DNS records
#   - vip                : remove vip-manager
#   - pg_service         : remove pg service from haproxy
#   - pgbouncer          : remove pgbouncer connection middleware
#
# postgres               : remove postgres instances
#   - pg_replica         : remove all replicas
#   - pg_primary         : remove primary
#   - pg_meta            : remove metadata from etcd
#
# pg_backup              : remove backup repo (disable with pg_rm_backup=false)
# pg_data                : remove postgres data (disable with pg_rm_data=false)
# pg_pkg                 : uninstall pg packages (enabled by default; disable with pg_rm_pkg=false)
#   - pg_ext             : uninstall postgres extensions alone

Related Administration Tasks

Notes

  • Do not remove a primary while replicas remain. The remaining replicas will trigger automatic failover. Remove all replicas before the primary. Whole-cluster removal already handles this order.
  • Refresh services after instance removal. A removed replica remains in the load balancer configuration, but failed health checks prevent it from affecting services. Reload services at an appropriate time to keep the production environment consistent with the inventory.

pgsql-user.yml

Use pgsql-user.yml to add business users to an existing PostgreSQL cluster.

Basic Usage

./pgsql-user.yml -l pg-meta -e username=dbuser_meta

Wrapper Scripts

bin/pgsql-user pg-meta dbuser_meta  # Create user dbuser_meta on cluster pg-meta

Workflow

  1. Define user in the config inventory: all.children.<pg_cluster>.vars.pg_users[i]
  2. Execute playbook specifying cluster and username: pgsql-user.yml -l <pg_cluster> -e username=<name>

The playbook will:

  1. Generate user creation SQL at /pg/tmp/pg-user-{{ user.name }}.sql
  2. Execute user creation/update SQL on the cluster primary
  3. If pgbouncer_enabled: true, update /etc/pgbouncer/userlist.txt and useropts.txt
  4. Reload pgbouncer to apply configuration

User Definition Example

pg_users:
  - name: dbuser_meta               # Required, username is the only mandatory field
    password: DBUser.Meta           # Optional, can be scram-sha-256 hash or plaintext
    login: true                     # Optional, can login, default: true
    superuser: false                # Optional, is superuser, default: false
    createdb: false                 # Optional, can create database, default: false
    createrole: false               # Optional, can create role, default: false
    inherit: true                   # Optional, inherit privileges, default: true
    replication: false              # Optional, can replicate, default: false
    bypassrls: false                # Optional, bypass RLS, default: false
    pgbouncer: true                 # Optional, add to pgbouncer userlist, default: false
    connlimit: -1                   # Optional, connection limit, -1 means unlimited
    expire_in: 3650                 # Optional, expire in N days (overrides expire_at)
    expire_at: '2030-12-31'         # Optional, specify expiration date
    comment: pigsty admin user      # Optional, user comment
    roles: [dbrole_admin]           # Optional, roles to grant
    parameters: {}                  # Optional, role-level parameters
    pool_mode: transaction          # Optional, pgbouncer user-level pool mode
    pool_connlimit: 100             # Optional, user-level max connections; omitted values inherit global default 100

For details, see: Admin SOP: Create User


pgsql-db.yml

Use pgsql-db.yml to add business databases to an existing PostgreSQL cluster.

Basic Usage

./pgsql-db.yml -l pg-meta -e dbname=meta

Wrapper Scripts

bin/pgsql-db pg-meta meta  # Create database meta on cluster pg-meta

Workflow

  1. Define database in the config inventory: all.children.<pg_cluster>.vars.pg_databases[i]
  2. Execute playbook specifying cluster and database name: pgsql-db.yml -l <pg_cluster> -e dbname=<name>

The playbook will:

  1. Generate database creation SQL at /pg/tmp/pg-db-{{ database.name }}.sql
  2. Execute database creation/update SQL on the cluster primary
  3. If db.register_datasource is true, register database as Grafana datasource
  4. Update /etc/pgbouncer/database.txt and reload pgbouncer

Database Definition Example

pg_databases:
  - name: meta                      # Required, database name is the only mandatory field
    baseline: cmdb.sql              # Optional, database initialization SQL file path
    pgbouncer: true                 # Optional, add to pgbouncer, default: true
    schemas: [pigsty]               # Optional, additional schemas to create
    extensions:                     # Optional, extensions to install
      - { name: postgis, schema: public }
      - { name: timescaledb }
    comment: pigsty meta database   # Optional, database comment
    owner: postgres                 # Optional, database owner
    template: template1             # Optional, template database
    encoding: UTF8                  # Optional, character encoding
    locale: C                       # Optional, locale setting
    tablespace: pg_default          # Optional, default tablespace
    allowconn: true                 # Optional, allow connections
    revokeconn: false               # Optional, revoke public connect privilege
    register_datasource: true       # Optional, register as Grafana datasource
    connlimit: -1                   # Optional, connection limit
    pool_auth_user: dbuser_meta     # Optional, auth query user (with pgbouncer_auth_query)
    pool_mode: transaction          # Optional, pgbouncer pool mode
    pool_size: 50                   # Optional, pgbouncer default pool size
    pool_reserve: 30                # Optional, pgbouncer reserve pool size
    pool_size_min: 0                # Optional, pgbouncer minimum pool size
    pool_connlimit: 100             # Optional, pgbouncer max database connections

For details, see: Admin SOP: Create Database


pgsql-monitor.yml

Use pgsql-monitor.yml to add remote PostgreSQL instances to Pigsty monitoring.

Basic Usage

./pgsql-monitor.yml -e clsname=pg-foo  # Monitor remote cluster pg-foo

Wrapper Scripts

bin/pgmon-add pg-foo              # Monitor a remote pgsql cluster pg-foo
bin/pgmon-add pg-foo pg-bar       # Monitor multiple clusters simultaneously

Configuration

First, define pg_exporters in the infra group variables:

infra:
  hosts:
    10.10.10.10:
      pg_exporters:  # List all remote instances, assign unique unused local ports
        20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
        20002: { pg_cluster: pg-foo, pg_seq: 2, pg_host: 10.10.10.11 }

Architecture Diagram

     ------ infra ------
     |                 |
     | victoria-metrics|            v---- pg-foo-1 ----v
     |       ^         |  metrics   |         ^        |
     |   pg_exporter <-|------------|----  postgres    |
     |   (port: 20001) |            | 10.10.10.10:5432 |
     |       ^         |            ^------------------^
     |       ^         |                      ^
     |       ^         |            v---- pg-foo-2 ----v
     |       ^         |  metrics   |         ^        |
     |   pg_exporter <-|------------|----  postgres    |
     |   (port: 20002) |            | 10.10.10.11:5433 |
     -------------------            ^------------------^

Configurable Parameters

pg_exporter_config: pg_exporter.yml    # pg_exporter config file name
pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector TTL stages
pg_exporter_port: 9630                 # pg_exporter listen port
pg_exporter_params: 'sslmode=disable'  # DSN extra URL parameters
pg_exporter_url: ''                    # Directly override auto-generated DSN
pg_exporter_auto_discovery: true       # Enable auto database discovery
pg_exporter_exclude_database: 'template0,template1,postgres'  # Databases to exclude
pg_exporter_include_database: ''       # Databases to include only
pg_exporter_connect_timeout: 200       # Connection timeout (milliseconds)
pg_monitor_username: dbuser_monitor    # Monitor username
pg_monitor_password: DBUser.Monitor    # Monitor password

Remote Database Setup

Remote PostgreSQL instances need a monitoring user:

CREATE USER dbuser_monitor;
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';
ALTER USER dbuser_monitor PASSWORD 'DBUser.Monitor';
GRANT pg_monitor TO dbuser_monitor;
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

Limitations

  • Only postgres metrics available
  • node, pgbouncer, patroni, haproxy metrics not available

For details, see: Admin SOP: Monitor RDS


pgsql-migration.yml

pgsql-migration.yml generates migration manuals and scripts for existing PostgreSQL clusters. They use logical replication for migration without downtime.

Basic Usage

./pgsql-migration.yml -e@files/migration/pg-meta.yml

Workflow

  1. Define a migration task file, such as files/migration/pg-meta.yml.
  2. Run the playbook to generate the migration manual and scripts.
  3. Follow the manual and run the scripts in order.

Migration Task Definition Example

# files/migration/pg-meta.yml
context_dir: ~/migration           # Migration manual and scripts output directory
src_cls: pg-meta                   # Source cluster name (required)
src_db: meta                       # Source database name (required)
src_ip: 10.10.10.10                # Source cluster primary IP (required)
dst_cls: pg-test                   # Target cluster name (required)
dst_db: test                       # Target database name (required)
dst_ip: 10.10.10.11                # Target cluster primary IP (required)

# Optional parameters
pg_dbsu: postgres
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor

For details, see: Admin SOP: Migrate Cluster


pgsql-pitr.yml

The pgsql-pitr.yml playbook performs PostgreSQL Point-In-Time Recovery (PITR).

Basic Usage

# Recover to latest state (end of WAL archive stream)
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {}}'

# Recover to specific point in time
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"time": "2025-07-13 10:00:00+00"}}'

# Recover to specific LSN
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"lsn": "0/4001C80"}}'

# Recover to specific transaction ID
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"xid": "250000"}}'

# Recover to named restore point
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"name": "some_restore_point"}}'

# Recover from another cluster's backup
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": {"cluster": "pg-meta"}}'

PITR Task Parameters

pg_pitr:                           # Define PITR task
  cluster: "pg-meta"               # Source cluster name (for restoring from another cluster's backup)
  type: default                    # Recovery target type: default, time, xid, name, lsn, immediate
  time: "2025-01-01 10:00:00+00"   # Recovery target: point in time
  name: "some_restore_point"       # Recovery target: named restore point
  xid: "100000"                    # Recovery target: transaction ID
  lsn: "0/3000000"                 # Recovery target: log sequence number
  set: latest                      # Backup set to restore from, default: latest
  timeline: latest                 # Target timeline, can be integer, default: latest
  exclusive: false                 # Exclude target point, default: false
  action: pause                    # Post-recovery action: pause, promote, shutdown
  archive: true                    # Keep archive settings, default: true; set false for exploratory recovery
  backup: false                    # Backup existing data to /pg/data-backup before restore? default: false
  db_include: []                   # Include only these databases
  db_exclude: []                   # Exclude these databases
  link_map: {}                     # Tablespace link mapping
  process: 4                       # Parallel recovery processes, defaults to node_cpu
  repo: {}                         # Recovery source repo configuration
  data: /pg/data                   # Recovery data directory
  port: 5432                       # Recovery instance listen port

Subtasks

This playbook contains the following subtasks:

# down                 : stop HA and shutdown patroni and postgres
#   - pause            : pause patroni auto failover
#   - stop             : stop patroni and postgres services
#     - stop_patroni   : stop patroni service
#     - stop_postgres  : stop postgres service
#
# pitr                 : execute PITR recovery process
#   - config           : generate pgbackrest config and recovery script
#   - backup           : perform optional backup to original data
#   - restore          : run pgbackrest restore command
#   - recovery         : start postgres and complete recovery
#   - verify           : verify recovered cluster control data
#
# up                   : start postgres/patroni and restore HA
#   - etcd             : clean etcd metadata before startup
#   - start            : start patroni and postgres services
#     - start_postgres : start postgres service
#     - start_patroni  : start patroni service
#   - resume           : resume patroni auto failover

Recovery Target Types

TypeDescriptionExample
defaultRecover to end of WAL archive stream (latest state){"pg_pitr": {}}
timeRecover to specific point in time{"pg_pitr": {"time": "2025-07-13 10:00:00"}}
xidRecover to specific transaction ID{"pg_pitr": {"xid": "250000"}}
nameRecover to named restore point{"pg_pitr": {"name": "before_ddl"}}
lsnRecover to specific LSN{"pg_pitr": {"lsn": "0/4001C80"}}
immediateStop immediately after reaching consistent state{"pg_pitr": {"type": "immediate"}}

For details, see: Backup & Recovery Tutorial

13 - Extensions

Use PostgreSQL extensions with Pigsty

Pigsty provides 600 packaged extensions that are ready to use. The extensions cover 16 major categories, including time-series, geospatial, vector, full-text search, analytics, and feature enhancements.

To use extensions in Pigsty, complete four steps: Download, Install, Config/Load, and Create.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]   # Create: Create extensions in database
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # Config: Preload extension libraries
    pg_extensions: [ postgis, timescaledb, pgvector ]  # Install: Install extension packages
Pigsty PostgreSQL extension ecosystem

13.1 - Quick Start

Four-step process overview for using extensions

To use extensions in Pigsty, complete four steps: Download, Install, Config, and Create.

  1. Download: Download extension packages to the local repository. The default local repository only guarantees the base kernel and pgsql-main package set.
  2. Install: Install extension packages on cluster nodes
  3. Config: Some extensions need to be preloaded or configured with parameters
  4. Create: Execute CREATE EXTENSION in the database to create the extension

Declarative Configuration

Declare extensions in the Pigsty inventory. During cluster initialization, Pigsty installs their packages and creates the extensions automatically:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]   # Create extensions in database
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # Preload extension libraries
    pg_extensions: [ postgis, timescaledb, pgvector ]  # Install extension packages

Run ./pgsql.yml to initialize the cluster. The three extensions postgis, timescaledb, and vector will then be available in the meta database.


Imperative Operations

For an existing cluster, you can use commands to add extensions:

# 1. Install extension packages
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pgvector"]}'

# 2. Preload extension (if needed, requires restart after modification)
pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'

# 3. Create extension in database
psql -d meta -c 'CREATE EXTENSION vector;'

You can also install extension packages with the pig package manager. Then run CREATE EXTENSION in the database:

pig install pgvector                         # Install extension package
psql -d meta -c 'CREATE EXTENSION vector;'   # Create extension in database

Process Quick Reference

StepParameter/CommandDescription
Downloadrepo_extra_packagesSpecify extension packages to download to local repository
Installpg_extensionsSpecify extension packages to install on cluster
Configpg_libsPreload extensions to shared_preload_libraries
Createpg_databases.extensionsAutomatically execute CREATE EXTENSION in database

For detailed instructions, see each subsection: Download, Install, Config, Create

13.2 - Introduction

Core concepts of PostgreSQL extensions and the Pigsty extension ecosystem

Pigsty includes 600 precompiled PostgreSQL extensions that are ready to use.


What are Extensions

PostgreSQL extensions add database functionality without changes to the core code. An extension typically consists of three parts:

  • Control file (.control): Required, contains extension metadata
  • SQL scripts (.sql): Optional, defines functions, types, operators, and other database objects
  • Dynamic library (.so): Optional, provides high-performance functionality implemented in C

Extensions can add data types, index methods, functions, and operators to PostgreSQL. They can also add foreign data access, procedural languages, performance monitoring, security auditing, and other features.


Core Extensions

The following table shows representative extensions that Pigsty includes:

ExtensionDescription
PostGISGeospatial data types and indexes, de facto GIS standard
TimescaleDBTime-series database with continuous aggregates, columnar storage, auto-compression
PGVectorVector data type with HNSW/IVFFlat indexes, essential for AI applications
CitusDistributed database with horizontal sharding capabilities
pg_duckdbEmbedded DuckDB analytical engine for OLAP acceleration
pg_searchParadeDB search extension, providing BM25 and full-text search capabilities
Apache AGEGraph database supporting OpenCypher query language
pg_graphqlNative GraphQL query support

Most extensions can coexist. You can combine their features to perform tasks that one extension cannot perform alone.


Extension Categories

Pigsty organizes extensions into 16 categories:

CategoryAliasDescriptionTypical Extensions
Time-seriestimeTime-series data processingtimescaledb, pg_cron, periods
GeospatialgisGeospatial datapostgis, h3, pgrouting
VectorragVector retrieval and AIpgvector, vchord, pg_vectorize
SearchftsFull-text searchpgroonga, zhparser, pg_bigm
AnalyticsolapOLAP and analyticspg_duckdb, pg_mooncake, citus
FeaturefeatFeature enhancementsage, pg_graphql, hll, rum
LanguagelangProcedural languagesplpython3u, pljava, plv8
TypetypeData typeshstore, ltree, ip4r
UtilityutilUtility toolshttp, pg_net, pgjwt
FunctionfuncFunction librariespg_uuidv7, topn, tdigest
AdminadminOperations managementpg_repack, pg_squeeze, pgagent
StatstatMonitoring statisticspg_stat_statements, pg_qualstats, auto_explain
SecuritysecSecurity auditingpgaudit, pgsodium, pg_tde
FDWfdwForeign data accesspostgres_fdw, mysql_fdw, oracle_fdw
CompatibilitysimDatabase compatibilityorafce, babelfish
ETLetlData synchronizationpglogical, wal2json, decoderbufs

Use category aliases to install all extensions in a category together. For example: pg_extensions: [ pgsql-gis, pgsql-rag ].


Predefined Extension Stacks

Pigsty provides predefined extension stacks for different uses:

StackIncluded Extensions
gis-stackpostgis, pgrouting, pointcloud, h3, q3c, ogr_fdw
rag-stackpgvector, vchord, pgvectorscale, pg_similarity, pg_tiktoken
fts-stackpgroonga, pg_bigm, zhparser, hunspell
olap-stackpg_duckdb, pg_mooncake, timescaledb, pg_partman, plproxy
feat-stackage, hll, rum, pg_graphql, pg_jsonschema, jsquery
stat-stackpg_show_plans, pg_stat_kcache, pg_qualstats, pg_wait_sampling
supa-stackpg_graphql, pg_jsonschema, wrappers, pgvector, pgsodium, vault

To install an entire stack, add its name to pg_extensions.


Extension Resources

13.3 - Packages

Extension package aliases and category naming conventions

Pigsty uses package aliases to simplify extension installation and management.


Package Alias Mechanism

Extensions have different names at different layers:

LayerExample pgvectorExample postgis
Extension Namevectorpostgis, postgis_topology, …
Package Aliaspgvectorpostgis
RPM Package Namepgvector_18postgis36_18*
DEB Package Namepostgresql-18-pgvectorpostgresql-18-postgis-3*

Use a package alias to specify an extension without its RPM or DEB package name:

pg_extensions: [ pgvector, postgis, timescaledb ]  # Use package aliases

Pigsty converts each alias to the correct package names for the operating system and PostgreSQL version.

Note: When using CREATE EXTENSION, you use the extension name (e.g., vector), not the package alias (pgvector).


Category Aliases

Pigsty organizes all extensions into 16 categories. Use category aliases to install extensions from a category together:

# Use generic category aliases (auto-adapt to current PG version)
pg_extensions: [ pgsql-gis, pgsql-rag, pgsql-fts ]

# Or use version-specific category aliases
pg_extensions: [ pg18-gis, pg18-rag, pg18-fts ]

You can install all category extensions together, except for the olap category. In the olap category, pg_duckdb and pg_mooncake are mutually exclusive.


Category List

CategoryDescriptionTypical Extensions
timeTime-seriestimescaledb, pg_cron, periods
gisGeospatialpostgis, h3, pgrouting
ragVector/RAGpgvector, pgml, vchord
ftsFull-text Searchpg_trgm, zhparser, pgroonga
olapAnalyticscitus, pg_duckdb, pg_mooncake
featFeatureage, pg_graphql, rum
langLanguageplpython3u, pljava, plv8
typeData Typehstore, ltree, citext
utilUtilityhttp, pg_net, pgjwt
funcFunctionpgcrypto, uuid-ossp, pg_uuidv7
adminAdminpg_repack, pgagent, pg_squeeze
statStatisticspg_stat_statements, pg_qualstats, auto_explain
secSecuritypgaudit, pgcrypto, pgsodium
fdwForeign Data Wrapperpostgres_fdw, mysql_fdw, oracle_fdw
simCompatibilityorafce, babelfishpg_tds
etlData/ETLpglogical, wal2json, decoderbufs

Browse Extension Catalog

Use the Pigsty Extension Catalog to find these details for all available extensions:

  • Extension name, description, version
  • Supported PostgreSQL versions
  • Supported OS distributions
  • Installation methods, preloading requirements
  • License, source repository

13.4 - Download

Download extension packages to the local software repository

Before you install extensions, make sure that their packages are available from the local repository or an upstream repository.


Default Behavior

By default, Pigsty downloads the base PostgreSQL 18 kernel packages to the local software repository. The default extra download set is repo_extra_packages_default: [ pgsql-main ]. This set includes the PostgreSQL kernel, client, and procedural languages. It also includes basic extension packages such as pg_repack, wal2json, and pgvector.

If you need other extensions from the 600-extension catalog, add them to repo_extra_packages. Pigsty does not download every extension to the local repository by default.

A local repository provides these benefits:

  • Faster installation without repeated downloads
  • Less network traffic
  • More reliable delivery
  • Consistent package versions

Download New Extensions

To download additional extensions, add them to repo_extra_packages and rebuild the repository:

all:
  vars:
    repo_extra_packages: [ pgvector, postgis, timescaledb, pg_duckdb ]
# Re-download packages to local repository
./infra.yml -t repo_build

# Refresh package source cache on all nodes
./node.yml -t node_repo

Using Upstream Repositories

You can also install packages directly from upstream repositories on the internet. You do not need to download them in advance:

# Add upstream software sources on nodes
./node.yml -t node_repo -e node_repo_modules=node,pgsql

Use this method for:

  • Quick testing of latest versions
  • Installing rare extensions
  • Environments with good network conditions

This method has these risks:

  • An unstable network can interrupt installation.
  • Package versions can differ.

Extension Sources

Extension packages come from two main sources:

RepositoryDescription
PGDGPostgreSQL official repository, providing core extensions
PigstyPigsty supplementary repository, providing additional extensions

The Pigsty repository only includes extensions that are absent from the PGDG repository. If PGDG adds an extension, Pigsty will remove its package or keep it consistent with PGDG.

Repository URLs:

For detailed repository configuration, see Extension Repository.

13.5 - Install

Install extension packages on cluster nodes

Pigsty uses the operating system’s package manager (yum/apt) to install extension packages.


Use these two parameters to specify which extensions to install:

ParameterPurposeDefault Behavior
pg_packagesGlobal common packagesEnsure present (no upgrade)
pg_extensionsCluster-specific extensionsInstall latest version

Use pg_packages for essential extensions and base components that all clusters need. Typical components include the PostgreSQL kernel, Patroni, and pgBouncer.

Use pg_extensions for extensions that specific clusters need.

pg_packages:                           # Global base packages
  - pgsql-main pgsql-common
pg_extensions:                         # Cluster extensions
  - postgis timescaledb pgvector

Install During Cluster Initialization

Declare extensions in the cluster configuration. Pigsty installs them automatically during initialization:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector, pg_duckdb ]

When you run ./pgsql.yml to initialize the cluster, Pigsty installs the extensions automatically.


Install Extensions on Existing Cluster

For an initialized cluster, use one of these methods to install extensions:

Using Pigsty Playbook

# Install using playbook after modifying configuration
./pgsql.yml -l pg-meta -t pg_extension

# Or specify extensions directly on command line
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pg_duckdb"]}'

Using pig Package Manager

# Install extension using pig
pig install pg_duckdb

# Batch install
ansible pg-meta -b -a 'pig install pg_duckdb pgvector'

Using Package Manager Directly

# EL systems
sudo yum install -y pg_duckdb_18*

# Debian/Ubuntu systems
sudo apt install -y postgresql-18-pg-duckdb

Using Package Aliases

Pigsty supports standard package aliases. It converts each alias to the package name for the applicable PostgreSQL version:

pg_extensions:
  - pgvector           # Auto-translates to pgvector_18* (EL) or postgresql-18-pgvector (Debian)
  - postgis            # Auto-translates to postgis36_18* (EL) or postgresql-18-postgis-3* (Debian)
  - pgsql-gis          # Category alias, installs entire GIS category of extensions

You can also use raw package names directly:

pg_extensions:
  - pgvector_18*                    # EL system raw package name
  - postgresql-18-pgvector          # Debian system raw package name

For package alias definitions, see:


Verify Installation

After installation, run these checks in the database:

-- Check installed extensions
SELECT * FROM pg_available_extensions WHERE name = 'vector';

-- Check if extension files exist
\dx

13.6 - Config

Preload extension libraries and configure extension parameters

Some extensions require you to preload dynamic libraries or configure parameters before use. This section gives the configuration procedures.


Preload Extensions

After installation, you can enable most extensions with CREATE EXTENSION. Some extensions use PostgreSQL’s Hook mechanism and require preloading.

Set shared_preload_libraries to specify the libraries to preload. A database restart is required to apply the change.

Extensions Requiring Preload

Common extensions that require preloading:

ExtensionDescription
timescaledbTime-series database extension, must be placed first
citusDistributed database extension, must be placed first
pg_stat_statementsSQL statement statistics, enabled by default in Pigsty
auto_explainAutomatically log slow query execution plans, enabled by default in Pigsty
pg_cronScheduled task scheduling
pg_netAsynchronous HTTP requests
pg_tleTrusted language extensions
pgauditAudit logging
pg_stat_kcacheKernel statistics
pg_squeezeOnline table space reclamation
pgmlPostgresML machine learning

For the complete list, see the Extension Catalog (marked with LOAD).

Preload Order

Use the required loading order in shared_preload_libraries:

  • Put timescaledb and citus first.
  • If you use both, citus should come before timescaledb.
  • Statistics extensions should come after pg_stat_statements to use the same query_id.
pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain'

Configure During Cluster Initialization

For a new cluster, use pg_libs to specify the extensions to preload:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_extensions: [ timescaledb, postgis, pgvector ]

During cluster initialization, Pigsty writes the value of pg_libs to shared_preload_libraries.

Default Value

The default value of pg_libs is pg_stat_statements, auto_explain. These two Contrib extensions provide basic observability:

  • pg_stat_statements: Track execution statistics of all SQL statements
  • auto_explain: Automatically log execution plans for slow queries

Modify Configuration on Existing Cluster

For initialized clusters, use patronictl to modify shared_preload_libraries:

# Add timescaledb to preload libraries
pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'

# Restart cluster to apply configuration
pg restart pg-meta

You can also directly modify postgresql.conf or use ALTER SYSTEM:

ALTER SYSTEM SET shared_preload_libraries = 'timescaledb, pg_stat_statements, auto_explain';

After the change, restart the PostgreSQL service.


Extension Parameter Configuration

Many extensions have parameters that you can configure in the following locations:

During Cluster Initialization

Use pg_parameters to set extension parameters:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'pg_cron, pg_stat_statements, auto_explain'
    pg_parameters:
      cron.database_name: postgres           # Database used by pg_cron
      pg_stat_statements.track: all          # Track all statements
      auto_explain.log_min_duration: 1000    # Log queries exceeding 1 second

Runtime Modification

Use ALTER SYSTEM or patronictl:

-- Modify parameter
ALTER SYSTEM SET pg_stat_statements.track = 'all';

-- Reload configuration
SELECT pg_reload_conf();
# Modify using patronictl
pg edit-config pg-meta --force -p 'pg_stat_statements.track=all'

Important Notes

  1. Preload errors prevent startup: If an extension in shared_preload_libraries does not exist or fails to load, PostgreSQL will not start. Install each extension correctly before you add it to the preload list.

  2. Modification requires restart: After you change shared_preload_libraries, restart the PostgreSQL service to apply the change.

  3. Partial functionality available: Some extensions provide limited functionality without preloading. Full functionality requires preloading.

  4. View current configuration: Run this command to show the current preload libraries:

SHOW shared_preload_libraries;

13.7 - Create

Create and enable extensions in databases

After you install extension packages, run CREATE EXTENSION in the database to use their features.


View Available Extensions

After you install extension packages, run these queries to show the available extensions:

-- View all available extensions
SELECT * FROM pg_available_extensions;

-- View specific extension
SELECT * FROM pg_available_extensions WHERE name = 'vector';

-- View enabled extensions
SELECT * FROM pg_extension;

Create Extensions

Use CREATE EXTENSION to enable extensions in the database:

-- Create extension
CREATE EXTENSION vector;

-- Create extension in specific schema
CREATE EXTENSION postgis SCHEMA public;

-- Automatically install dependent extensions
CREATE EXTENSION postgis_topology CASCADE;

-- Create if not exists
CREATE EXTENSION IF NOT EXISTS vector;

Note: CREATE EXTENSION uses the extension name (e.g., vector), not the package alias (pgvector).


Create During Cluster Initialization

Declare extensions in pg_databases. Pigsty creates them during cluster initialization:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions:
          - { name: vector }                         # Use default schema
          - { name: postgis, schema: public }        # Specify schema
          - { name: pg_stat_statements, schema: monitor }

After it creates the database, Pigsty runs CREATE EXTENSION automatically.


Extensions Requiring Preload

Some extensions require preloading before creation. Add them to shared_preload_libraries, then restart PostgreSQL:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_databases:
      - name: meta
        extensions:
          - { name: timescaledb }  # Requires preload

If you create such an extension without preloading it, PostgreSQL returns an error.

Common extensions requiring preload: timescaledb, citus, pg_cron, pg_net, pgaudit, etc. See Configure Extensions.


Extension Dependencies

Some extensions depend on other extensions. Create the dependencies first:

-- postgis_topology depends on postgis
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;

-- Or use CASCADE to automatically install dependencies
CREATE EXTENSION postgis_topology CASCADE;

Extensions Not Requiring Creation

A few extensions have no SQL interfaces and do not require CREATE EXTENSION:

ExtensionDescription
wal2jsonLogical decoding plugin, used directly in replication slots
decoderbufsLogical decoding plugin
decoder_rawLogical decoding plugin

You can use these extensions immediately after installation. For example:

-- Create logical replication slot using wal2json
SELECT * FROM pg_create_logical_replication_slot('test_slot', 'wal2json');

View Extension Information

-- View extension details
\dx+ vector

-- View objects contained in extension
SELECT * FROM pg_extension_config_dump('vector');

-- View extension version
SELECT extversion FROM pg_extension WHERE extname = 'vector';

13.8 - Update

Upgrade PostgreSQL extension versions

Update extensions at two levels: packages in the operating system, and extension objects in the database.


Update Packages

Use package managers to update extension packages:

# EL systems
sudo yum update pgvector_18*

# Debian/Ubuntu systems
sudo apt update && sudo apt upgrade postgresql-18-pgvector

Use Pigsty to update packages on multiple nodes:

# Update extension packages for specified cluster
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pgvector"]}'

# Using pig package manager
pig update pgvector

Update Extension Objects

After you update packages, you may also need to update the extension objects in the database.

View Updatable Extensions

-- View installed extensions and their versions
SELECT name, default_version, installed_version
FROM pg_available_extensions
WHERE installed_version IS NOT NULL;

-- View upgradable extensions
SELECT name, installed_version, default_version
FROM pg_available_extensions
WHERE installed_version IS NOT NULL
  AND installed_version <> default_version;

Execute Extension Update

-- Update to latest version
ALTER EXTENSION pgvector UPDATE;

-- Update to specific version
ALTER EXTENSION pgvector UPDATE TO '0.8.0';

View Update Paths

-- View available upgrade paths for extension
SELECT * FROM pg_extension_update_paths('pgvector');

Important Notes

  1. Backup first: Back up the database before you update extensions. This is particularly important if the update changes data types.

  2. Check compatibility: Some major version upgrades may be incompatible. Read the extension’s upgrade documentation.

  3. Preloaded extensions: If you update a preloaded extension, such as timescaledb, you may need to restart the database.

  4. Dependencies: If other extensions depend on the updated extension, update them in dependency order.

  5. Replication environments: Test updates on replicas first. After you verify the results, update the primary.


Common Issues

Update Failure

If ALTER EXTENSION UPDATE fails, it may be because:

  • No available upgrade path
  • Extension is in use
  • Insufficient permissions
-- View extension dependencies
SELECT * FROM pg_depend WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');

Rollback Update

PostgreSQL extensions usually do not support direct rollback. To roll back, use one of these methods:

  1. Restore from backup
  2. Uninstall the new extension version, install the old package version, and recreate the extension.

13.9 - Remove

Uninstall PostgreSQL extensions

Extension removal has two levels: drop extension objects from the database, and uninstall packages from the operating system.


Drop Extension Objects

Use DROP EXTENSION to remove extensions from the database:

-- Drop extension
DROP EXTENSION pgvector;

-- If there are dependent objects, cascade delete is required
DROP EXTENSION pgvector CASCADE;

Warning: CASCADE will drop all objects that depend on this extension (tables, functions, views, etc.). Use with caution.

Check Extension Dependencies

We recommend that you check dependencies before you drop an extension:

-- View objects that depend on an extension
SELECT
    classid::regclass,
    objid,
    deptype
FROM pg_depend
WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');

-- View tables using extension types
SELECT
    c.relname AS table_name,
    a.attname AS column_name,
    t.typname AS type_name
FROM pg_attribute a
JOIN pg_class c ON a.attrelid = c.oid
JOIN pg_type t ON a.atttypid = t.oid
WHERE t.typname = 'vector';

Remove Preload

If the extension is in shared_preload_libraries, remove it from the preload list after you drop it:

# Modify shared_preload_libraries, remove extension
pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'

# Restart to apply configuration
pg restart pg-meta

Uninstall Packages

After dropping the extension from the database, you can optionally uninstall the package:

# EL systems
sudo yum remove pgvector_18*

# Debian/Ubuntu systems
sudo apt remove postgresql-18-pgvector

# Using pig package manager
pig remove pgvector

You can usually keep the package without problems. Uninstall it only to free disk space or resolve conflicts.


Important Notes

  1. Data loss risk: CASCADE drops dependent objects. This can cause data loss.

  2. Application compatibility: Before you drop the extension, make sure that applications no longer use its features.

  3. Preload order: If you drop a preloaded extension, also remove it from shared_preload_libraries. Otherwise, the database may fail to start.

  4. Replication environments: In replication environments, DROP EXTENSION automatically replicates to replicas.


Operation Sequence

Use this sequence to remove an extension completely:

# 1. Check dependencies
psql -d mydb -c "SELECT * FROM pg_depend WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');"

# 2. Drop extension from database
psql -d mydb -c "DROP EXTENSION pgvector;"

# 3. If it's a preloaded extension, remove from shared_preload_libraries
pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'

# 4. Restart database (if preload configuration was modified)
pg restart pg-meta

# 5. Optional: Uninstall package
sudo yum remove pgvector_18*

13.10 - Default Extensions

PostgreSQL extensions installed by default in Pigsty

During PostgreSQL cluster initialization, Pigsty installs and enables some core extensions by default.


Default Installed Extensions

Pigsty installs these extensions by default through pg_packages:

ExtensionDescription
pg_repackHandle table bloat online, important maintenance tool
wal2jsonLogical decoding outputs JSON format changes, commonly used in CDC scenarios
pgvectorVector data type and indexes, installed with pgsql-main by default

The default value of pg_extensions is an empty array []. Declare additional extensions as required. For example:

ExtensionDescription
postgisGeospatial database extension
timescaledbTime-series database extension
pgvectorVector data type and indexes

Default Enabled Extensions

Pigsty enables these extensions in all databases by default through pg_default_extensions:

ExtensionSchemaDescription
pg_stat_statementsmonitorSQL statement execution statistics
pgstattuplemonitorTuple-level statistics
pg_buffercachemonitorBuffer cache inspection
pageinspectmonitorPage-level inspection
pg_prewarmmonitorRelation prewarming
pg_visibilitymonitorVisibility map inspection
pg_freespacemapmonitorFree space map inspection
postgres_fdwpublicPostgreSQL foreign data wrapper
file_fdwpublicFile foreign data wrapper
btree_gistpublicB-tree GiST operator classes
btree_ginpublicB-tree GIN operator classes
pg_trgmpublicTrigram matching
intaggpublicInteger aggregator
intarraypublicInteger array functions
pg_repackrepackOnline table reorganization

These extensions provide basic monitoring and operations tools. They also add database features.


Default Preloaded Extensions

By default, pg_libs adds these extensions to shared_preload_libraries:

ExtensionDescription
pg_stat_statementsTrack execution statistics of all SQL statements
auto_explainAutomatically log execution plans for slow queries

These two extensions provide basic observability. We strongly recommend that you keep them.


Customize Default Extensions

To change which extensions Pigsty installs and enables by default, edit these configuration parameters:

all:
  vars:
    # Modify default extension packages
    pg_packages:
      - pgsql-main pgsql-common
      - pg_repack_$v* wal2json_$v*

    # Modify default installed extensions
    pg_extensions: [ postgis, timescaledb, pgvector ]

    # Modify default preloaded extensions
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'

    # Modify default enabled extensions
    pg_default_extensions:
      - { name: pg_stat_statements, schema: monitor }
      - { name: pg_repack }
      # ... add more

For extension procedures and concepts, see:

13.11 - Repository

Pigsty extension software repository configuration

Pigsty provides supplementary extension repositories. They supply extension packages in addition to those in the official PGDG repository.


YUM Repository

Use this repository with EL 8/9/10 and compatible systems, such as RHEL, Rocky, AlmaLinux, and CentOS.

Add Repository

# Add GPG public key
curl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null

# Add repository configuration
curl -fsSL https://repo.pigsty.io/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

# Refresh cache
sudo yum makecache

China Mainland Mirror

curl -fsSL https://repo.pigsty.cc/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.cc/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

Repository URLs


APT Repository

Use this repository with Debian 12/13, Ubuntu 22.04/24.04/26.04, and compatible systems.

Add Repository

# Add GPG public key
curl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg

# Get distribution codename and add repository
distro_codename=$(lsb_release -cs)
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main
EOF

# Refresh cache
sudo apt update

China Mainland Mirror

curl -fsSL https://repo.pigsty.cc/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg

distro_codename=$(lsb_release -cs)
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.cc/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.cc/apt/pgsql/${distro_codename} ${distro_codename} main
EOF

Repository URLs


GPG Signature

All packages are signed with GPG:

  • Fingerprint: 9592A7BC7A682E7333376E09E7935D8DB9BD8B20
  • Short ID: B9BD8B20

Repository Policy

The Pigsty repository follows these principles:

  1. Supplementary: The repository only includes extensions that are absent from the PGDG repository.
  2. Consistency: If PGDG adds an extension, Pigsty will remove its package or keep it consistent with PGDG.
  3. Compatibility: The repository supports PostgreSQL major versions 14-18.
  4. Multi-platform: The repository supports x86_64 and aarch64 architectures.

14 - Param Templates

Use Pigsty’s built-in Patroni config templates or customize your own

Pigsty provides four preset Patroni/PostgreSQL config templates optimized for different workloads:

TemplateCPU CoresUse CaseCharacteristics
/docs/pgsql/template/oltp.yml4-128COLTP transactionsHigh concurrency, low latency
/docs/pgsql/template/olap.yml4-128COLAP analyticsLarge queries, high parallelism
/docs/pgsql/template/crit.yml4-128CConsistency-firstConsistency-first, detailed auditing
/docs/pgsql/template/tiny.yml1-3CTiny instancesResource-constrained envs

Use pg_conf to select a template. The default is /docs/pgsql/template/oltp.yml.

The database tuning template pg_conf should be paired with the OS tuning template node_tune.

All four standard templates set wal_level to logical. PostgreSQL 18.6 adds the output_plugin_libraries security allowlist. Pigsty permits the built-in pgoutput and test_decoding plugins, plus wal2json from the default pgsql-main package set. Before you use another logical-decoding output plugin, review its code and privilege boundary. Then add its exact library name through pg_parameters. Patroni filters the template setting on older PostgreSQL versions that do not support it.


Usage

Set pg_conf in the cluster definition. We recommend a corresponding node_tune setting for OS tuning.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: oltp.yml    # PostgreSQL config template (default)
    node_tune: oltp      # OS tuning template (default)

For critical financial workloads, use /docs/pgsql/template/crit.yml:

pg-finance:
  hosts:
    10.10.10.21: { pg_seq: 1, pg_role: primary }
    10.10.10.22: { pg_seq: 2, pg_role: replica }
    10.10.10.23: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-finance
    pg_conf: crit.yml    # PostgreSQL critical template
    node_tune: crit      # OS critical tuning

For low-spec VMs or dev environments, use /docs/pgsql/template/tiny.yml:

pg-dev:
  hosts:
    10.10.10.31: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-dev
    pg_conf: tiny.yml    # PostgreSQL tiny template
    node_tune: tiny      # OS tiny tuning

Comparison

The four templates differ significantly in key parameters:

Connections & Memory

ParameterOLTPOLAPCRITTINY
max_connections500/1000500500/1000250
work_mem range64MB-1GB64MB-8GB64MB-1GB16MB-256MB
maintenance_work_mem25% shmem50% shmem25% shmem25% shmem
max_locks_per_transaction1-2x maxconn2-4x maxconn1-2x maxconn1-2x maxconn

Parallel Query

ParameterOLTPOLAPCRITTINY
max_worker_processesmax(cpu+16, 24)max(cpu+20, 28)max(cpu+16, 24)max(cpu+12, 20)
max_parallel_workers50% cpu80% cpu50% cpu50% cpu
max_parallel_workers_per_gather20% cpu (max 8)50% cpu0 (off)0 (off)
parallel_setup_cost2000100020001000
parallel_tuple_cost0.20.10.20.1

Sync Replication

ParameterOLTPOLAPCRITTINY
synchronous_modedepends pg_rpodepends pg_rpoforced ondepends pg_rpo
data_checksumsoptionaloptionalforced onoptional

Vacuum Config

ParameterOLTPOLAPCRITTINY
vacuum_cost_delay20ms10ms20ms20ms
vacuum_cost_limit20001000020002000
autovacuum_max_workers3332

Timeout & Security

ParameterOLTPOLAPCRITTINY
idle_in_transaction_session_timeout10minoff1min10min
log_min_duration_statement100ms1000ms100ms100ms
default_statistics_target4001000400200
track_activity_query_size8KB8KB32KB8KB
log_connectionsauthauthfulldefault

IO Config (PG18)

ParameterOLTPOLAPCRITTINY
io_workers25% cpu (4-16)50% cpu (4-32)25% cpu (4-8)3
temp_file_limit1/20 disk, max 100GB1/5 disk, max 400GB1/20 disk, max 100GB1/20 disk, max 100GB

Selection Guide

  • OLTP Template: Default choice for most transaction processing. Ideal for e-commerce, social, gaming apps.

  • OLAP Template: For data warehouses, BI reports, ETL. Allows large queries, high parallelism, relaxed timeouts.

  • CRIT Template: For financial transactions, core accounting with strict consistency/security requirements. Forced sync replication, checksums, full audit.

  • TINY Template: For dev/test environments, resource-constrained VMs, Raspberry Pi. Minimizes resource usage, disables parallel queries.


Custom Templates

Use an existing template as the basis for a custom template. The templates are in roles/pgsql/templates/:

roles/pgsql/templates/
├── oltp.yml    # OLTP template (default)
├── olap.yml    # OLAP template
├── crit.yml    # CRIT critical template
└── tiny.yml    # TINY micro template

To create a custom template:

  1. Copy an existing template.
  2. Change the parameters as required.
  3. Put the file in roles/pgsql/templates/.
  4. Select it with pg_conf.

Example:

cp roles/pgsql/templates/oltp.yml roles/pgsql/templates/myapp.yml
# Edit myapp.yml as needed

Then select the template in the cluster configuration:

pg-myapp:
  vars:
    pg_conf: myapp.yml

Templates use Jinja2 syntax. They calculate parameter values from node resources: CPU, memory, and disk.


Tuning Strategy

For technical details on template parameter optimization, see Tuning Strategy:

  • Memory tuning (shared buffers, work mem, max connections)
  • CPU tuning (parallel query worker config)
  • Storage tuning (WAL size, temp file limits)
  • Manual parameter adjustment

  • pg_conf: PostgreSQL config template
  • node_tune: OS tuning template, should match pg_conf
  • pg_rto: Recovery time objective, affects failover timeout
  • pg_rpo: Candidate-replica lag threshold; setting it to 0 enables synchronous replication in the general templates
  • pg_max_conn: Override template max connections
  • pg_shared_buffer_ratio: Shared buffer memory ratio
  • pg_storage_type: Storage type, affects IO params

14.1 - Parameter Optimization Policy

Learn the parameter optimization strategies Pigsty uses for the 4 different PostgreSQL workload scenarios.

Pigsty provides four parameter templates for different workloads. Select a template with pg_conf.

  • tiny.yml: Optimized for small nodes, VMs, and demos (the template is labeled for 1-3 cores)
  • oltp.yml: Optimized for OLTP workloads and latency-sensitive applications (4C8GB+) (default template)
  • olap.yml: Optimized for OLAP workloads and throughput (4C8G+)
  • crit.yml: Optimized for data consistency and critical applications (4C8G+)

The following sections describe the optimization settings for these four templates.


Memory Parameter Tuning

Pigsty detects the system’s memory size. It uses this value to set the maximum number of connections and memory-related parameters.

  • pg_max_conn: PostgreSQL maximum connections, auto will use recommended values for different scenarios
  • pg_shared_buffer_ratio: Shared buffer memory ratio, default is 0.25

By default, Pigsty assigns 25% of memory to PostgreSQL shared buffers. Connections, work_mem, background processes, and the operating-system cache share the remaining memory.

If you do not set pg_max_conn, Pigsty selects the maximum connections from these defaults:

  • oltp: 500 (pgbouncer) / 1000 (postgres)
  • crit: 500 (pgbouncer) / 1000 (postgres)
  • tiny: 250
  • olap: 500

For OLTP and CRIT, the maximum is 500 when the service connects through PgBouncer. If the service connects directly to PostgreSQL, the maximum doubles to 1000.

Pigsty calculates work_mem as shared memory size divided by maximum connections. It limits the result to 64MB ~ 1GB.

{% raw %}
{% if pg_max_conn != 'auto' and pg_max_conn|int >= 20 %}{% set pg_max_connections = pg_max_conn|int %}{% else %}{% if pg_default_service_dest|default('postgres') == 'pgbouncer' %}{% set pg_max_connections = 500 %}{% else %}{% set pg_max_connections = 1000 %}{% endif %}{% endif %}
{% set pg_max_prepared_transactions = pg_max_connections if 'citus' in pg_libs else 0 %}
{% set pg_max_locks_per_transaction = (2 * pg_max_connections)|int if 'citus' in pg_libs or 'timescaledb' in pg_libs else pg_max_connections %}
{% set pg_shared_buffers = (node_mem_mb|int * pg_shared_buffer_ratio|float) | round(0, 'ceil') | int %}
{% set pg_maintenance_mem = (pg_shared_buffers|int * 0.25)|round(0, 'ceil')|int %}
{% set pg_effective_cache_size = node_mem_mb|int - pg_shared_buffers|int  %}
{% set pg_workmem =  ([ ([ (pg_shared_buffers / pg_max_connections)|round(0,'floor')|int , 64 ])|max|int , 1024])|min|int %}
{% endraw %}

CPU Parameter Tuning

PostgreSQL has 4 important parameters for parallel queries. Pigsty sets them from the number of CPU cores. The templates first calculate a worker budget for parallel queries and extensions. They add eight reserved slots when they write max_worker_processes. The final GUC is therefore eight higher than the intermediate variable near the start of each template.

OLTPSetting LogicRange Limits
max_worker_processesmax(CPU + 8, 16) + 8max(CPU + 16, 24)
max_parallel_workersmax(ceil(50% CPU), 2)1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workersmax(ceil(33% CPU), 2)1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gathermin(max(ceil(20% CPU), 2),8)1/5 CPU rounded down, minimum 2, max 8
OLAPSetting LogicRange Limits
max_worker_processesmax(CPU + 12, 20) + 8max(CPU + 20, 28)
max_parallel_workersmax(ceil(80% CPU, 2))4/5 CPU rounded up, minimum 2
max_parallel_maintenance_workersmax(ceil(33% CPU), 2)1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gathermax(floor(50% CPU), 2)1/2 CPU rounded up, minimum 2
CRITSetting LogicRange Limits
max_worker_processesmax(CPU + 8, 16) + 8max(CPU + 16, 24)
max_parallel_workersmax(ceil(50% CPU), 2)1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workersmax(ceil(33% CPU), 2)1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather0, enable as needed
TINYSetting LogicRange Limits
max_worker_processesmax(CPU + 4, 12) + 8max(CPU + 12, 20)
max_parallel_workersmax(floor(50% CPU), 1)50% CPU rounded down, minimum 1
max_parallel_maintenance_workersmax(floor(33% CPU), 1)33% CPU rounded down, minimum 1
max_parallel_workers_per_gather0Disables parallel gather per query

The CRIT and TINY templates set max_parallel_workers_per_gather = 0 to disable parallel queries. To enable parallel queries, change this parameter as required.

The OLTP and CRIT templates set the following parameters. They double parallel query costs to make the planner less likely to choose parallel queries.

parallel_setup_cost: 2000           # double from 100 to increase parallel cost
parallel_tuple_cost: 0.2            # double from 0.1 to increase parallel cost
min_parallel_table_scan_size: 32MB  # 4x default 8MB, prefer non-parallel scan
min_parallel_index_scan_size: 2MB   # 4x default 512kB, prefer non-parallel scan

Changes to max_worker_processes take effect only after a restart. A replica fails to start if its value is higher than the primary value. Change this parameter through Patroni configuration management. Patroni keeps primary and replica settings consistent and prevents new replicas from failing to start during failover.


Storage Space Parameters

Pigsty detects the total disk capacity for the /data/postgres main data directory. It uses this value to set these parameters:

{% raw %}
{% set pg_size_twentieth = ([([(node_fs_bytes|int / 21474836480)|round(0, 'ceil')|int, 1])|max, 100])|min %}
min_wal_size: {{ ([pg_size_twentieth, 200])|min }}GB                  # 1/20 disk size, max 200GB
max_wal_size: {{ ([pg_size_twentieth * 4, 2000])|min }}GB             # 2/10 disk size, max 2000GB
max_slot_wal_keep_size: {{ ([pg_size_twentieth * 6, 3000])|min }}GB   # 3/10 disk size, max 3000GB
temp_file_limit: {{ ([pg_size_twentieth, 200])|min }}GB               # 1/20 of disk size, max 200GB
{% endraw %}
  • pg_size_twentieth is one twentieth of disk capacity rounded up, clamped to 1-100GB.
  • Therefore, in the three standard templates, the effective cap for temp_file_limit and min_wal_size is 100GB.
  • The effective cap for max_wal_size is 400GB.
  • The effective cap for max_slot_wal_keep_size is 600GB.

The OLAP template sets temp_file_limit to pg_size_twentieth × 4, with an effective limit of 400GB. Existing 200GB/2TB/3TB comments at the ends of template lines omit the 100GB limit already applied to pg_size_twentieth. Use the rendered expression as the authoritative value.


Manual Parameter Tuning

You can manually adjust PostgreSQL parameters in addition to the values that Pigsty sets automatically.

Use the pg edit-config <cluster> command to interactively edit cluster configuration:

pg edit-config pg-meta

Or use the -p parameter to directly set parameters:

pg edit-config -p log_min_duration_statement=1000 pg-meta
pg edit-config --force -p shared_preload_libraries='timescaledb, pg_cron, pg_stat_statements, auto_explain' pg-meta

You can also use the Patroni REST API to modify configuration:

curl -u 'postgres:Patroni.API' \
    -d '{"postgresql":{"parameters": {"log_min_duration_statement":200}}}' \
    -s -X PATCH http://10.10.10.10:8008/config | jq .

14.2 - OLTP Template

PostgreSQL config template optimized for online transaction processing workloads

oltp.yml is Pigsty’s default configuration template for online transaction processing (OLTP). It is designed for CPUs with 4-128 cores and provides high concurrency, low latency, and high throughput.

Pair with node_tune = oltp for OS-level tuning.


Use Cases

Use the OLTP template for:

  • E-commerce: Order processing, inventory, user transactions
  • Social apps: User feeds, messaging, following relationships
  • Gaming backends: Player data, leaderboards, game state
  • SaaS applications: Multi-tenant business systems
  • Web apps: CRUD-intensive workloads

Workload characteristics:

  • Many short transactions (millisecond-level)
  • High concurrent connections (hundreds to thousands)
  • Read/write ratio typically 7:3 to 9:1
  • Latency-sensitive, requires fast response
  • High data consistency requirements

Usage

oltp.yml is the default template. You do not need to select it explicitly:

pg-oltp:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-oltp
    # pg_conf: oltp.yml  # PostgreSQL config template (default)
    # node_tune: oltp    # OS tuning template (default)

To select it explicitly, use:

pg-oltp:
  vars:
    pg_conf: oltp.yml    # PostgreSQL config template
    node_tune: oltp      # OS tuning template

Parameter Details

Connection Management

max_connections: 500/1000   # depends on pgbouncer usage
superuser_reserved_connections: 10
  • When pg_default_service_dest is pgbouncer, max_connections is set to 500
  • When traffic connects directly to PostgreSQL, max_connections is set to 1000
  • Override via pg_max_conn parameter

Memory Config

OLTP template memory allocation strategy:

ParameterFormulaDescription
shared_buffersmem × pg_shared_buffer_ratioDefault ratio 0.25
maintenance_work_memshared_buffers × 25%For VACUUM, CREATE INDEX
work_mem64MB - 1GBBased on shared_buffers/max_connections
effective_cache_sizetotal mem - shared_buffersEstimated cache memory

work_mem calculation:

work_mem = min(max(shared_buffers / max_connections, 64MB), 1GB)

This calculation gives each connection sufficient sort/hash memory without excessive allocation.

Parallel Query

OLTP template moderately limits parallel queries to prevent resource contention:

max_worker_processes: max(cpu + 16, 24)
max_parallel_workers: 50% × cpu (min 2)
max_parallel_workers_per_gather: 20% × cpu (2-8)
max_parallel_maintenance_workers: 33% × cpu (min 2)

Parallel cost estimates are increased to favor serial execution:

parallel_setup_cost: 2000      # 2x default (1000)
parallel_tuple_cost: 0.2       # 2x default (0.1)
min_parallel_table_scan_size: 32MB   # 4x default (8MB), prefer non-parallel scan
min_parallel_index_scan_size: 2MB    # 4x default (512kB), prefer non-parallel scan

WAL Config

min_wal_size: disk/20 (effective max 100GB)
max_wal_size: disk/5 (effective max 400GB)
max_slot_wal_keep_size: disk×3/10 (effective max 600GB)
wal_buffers: 16MB
wal_writer_delay: 20ms
wal_writer_flush_after: 1MB
commit_delay: 20
commit_siblings: 10
checkpoint_timeout: 15min
checkpoint_completion_target: 0.80

These settings balance data safety and write performance.

Vacuum Config

vacuum_cost_delay: 20ms         # sleep after each vacuum round
vacuum_cost_limit: 2000         # cost limit per vacuum round
autovacuum_max_workers: 3
autovacuum_naptime: 1min
autovacuum_vacuum_scale_factor: 0.08    # 8% table change triggers vacuum
autovacuum_analyze_scale_factor: 0.04   # 4% table change triggers analyze
autovacuum_freeze_max_age: 1000000000

Conservative vacuum settings avoid impacting online transaction performance.

Query Optimization

random_page_cost: 1.1           # SSD optimized
effective_io_concurrency: 200   # SSD concurrent IO
default_statistics_target: 400  # Statistics precision

These settings help the planner generate better query plans.

Logging & Monitoring

log_min_duration_statement: 100         # log queries > 100ms
log_statement: ddl                      # log DDL statements
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024                    # log temp files > 1MB
log_autovacuum_min_duration: 1s
track_io_timing: on
track_functions: all
track_activity_query_size: 8192

Client Timeouts

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 10min

The 10-minute idle transaction timeout prevents inactive transactions from keeping locks indefinitely.

Extension Config

shared_preload_libraries: 'pg_stat_statements, auto_explain'

# auto_explain
auto_explain.log_min_duration: 1s
auto_explain.log_analyze: on
auto_explain.log_verbose: on
auto_explain.log_timing: on
auto_explain.log_nested_statements: true

# pg_stat_statements
pg_stat_statements.max: 10000
pg_stat_statements.track: all
pg_stat_statements.track_utility: off
pg_stat_statements.track_planning: off

Template Comparison

FeatureOLTPOLAPCRIT
max_connections500-1000500500-1000
work_mem64MB-1GB64MB-8GB64MB-1GB
Parallel queryModerate limitAggressiveDisabled
Vacuum intensityConservativeAggressiveConservative
Txn timeout10minDisabled1min
Slow query threshold100ms1000ms100ms

Why OLTP over OLAP?

  • Queries are mostly simple point/range lookups
  • Transaction response time requires milliseconds
  • High concurrent connections
  • No complex analytical queries

Why OLTP over CRIT?

  • Small probability of data loss acceptable (async replication)
  • Complete audit logs not required
  • Better write performance desired

Performance Tuning Tips

Connection Pooling

For high concurrency, use PgBouncer connection pool:

pg-oltp:
  vars:
    pg_default_service_dest: pgbouncer  # default
    pgbouncer_poolmode: transaction     # transaction-level pooling

Read Separation

Use read replicas to share read load:

pg-oltp:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }

Monitoring Metrics

Focus on these metrics:

  • Connections: Active/waiting connection counts
  • Transaction rate: TPS, commit/rollback ratio
  • Response time: Query latency percentiles (p50/p95/p99)
  • Lock waits: Lock wait time, deadlock counts
  • Replication lag: Replica delay time and bytes

References

14.3 - OLAP Template

PostgreSQL config template optimized for online analytical processing workloads

olap.yml is optimized for online analytical processing (OLAP) on CPUs with 4-128 cores. It supports large queries, high parallelism, relaxed timeouts, and aggressive vacuum.

Pair with node_tune = olap for OS-level tuning.


Use Cases

Use the OLAP template for:

  • Data warehouses: Historical data storage, multidimensional analysis
  • BI reports: Complex report queries, dashboard data sources
  • ETL processing: Data extraction, transformation, loading
  • Data analysis: Ad-hoc queries, data exploration
  • HTAP mixed workloads: Analytical replicas

Workload characteristics:

  • Complex queries (seconds to minutes)
  • Low concurrent connections (tens to hundreds)
  • Read-intensive, writes typically batch operations
  • Throughput-sensitive, tolerates higher latency
  • Scans large data volumes

Usage

Specify pg_conf = olap.yml in cluster definition:

pg-olap:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-olap
    pg_conf: olap.yml    # PostgreSQL analytics template
    node_tune: olap      # OS analytics tuning

Use olap.yml template for dedicated offline replicas:

pg-mixed:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline, pg_conf: olap.yml }  # offline analytics replica
  vars:
    pg_cluster: pg-mixed
    pg_conf: oltp.yml    # primary and online replicas use OLTP
    node_tune: oltp      # OS OLTP tuning

Parameter Details

Connection Management

max_connections: 500
superuser_reserved_connections: 10

OLAP workloads usually need fewer connections. The limit of 500 is sufficient for most analytical workloads.

Memory Config

The OLAP template assigns more memory to operations:

ParameterFormulaDescription
shared_buffersmem × pg_shared_buffer_ratioDefault ratio 0.25
maintenance_work_memshared_buffers × 50%Faster index creation and VACUUM
work_mem64MB - 8GBLarger sort/hash memory
effective_cache_sizetotal mem - shared_buffersEstimated cache memory

work_mem calculation (differs from OLTP):

work_mem = min(max(shared_buffers / max_connections, 64MB), 8GB)

Larger work_mem allows bigger sort and hash operations in memory, avoiding disk spill.

Locks & Transactions

max_locks_per_transaction: 2-4x maxconn   # OLTP: 1-2x

OLAP queries may access more tables through partitions and multiple JOINs. They therefore require more lock slots.

Parallel Query

The OLAP template assigns more workers to parallel queries:

max_worker_processes: max(cpu + 20, 28)      # OLTP: max(cpu + 16, 24)
max_parallel_workers: 80% × cpu (min 2)      # OLTP: 50%
max_parallel_workers_per_gather: 50% × cpu   # OLTP: 20% (max 8)
max_parallel_maintenance_workers: 33% × cpu

Parallel cost estimates use defaults to favor parallel plans:

# parallel_setup_cost: 1000    # default, not doubled
# parallel_tuple_cost: 0.1     # default, not doubled

The template enables partition-wise optimization:

enable_partitionwise_join: on       # smart partition JOIN
enable_partitionwise_aggregate: on  # smart partition aggregation

IO Config (PG18)

io_workers: 50% × cpu (4-32)    # OLTP: 25% (4-16)

More IO workers support parallel large table scans.

WAL Config

min_wal_size: disk/20 (effective max 100GB)
max_wal_size: disk/5 (effective max 400GB)
max_slot_wal_keep_size: disk×3/10 (effective max 600GB)
temp_file_limit: disk/5 (effective max 400GB)   # OLTP: disk/20, effective max 100GB

Larger temp_file_limit allows bigger intermediate results to spill to disk.

Vacuum Config

OLAP template uses aggressive vacuum settings:

vacuum_cost_delay: 10ms         # OLTP: 20ms, faster vacuum
vacuum_cost_limit: 10000        # OLTP: 2000, more work per round
autovacuum_max_workers: 3
autovacuum_naptime: 1min
autovacuum_vacuum_scale_factor: 0.08
autovacuum_analyze_scale_factor: 0.04

Analytical databases often have bulk writes requiring aggressive vacuum to reclaim space.

Query Optimization

random_page_cost: 1.1
effective_io_concurrency: 200
default_statistics_target: 1000    # OLTP: 400, more precise stats

A higher default_statistics_target provides more accurate query plans. This is important for complex analytics.

Logging & Monitoring

log_min_duration_statement: 1000    # OLTP: 100ms, relaxed threshold
log_statement: ddl
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024
log_autovacuum_min_duration: 1s
track_io_timing: on
track_cost_delay_timing: on         # PG18+, track vacuum cost delay
track_functions: all
track_activity_query_size: 8192

Client Timeouts

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 0   # OLTP: 10min, disabled

Analytical queries may need to keep transactions open for long periods. Thus, the template disables the idle timeout.


Key Differences from OLTP

ParameterOLAPOLTPReason
max_connections500500-1000Fewer analytical connections
work_mem limit8GB1GBSupport larger in-memory sorts
maintenance_work_mem50% buffer25% bufferFaster index creation
max_locks_per_transaction2-4x1-2xMore tables in queries
max_parallel_workers80% cpu50% cpuAggressive parallelism
max_parallel_workers_per_gather50% cpu20% cpuAggressive parallelism
parallel_setup_cost10002000Default, encourages parallel
parallel_tuple_cost0.10.2Default, encourages parallel
enable_partitionwise_joinonoffPartition optimization
enable_partitionwise_aggregateonoffPartition optimization
vacuum_cost_delay10ms20msAggressive vacuum
vacuum_cost_limit100002000Aggressive vacuum
temp_file_limit1/5 disk1/20 diskAllow larger temp files
io_workers50% cpu25% cpuMore parallel IO
log_min_duration_statement1000ms100msRelaxed slow query threshold
default_statistics_target1000400More precise stats
idle_in_transaction_session_timeoutDisabled10minAllow long transactions

Performance Tuning Tips

With TimescaleDB

You can use the OLAP template with TimescaleDB:

pg-timeseries:
  vars:
    pg_conf: olap.yml
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_extensions:
      - timescaledb

With pg_duckdb

To improve analytical performance, combine the template with pg_duckdb:

pg-analytics:
  vars:
    pg_conf: olap.yml
    pg_libs: 'pg_duckdb, pg_stat_statements, auto_explain'

Columnar Storage

Consider columnar storage extensions:

pg_extensions:
  - citus_columnar  # or pg_mooncake

Resource Isolation

For mixed workloads, isolate analytics to dedicated replicas:

pg-mixed:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }               # OLTP writes
    10.10.10.12: { pg_seq: 2, pg_role: replica }               # OLTP reads
    10.10.10.13: { pg_seq: 3, pg_role: offline }               # OLAP analytics
  vars:
    pg_cluster: pg-mixed

Monitoring Metrics

Focus on these metrics:

  • Query time: Long query execution time distribution
  • Parallelism: Parallel worker utilization
  • Temp files: Temp file size and count
  • Disk IO: Sequential and index scan IO volume
  • Cache hit ratio: shared_buffers and OS cache hit rates

References

14.4 - CRIT Template

PostgreSQL parameter template for consistency-first workloads, with strict synchronous replication, data checksums, and detailed connection logging.

crit.yml is for transactional workloads with increased consistency and audit requirements. It forces data checksums and Patroni strict synchronous mode. It also adds connection logging and adjusts selected WAL, timeout, and parallel-query parameters.

The template increases write latency and may block writes when no synchronous replica is available. Before use, confirm consistency objectives, failure domains, client commit settings, and availability requirements.

Also evaluate node_tune: crit. You can select host tuning and database parameters independently.


Usage

pg-critical:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-critical
    pg_conf: crit.yml
    node_tune: crit

A three-node topology lets the cluster select another synchronous replica after one node fails. Continued write availability depends on the remaining node state, DCS, network, and synchronous-replica selection. Test failures on the target topology.


Strict Synchronous Replication

CRIT does not derive synchronous mode from pg_rpo. It always enables these settings:

synchronous_mode: true
synchronous_mode_strict: true

synchronous_mode_strict prevents Patroni from switching to asynchronous replication when no synchronous replica is available. The primary then blocks writes that require synchronous acknowledgment.

The mode targets preservation of acknowledged transactions when:

  • the session has not lowered synchronous_commit to local, off, or another asynchronous level;
  • a synchronous replica acknowledges WAL during commit;
  • failover selects only an eligible node containing the required WAL.

Validate RPO against client parameters, replication state, and the failure model. The template name alone does not establish RPO.

To require acknowledgment from multiple synchronous replicas, change Patroni dynamic configuration:

pg edit-config pg-critical
synchronous_node_count: 2

A higher synchronous-replica count imposes stricter conditions for accepting writes.


Data Checksums

CRIT initialization always includes:

initdb:
  - data-checksums

This overrides a disabled pg_checksum setting and enables page checksums for a new cluster. Checksums detect page damage after a write. They do not detect logical errors or every memory error.


Connection and Query Logging

CRIT logs DDL, statements taking longer than 100 ms, and disconnection events:

log_statement: ddl
log_min_duration_statement: 100
log_disconnections: 'on'

PostgreSQL 18 and later use:

log_connections: 'receipt,authentication,authorization'

Earlier versions use log_connections: on. These records support connection auditing but do not provide fine-grained SQL audit logs. Enable pgaudit separately to record object reads and writes, roles, or statement classes.

CRIT sets track_activity_query_size to 32 KiB to retain longer active-query text. Logs may contain SQL and business data. Restrict access and set a suitable retention period.


Watchdog

CRIT changes Patroni watchdog from disabled to automatic:

watchdog:
  mode: automatic
  device: /dev/watchdog

automatic activates only when the system has a usable watchdog device. If fencing must be mandatory, verify hardware, virtualization support, and device permissions before you explicitly set required. An incorrect configuration can prevent primary startup or interrupt failover.


Key Parameter Differences

ParameterCRITOLTP DefaultEffect
synchronous_modeAlways enabledDerived from pg_rpoConsistency first
synchronous_mode_stricttrueGeneral template behaviorBlocks writes without a synchronous replica
data-checksumsAlways enabledControlled by pg_checksumPage-damage detection
max_parallel_workers_per_gather0Calculated from CPUReduces parallel-query variability
wal_writer_delay10ms20msProcesses WAL more frequently
wal_writer_flush_after01MBChanges WAL flush behavior
idle_replication_slot_timeout3d7dRemoves idle replication slots sooner
idle_in_transaction_session_timeout1min10minTerminates idle transactions sooner
track_activity_query_size32KiB8KiBRetains longer query text
log_connectionsDetailed connection eventsPostgreSQL 18 logs authorization by defaultAdds connection-audit detail
log_disconnectionsonoffRecords disconnections

CRIT also disables parallel gather for individual queries and adjusts parallel costs, autovacuum, WAL, and statistics parameters. The release file roles/pgsql/templates/crit.yml defines the active values.


Preloaded Extensions

CRIT generates shared_preload_libraries from pg_libs. The role default is:

pg_libs: 'pg_stat_statements, auto_explain'

Selecting crit.yml alone does not load passwordcheck. If you need password-complexity checks, configure it explicitly:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain'

ha/safe includes this override. To use pgaudit, also add it to pg_libs and configure the audit scope:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain, pgaudit'
pg_parameters:
  pgaudit.log: 'ddl, role, write'

Performance and Availability Impact

  • Synchronous commit waits for a synchronous replica; write latency includes at least replica network and WAL durability time.
  • Strict synchronous mode blocks writes when no synchronous replica is available.
  • Disabling parallel gather can reduce throughput for large queries, but also reduces resource variability from parallel execution.
  • More detailed logging and statistics consume additional I/O, CPU, and storage.
  • A shorter idle-transaction timeout may terminate application sessions that hold a transaction open without executing statements.

The impact depends on hardware, network, queries, and client behavior. Test the actual workload. Do not rely on a fixed latency or throughput percentage.


Launch Checklist

  • Deploy at least one usable synchronous replica and verify write behavior during node failure
  • Check whether applications change synchronous_commit
  • Select watchdog automatic or required according to availability requirements
  • Verify collection, access control, and retention for connection logs
  • Configure pg_libs and extension parameters explicitly when password checks or SQL auditing are required
  • Test write latency, throughput, and idle-transaction timeouts with the production workload
  • Exercise primary, synchronous-replica, DCS, and network-partition failures

14.5 - TINY Template

PostgreSQL config template optimized for micro instances and resource-constrained environments

tiny.yml is optimized for micro instances and environments with limited resources. It is designed for CPUs with 1-3 cores. It minimizes resource use, limits memory allocation, and disables parallel queries.

Pair with node_tune = tiny for OS-level tuning.


Use Cases

Use the TINY template for:

  • Dev/test: Local development, CI/CD testing
  • Low-spec VMs: 1-2 core CPU, 1-4GB RAM cloud instances
  • Edge computing: Raspberry Pi, embedded devices
  • Demos: Quick Pigsty experience
  • Personal projects: Resource-limited blogs, small apps

Resource constraints:

  • 1-3 CPU cores
  • 1-8 GB RAM
  • Limited disk space
  • May share resources with other services

Usage

Specify pg_conf = tiny.yml in cluster definition:

pg-dev:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-dev
    pg_conf: tiny.yml    # PostgreSQL micro instance template
    node_tune: tiny      # OS micro instance tuning

Single-node development:

pg-local:
  hosts:
    127.0.0.1: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-local
    pg_conf: tiny.yml    # PostgreSQL micro instance template
    node_tune: tiny      # OS micro instance tuning

Parameter Details

Connection Management

max_connections: 250   # OLTP: 500-1000, reduced connection overhead
superuser_reserved_connections: 10

Micro instances need fewer concurrent connections. The limit of 250 is sufficient for development and testing.

Memory Config

TINY template uses conservative memory allocation:

ParameterFormulaDescription
shared_buffersmem × pg_shared_buffer_ratioDefault ratio 0.25
maintenance_work_memshared_buffers × 25%For VACUUM, CREATE INDEX
work_mem16MB - 256MBSmaller sort/hash memory
effective_cache_sizetotal mem - shared_buffersEstimated cache memory

work_mem calculation (differs from OLTP):

work_mem = min(max(shared_buffers / max_connections, 16MB), 256MB)

Smaller work_mem limit (256MB vs OLTP’s 1GB) prevents memory exhaustion.

Parallel Query (Fully Disabled)

TINY template completely disables parallel queries:

max_worker_processes: max(cpu + 12, 20)     # OLTP: max(cpu + 16, 24)
max_parallel_workers: 50% × cpu (min 1)      # OLTP: 50% (min 2)
max_parallel_workers_per_gather: 0           # parallel queries disabled
max_parallel_maintenance_workers: 33% × cpu (min 1)

max_parallel_workers_per_gather: 0 prevents queries from starting parallel workers. This avoids resource contention on systems with few CPU cores.

IO Config (PG18)

io_workers: 3   # fixed value, OLTP: 25% cpu (4-16)

A fixed, low I/O worker count is suitable for environments with limited resources.

Vacuum Config

vacuum_cost_delay: 20ms
vacuum_cost_limit: 2000
autovacuum_max_workers: 2          # OLTP: 3, one fewer worker
autovacuum_naptime: 1min
# autovacuum_vacuum_scale_factor uses default
# autovacuum_analyze_scale_factor uses default

Fewer autovacuum workers reduce background resource usage.

Query Optimization

random_page_cost: 1.1
effective_io_concurrency: 200
default_statistics_target: 200     # OLTP: 400, lower precision saves space

Lower default_statistics_target reduces pg_statistic table size.

Logging Config

log_min_duration_statement: 100    # same as OLTP
log_statement: ddl
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024
# log_connections uses default (no extra logging)

The TINY template disables additional connection logging to reduce log volume.

Client Timeouts

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 10min   # same as OLTP

Extension Config

shared_preload_libraries: 'pg_stat_statements, auto_explain'

pg_stat_statements.max: 2500      # OLTP: 10000, reduced memory usage
pg_stat_statements.track: all
pg_stat_statements.track_utility: off
pg_stat_statements.track_planning: off

The template reduces pg_stat_statements.max from 10000 to 2500. This saves approximately 75% of its memory use.


Key Differences from OLTP

ParameterTINYOLTPReason
max_connections250500-1000Reduce connection overhead
work_mem limit256MB1GBPrevent memory exhaustion
max_worker_processesmax(cpu+12, 20)max(cpu+16, 24)Fewer background processes
max_parallel_workers_per_gather020% cpuDisable parallel queries
autovacuum_max_workers23Reduce background load
default_statistics_target200400Save space
pg_stat_statements.max250010000Reduce memory usage
io_workers325% cpuFixed low value

Resource Estimates

TINY template resource usage by configuration:

1 Core 1GB RAM

shared_buffers: ~256MB
work_mem: ~16MB
maintenance_work_mem: ~64MB
max_connections: 250
max_worker_processes: 20

PostgreSQL process memory: ~400-600MB

2 Core 4GB RAM

shared_buffers: ~1GB
work_mem: ~32MB
maintenance_work_mem: ~256MB
max_connections: 250
max_worker_processes: 20

PostgreSQL process memory: ~1.5-2GB

4 Core 8GB RAM

Consider using OLTP template instead:

pg-small:
  vars:
    pg_conf: oltp.yml   # 4C8G can use OLTP template

Performance Tuning Tips

Further Resource Reduction

For extremely constrained resources:

pg_parameters:
  max_connections: 100           # further reduce
  shared_buffers: 128MB          # further reduce
  maintenance_work_mem: 32MB
  work_mem: 8MB

Disable Unnecessary Extensions

pg_libs: 'pg_stat_statements'    # keep only essential extensions

Disable Unnecessary Features

pg_parameters:
  track_io_timing: off           # disable IO timing tracking
  track_functions: none          # disable function tracking

Use External Connection Pool

Even on micro instances, PgBouncer significantly improves concurrency:

pg-tiny:
  vars:
    pg_conf: tiny.yml
    pg_default_service_dest: pgbouncer
    pgbouncer_poolmode: transaction

Cloud Platform Recommendations

AWS

  • t3.micro: 1 vCPU, 1GB RAM - suitable for TINY
  • t3.small: 2 vCPU, 2GB RAM - suitable for TINY
  • t3.medium: 2 vCPU, 4GB RAM - consider OLTP

Alibaba Cloud

  • ecs.t6-c1m1.small: 1 vCPU, 1GB RAM - suitable for TINY
  • ecs.t6-c1m2.small: 1 vCPU, 2GB RAM - suitable for TINY
  • ecs.t6-c1m4.small: 1 vCPU, 4GB RAM - suitable for TINY

Tencent Cloud

  • SA2.SMALL1: 1 vCPU, 1GB RAM - suitable for TINY
  • SA2.SMALL2: 1 vCPU, 2GB RAM - suitable for TINY
  • SA2.SMALL4: 1 vCPU, 4GB RAM - suitable for TINY

Edge Device Deployment

Raspberry Pi 4

pg-pi:
  hosts:
    192.168.1.100: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-pi
    pg_conf: tiny.yml       # PostgreSQL micro instance template
    node_tune: tiny         # OS micro instance tuning
    pg_storage_type: SSD    # SSD storage recommended

Docker Container

pg-docker:
  hosts:
    172.17.0.2: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-docker
    pg_conf: tiny.yml       # PostgreSQL micro instance template
    node_tune: tiny         # OS micro instance tuning

Upgrading to OLTP

When the application needs more resources, upgrade to the OLTP template:

  1. Upgrade VM specs (4 core 8GB+)
  2. Modify cluster config:
pg-growing:
  vars:
    pg_conf: oltp.yml    # change from tiny.yml to oltp.yml
    node_tune: oltp      # change from tiny to oltp
  1. Reconfigure cluster or redeploy

References

15 - PG Kernels

How to use PostgreSQL kernel forks in Pigsty, such as Citus, Babelfish, IvorySQL, PolarDB, and more.

In Pigsty, you can replace the native PostgreSQL kernel with a variant that provides specialized features.

Pigsty supports multiple PostgreSQL kernels and compatibility branches within one operational framework. These kernels provide compatibility layers, multi-master replication, graph queries, MPP warehousing, transparent encryption, and other features.

The level of Pigsty integration differs between kernels. PostgreSQL, Citus, Babelfish, IvorySQL, PolarDB, AgensGraph, and pgEdge have defined templates and configuration procedures. Cloudberry and Greenplum usually use gpsql mode. For their MPP initialization and scale-out operations, upstream tools remain the preferred method.

KernelKey FeatureDescription
PostgreSQLNative kernel, full extension setVanilla PostgreSQL with 600 extensions
SupabaseBackend as a ServicePostgreSQL-based BaaS, Firebase alternative
CitusHorizontal scalingDistributed PostgreSQL via native extension
BabelfishSQL Server compatibleSQL Server wire-protocol compatibility (PG17/18)
IvorySQLOracle compatibleOracle syntax and PL/SQL compatibility
OpenHaloMySQL compatibleMySQL wire-protocol compatibility
PerconaTransparent data encryptionPercona distribution with pg_tde
DocumentDBMongoDB migrationDocumentDB + FerretDB wire compatibility
OrioleDBOLTP optimizationZheap, no bloat, S3 storage
PolarDBAurora-style RACRAC, China-local compliance scenario
CloudberryOpen-source MPP warehouseCloudberry integrated through gpsql mode
AgensGraphProperty graph + CypherGraph query capability inside PostgreSQL
pgEdgeSpock multi-master replicationDistributed PostgreSQL for edge scenarios
PostgreSQL forks and compatible kernels

Versions

KernelDebian / UbuntuEL
PostgreSQL / CitusPostgreSQL 18.6 (Ubuntu 18.6-1.pgdg26.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitPostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
IvorySQLPostgreSQL 18.4 (IvorySQL 5.4) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.5.0, 64-bitPostgreSQL 18.4 (IvorySQL 5.4) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.5.0, 64-bit
BabelfishBabelfish 17.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitBabelfish 17.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
PolarDBPostgreSQL 17.10 (PolarDB 17.10.1.0 build accf02e2) on x86_64-linux-gnuPostgreSQL 17.10 (PolarDB 17.10.1.0 build accf02e2) on x86_64-linux-gnu
PerconaPostgreSQL 18.4 - Percona Server for PostgreSQL 18.4.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitPostgreSQL 18.4 - Percona Server for PostgreSQL 18.4.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20250617 (Red Hat 14.3.1-2), 64-bit
OrioleDBOrioleDB 18.4 (OrioleDB 1.8-beta16) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitOrioleDB 18.4 (OrioleDB 1.8-beta16) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
OpenHaloopenHalo 14.18 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitopenHalo 14.18 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
DocumentDBPostgreSQL 18.6 (Ubuntu 18.6-1.pgdg26.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitPostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
AgensGraphPostgreSQL 17.10 (AgensGraph 2.17.0) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitPostgreSQL 17.10 (AgensGraph 2.17.0) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
pgEdgePostgreSQL 18.4 (pgEdge 5.0.10) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bitPostgreSQL 18.4 (pgEdge 5.0.10) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
CloudberryPostgreSQL 14.4 (Apache Cloudberry 2.0.0-incubating build 1) on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit

15.1 - PostgreSQL

Vanilla PostgreSQL kernel with 575 extensions

PostgreSQL is a widely used open-source database.

Pigsty installs PostgreSQL 18 by default. It supports PostgreSQL 14 ~ 18 and provides 600 PG extensions.


Quick Start

Install Pigsty using the pgsql configuration template.

./configure -c pgsql     # Use postgres kernel
./deploy.yml             # Deploy the Pigsty core chain with native PostgreSQL

Most configuration templates use the PostgreSQL kernel by default. Examples include:

  • meta : Default, postgres with core extensions (vector, postgis, timescale)
  • rich : postgres with all extensions installed
  • slim : postgres only, no monitoring infrastructure
  • ha/full : 4-node sandbox for HA demonstration
  • pgsql : minimal postgres kernel configuration example

Configuration

The native PostgreSQL kernel requires no special configuration changes:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ vector ]}
    pg_hba_rules:
      - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily
    pg_packages: [ pgsql-main, pgsql-common ]   # pg kernel and common utilities
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

Version Selection

To select a different PostgreSQL major version, use -v:

./configure -c pgsql            # Default is postgresql 18, no need to specify explicitly
./configure -c pgsql -v 18      # Explicitly use postgresql 18
./configure -c pgsql -v 17      # Use postgresql 17
./configure -c pgsql -v 16      # Use postgresql 16
./configure -c pgsql -v 15      # Use postgresql 15
./configure -c pgsql -v 14      # Use postgresql 14

If a PostgreSQL cluster is already installed, uninstall it before you install a new version:

./pgsql-rm.yml -l pg-meta # Uninstall cluster pg-meta

Extension Ecosystem

Pigsty provides PostgreSQL extensions. See the Extension Catalog for details.

15.2 - Supabase

How to self-host Supabase with Pigsty, deploy an open-source Firebase alternative with a complete backend stack in one click.

Supabase — Build in a weekend, Scale to millions

Supabase is an open-source Firebase alternative built on PostgreSQL. It provides authentication, ready-to-use APIs, edge functions, real-time subscriptions, object storage, and vector embeddings. The low-code backend platform can replace much of the backend development work. Database design and frontend knowledge can be sufficient to build a product.

Supabase’s motto is: “Build in a weekend, Scale to millions”. Its cloud service can be cost-effective for small deployments, such as 4c8g. For millions of users, consider self-hosting to meet functionality, performance, and cost requirements.

Pigsty provides an automated solution to self-host the complete Supabase stack. It includes PostgreSQL monitoring, IaC, PITR, and high availability. Compared with Supabase cloud services, this solution provides up to 600 ready-to-use PostgreSQL extensions. It can also make fuller use of the performance and cost advantages of modern hardware.

For the complete procedure, see: Supabase Self-Hosting Guide

Supabase

Quick Start

Pigsty’s default supabase.yml configuration template defines a single-node Supabase. The current application combination tracks self-hosted/v0.8.2 (commit 564eab8), with Envoy and external Pigsty-managed PostgreSQL 18 as the defaults. The template still supports PostgreSQL 15–18. See the self-hosting guide for complete versions and the upgrade procedure for existing deployments.

First, use Pigsty’s standard installation process to install the Silo and PostgreSQL instances required for Supabase:

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty
./configure -c supabase  # Important: modify passwords and other key info in config!
./deploy.yml         # Install Pigsty, PostgreSQL, and Silo

Before you deploy Supabase, adjust the Supabase parameters in pigsty.yml for your environment. In particular, change the passwords.

Then run docker.yml and app.yml to deploy the Supabase containers:

./docker.yml -l supabase  # Install Docker on the supabase group
./app.yml -l supabase     # Start stateless containers on the supabase group

In China, configure Docker mirrors or proxy servers to bypass GFW restrictions when you pull DockerHub images. Professional subscriptions include offline installation of Pigsty and Supabase without internet access.

By default, Pigsty exposes web services through Nginx on the administration node/INFRA node. Add local DNS resolution that points supa.pigsty to this node. Then open https://supa.pigsty in a browser to access the Supabase Studio management interface.

Default username and password: supabase / pigsty

demo/supabase.cast — /demo/supabase.cast


Configuration Details

./configure -c supabase generates ~/pigsty/pigsty.yml. Before running ./deploy.yml, review and update at least the passwords, keys, and domains in this file.

For complete configuration guidance, see the Supabase Self-Hosting Guide.

15.3 - Babelfish

Use Babelfish (PG17/18) in Pigsty to provide SQL Server protocol/T-SQL compatibility.

Babelfish is a PostgreSQL-based SQL Server compatibility layer, open-sourced by AWS.


Overview

Deploy Babelfish in mssql mode to add these features to PostgreSQL:

  • SQL Server wire protocol compatibility (TDS, 1433)
  • T-SQL compatibility
  • Unified integration with Pigsty capabilities (HA, backup, monitoring, IaC)

In Pigsty v4, Babelfish supports PostgreSQL 17/18. The default template uses pg_version: 17. Babelfish is part of Pigsty’s standard delivery process and supports all Linux platforms.


Current Behavior

The current behavior differs from older Babelfish/PG15 documentation as follows:

  • Supported kernel majors are PG17/18; the template defaults to PG17 (pg_version: 17)
  • Default package group: babelfish + pgsql-common + sqlcmd
  • Mainstream platform coverage:
    • OS: el8, el9, el10, d12, d13, u22, u24, u26
    • Arch: x86_64, aarch64
  • mssql template no longer requires an extra mssql repo module (defaults to node,infra,pgsql)

Older docs may still contain deprecated naming. Pigsty now consistently uses Babelfish and babelfish aliases.


Quick Start

Use the built-in Pigsty template:

./configure -c mssql [-v 17/18]
./deploy.yml

After deployment, connect directly with SQL Server clients:

sqlcmd -S <ip>,1433 -U dbuser_mssql -P DBUser.MSSQL -d mssql

Key Configuration

Core parameters in the mssql template:

pg_mode: mssql
pg_version: 17  # optional: 18
pg_packages: [ babelfish, pgsql-common, sqlcmd ]
pg_libs: 'babelfishpg_tds, pg_stat_statements, auto_explain'

pg_databases:
  - name: mssql
    baseline: mssql.sql
    extensions:
      - { name: uuid-ossp }
      - { name: babelfishpg_common }
      - { name: babelfishpg_tsql }
      - { name: babelfishpg_tds }
      - { name: babelfishpg_money }
      - { name: pg_hint_plan }
      - { name: system_stats }
      - { name: tds_fdw }
    parameters: { 'babelfishpg_tsql.migration_mode': 'multi-db' }

pg_hba_rules:
  - { user: dbuser_mssql, db: mssql, addr: intra, auth: md5, order: 525 }

pg_default_services:
  - { name: primary, port: 5433, dest: 1433 }
  - { name: replica, port: 5434, dest: 1433 }

Connectivity and Ports

Babelfish clusters expose two protocol endpoints:

  • PostgreSQL protocol: 5432
  • SQL Server protocol (TDS): 1433

Through Pigsty services, you can also use these ports:

  • 5433: fixed route to primary 1433
  • 5434: route to readable node 1433
# Primary write access
sqlcmd -S <any-node-ip>,5433 -U dbuser_mssql -P DBUser.MSSQL

# Read replica query
sqlcmd -S <any-node-ip>,5434 -U dbuser_mssql -P DBUser.MSSQL

Notes

  • Babelfish auth rules must use md5 instead of default scram-sha-256.
  • Default migration mode is multi-db; switch with babelfishpg_tsql.migration_mode if needed.
  • Some native PostgreSQL extensions cannot run directly on Babelfish kernels. Verify package availability and compatibility first.
  • For production, restrict HBA rules and network access. Do not keep the open rules from the demonstration configuration.


Available Extensions

The Babelfish kernel has 55 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
babelfishpg_common5.4.0Transact SQL Datatype Support
babelfishpg_money1.1.0babelfishpg_money
babelfishpg_tds1.0.0TDS protocol extension
babelfishpg_tsql5.4.0Transact SQL compatibility

15.4 - Percona

Percona Postgres distribution with TDE transparent encryption support

Percona Postgres is a patched Postgres kernel with pg_tde (Transparent Data Encryption) extension.

From v4.4.0, Pigsty installs Percona PostgreSQL under the private prefix /usr/pgtde-$v. Version v4.5.0 keeps this layout, with /usr/pgtde-18 for PostgreSQL 18. The pgtde package alias installs both the kernel package and its contrib package. These packages include pg_tde, PostGIS, pgvector, wal2json, pg_repack, pgaudit, and pg_stat_monitor.


Quick Start

Use Pigsty’s standard installation process with the pgtde configuration template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c pgtde     # Use percona postgres kernel
./deploy.yml             # Deploy the Pigsty core chain with Percona PostgreSQL

Configuration

To deploy a Percona cluster, adjust these parameters:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_mode: pgtde
    pg_cluster: pg-meta
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pgsql admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - name: meta
        baseline: cmdb.sql
        comment: pigsty tde database
        schemas: [pigsty]
        extensions: [ vector, postgis, pg_tde ,pgaudit, { name: pg_stat_monitor, schema: monitor } ]
    pg_hba_rules:
      - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily

    # Percona PostgreSQL TDE kernel settings
    pg_packages: [ pgtde, pgsql-common ]
    pg_libs: 'pg_tde, pgaudit, pg_stat_statements, pg_stat_monitor, auto_explain'

Pigsty’s pgsql repository module supplies the pgtde packages. The legacy percona module is not required by this template.


Available Extensions

The Percona Postgres kernel has 73 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
address_standardizer3.5.7Parses an address into its elements. Usually supports address normalization for geocoding.
address_standardizer_data_us3.5.7Address Standardizer US dataset example
pg_repack1.5.3Reorganize tables in PostgreSQL databases with minimal locks
pg_stat_monitor2.3.2Monitors PostgreSQL query performance. Based on the PostgreSQL contrib module pg_stat_statements. Provides aggregated statistics, client information, execution plans and plan details, and histogram information.
pg_tde2.2.1pg_tde access method
pgaudit18.0provides auditing functionality
postgis3.5.7PostGIS geometry and geography spatial types and functions
postgis_raster3.5.7PostGIS raster types and functions
postgis_sfcgal3.5.7PostGIS SFCGAL functions
postgis_tiger_geocoder3.5.7PostGIS tiger geocoder and reverse geocoder
postgis_topology3.5.7PostGIS topology spatial types and functions
set_user4.2.0similar to SET ROLE but with added logging
vector0.8.3vector data type and ivfflat and hnsw access methods

Key Features

  • Transparent Data Encryption: Provides data-at-rest encryption using the pg_tde extension
  • PostgreSQL 18 Compatible: Based on the Percona PostgreSQL 18 package set
  • Enterprise Extensions: Includes enterprise-grade features like pgaudit, pg_stat_monitor
  • Complete Ecosystem: Supports popular extensions like pgvector, PostGIS

Note: Currently in stable stage - thoroughly evaluate before production use.

15.5 - openHalo

MySQL-compatible Postgres 14 branch

OpenHalo is an open-source PostgreSQL kernel that provides MySQL wire-protocol compatibility.

openHalo is based on PostgreSQL 14.18 and provides wire-level compatibility with MySQL 5.7.32-log / 8.0. Pigsty delivers it through pg_mode: mysql and the openhalo package alias.

Pigsty supports OpenHalo deployment on all supported Linux platforms.


Quick Start

Use Pigsty’s standard installation flow with the mysql template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c mysql    # Use MySQL (openHalo) template
./deploy.yml            # Install (change passwords in pigsty.yml before production use)

Cluster Config

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - {name: postgres, extensions: [ aux_mysql ]} # mysql-compatible database
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # full backup at 1 AM daily

    # OpenHalo specific settings
    pg_mode: mysql
    pg_version: 14
    pg_packages: [ openhalo, pgsql-common ]

The OpenHalo extension aux_mysql provides functions and types for MySQL compatibility. Enable it in the postgres database for full compatibility behavior.

  • aux_mysql 1.5: MySQL Supplementary Extension
  • /usr/halo-14/share/postgresql/extension/aux_mysql.control
  • $libdir/mysm, mysm.so

Usage

MySQL connections use the postgres database. A MySQL database maps to a PostgreSQL schema. Thus, use mysql selects the mysql schema in the postgres database.

Use the PostgreSQL usernames and passwords for MySQL access.

Client Access

OpenHalo listens on port 3306 for MySQL wire protocol clients.

Pigsty’s conf/mysql installs a MySQL client by default.

mysql -h 127.0.0.1 -u dbuser_dba

OpenHalo upstream currently reports that Navicat works normally on this port. IntelliJ DataGrip may fail.


Compatibility Parameters

Pigsty sets database_compat_mode to mysql by default. To adjust compatibility behavior, use settings such as:

mysql.listener_on = true                        # enable MySQL listener; restart required
mysql.port = 3306                               # second_port for MySQL mode; restart required
mysql.halo_mysql_version = '5.7.32-log'         # restart required
mysql.ci_collation = true                       # restart required
mysql.explicit_defaults_for_timestamp = false   # restart required
mysql.auto_rollback_tx_on_error = false         # restart required

Patch Notes

Pigsty packages the OpenHalo kernel from HaloTech-Co-Ltd/openHalo with these adjustments:

  • Restore default database name from halo0root to postgres
  • Remove 1.0. prefix in the default version string, keeping 14.18
  • Adjust default config to enable MySQL compatibility and listen on 3306

Pigsty does not provide warranty coverage for OpenHalo kernel behavior. Contact the upstream vendor about kernel-specific issues.

Warning: This kernel is currently in beta1. Evaluate its risks carefully before production use.


Available Extensions

The OpenHalo kernel has 59 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
aux_mysql1.5MySQL Supplementary Extension
hstore_plpython2u1.0transform between hstore and plpython2u
hstore_plpythonu1.0transform between hstore and plpythonu
jsonb_plpython2u1.0transform between jsonb and plpython2u
jsonb_plpythonu1.0transform between jsonb and plpythonu
ltree_plpython2u1.0transform between ltree and plpython2u
ltree_plpythonu1.0transform between ltree and plpythonu

15.6 - OrioleDB

Next-generation OLTP engine for PostgreSQL

OrioleDB is a PostgreSQL storage engine extension. It claims to provide 4x OLTP performance and to eliminate xid wraparound and table bloat issues. It also claims to provide cloud-native storage, with data stored in S3.

Pigsty ships OrioleDB as a patched PostgreSQL kernel plus the OrioleDB extension.

You can run OrioleDB as an RDS with Pigsty. Current packages support PostgreSQL 16, 17, and 18 on supported Linux platforms. Set pg_mode to oriole to use the /usr/oriole-$v installation path. The orioledb package alias resolves to versioned kernel packages, such as orioledb-16, orioledb-17, and orioledb-18. The current Pigsty package line is OrioleDB 1.8 beta16.


Quick Start

Follow Pigsty’s standard installation process using the oriole configuration template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c oriole    # Use OrioleDB configuration template
./deploy.yml             # Install Pigsty with OrioleDB

For production deployment, change the password parameters in pigsty.yml before you run the install playbook.


Configuration

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty], extensions: [orioledb]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily

    # OrioleDB specific settings
    pg_mode: oriole                                         # oriole compatibility mode
    pg_version: 18                                          # OrioleDB supports PG 16, 17, and 18
    pg_packages: [ orioledb, pgsql-common ]                 # Install OrioleDB kernel
    pg_libs: 'orioledb, pg_stat_statements, auto_explain'   # Load OrioleDB extension

Usage

To use OrioleDB, install the orioledb package alias. Pigsty uses pg_version to select the PG16, PG17, or PG18 OrioleDB kernel package.

Initialize TPC-B-like tables with pgbench using 100 warehouses:

pgbench -is 100 meta
pgbench -nv -P1 -c10 -S -T1000 meta
pgbench -nv -P1 -c50 -S -T1000 meta
pgbench -nv -P1 -c10    -T1000 meta
pgbench -nv -P1 -c50    -T1000 meta

Next, rebuild these tables with the orioledb storage engine and compare performance:

-- Create OrioleDB tables
CREATE TABLE pgbench_accounts_o (LIKE pgbench_accounts INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_branches_o (LIKE pgbench_branches INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_history_o (LIKE pgbench_history INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_tellers_o (LIKE pgbench_tellers INCLUDING ALL) USING orioledb;

-- Copy data from regular tables to OrioleDB tables
INSERT INTO pgbench_accounts_o SELECT * FROM pgbench_accounts;
INSERT INTO pgbench_branches_o SELECT * FROM pgbench_branches;
INSERT INTO pgbench_history_o SELECT  * FROM pgbench_history;
INSERT INTO pgbench_tellers_o SELECT * FROM pgbench_tellers;

-- Drop original tables and rename OrioleDB tables
DROP TABLE pgbench_accounts, pgbench_branches, pgbench_history, pgbench_tellers;
ALTER TABLE pgbench_accounts_o RENAME TO pgbench_accounts;
ALTER TABLE pgbench_branches_o RENAME TO pgbench_branches;
ALTER TABLE pgbench_history_o RENAME TO pgbench_history;
ALTER TABLE pgbench_tellers_o RENAME TO pgbench_tellers;

Key Features

  • No XID Wraparound: Eliminates transaction ID wraparound maintenance
  • No Table Bloat: Advanced storage management prevents table bloat
  • Cloud Storage: Native support for S3-compatible object storage
  • OLTP Optimized: Designed for transactional workloads
  • Improved Performance: Better space utilization and query performance

Note: Currently in Beta stage - thoroughly evaluate before production use.


Available Extensions

The OrioleDB kernel has 53 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
orioledb1.8OrioleDB – the next generation transactional engine

15.7 - Cloudberry

Use the Cloudberry open-source MPP data warehouse kernel in Pigsty and manage nodes, monitoring, and configuration through gpsql mode.

Cloudberry is an open-source MPP data warehouse kernel derived from the Greenplum ecosystem. It supports large-scale parallel analytics workloads.


Overview

In Pigsty, Cloudberry uses gpsql mode and shares the same identity model, monitoring logic, and directory conventions as Greenplum / MatrixDB.

  • Kernel package: cloudberry
  • Mode identifier: pg_mode: gpsql
  • Role flag: gp_role: master | segment
  • Current repo version: Cloudberry 2.1.0
  • Current main package version: DEB 2.1.0-2PIGSTY, RPM 2.1.0-3PIGSTY
  • Default binary directory: /usr/cloudberry

Pigsty currently provides package delivery, node management, monitoring integration, access control, and configuration management for Cloudberry. For MPP cluster initialization, scale-out, rebalance, and other upstream operations, you should use the official Cloudberry tools.

The current Pigsty repository provides cloudberry, cloudberry-backup, and cloudberry-pxf packages for both DEB and RPM platforms.


Installation

There is no standalone cloudberry deployment template yet. The usual procedure is:

  1. Enroll the target nodes into Pigsty.
  2. Install the cloudberry kernel package.
  3. Describe the coordinator / segment topology with gpsql mode.
  4. Use Pigsty to unify monitoring, accounts, access control, and backup integration.

If you only need to install the kernel package on a node:

./node.yml -t node_repo    -e '{"node_repo_modules":"local,node,pgsql"}'
./node.yml -t node_install -e '{"node_packages":["cloudberry"]}'

For an existing Cloudberry cluster, usually keep the original initialization procedure. Add the Pigsty inventory and monitoring configuration incrementally.


Configuration

Cloudberry uses gpsql mode. There is no dedicated cloudberry mode. In addition to standard PostgreSQL identity parameters, you need pg_shard and gp_role. To label shard groups explicitly, you can also add pg_group.

This example shows a minimal topology:

all:
  children:
    cb-mdw:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: cb-mdw
        pg_mode: gpsql
        pg_shard: cb
        gp_role: master
        pg_packages: [ cloudberry, pgsql-common ]

    cb-sdw:
      hosts:
        10.10.10.11:
          nodename: cb-sdw-1
          pg_instances:
            6000: { pg_cluster: cb-seg1, pg_seq: 1, pg_role: primary, pg_exporter_port: 9633 }
        10.10.10.12:
          nodename: cb-sdw-2
          pg_instances:
            6000: { pg_cluster: cb-seg2, pg_seq: 1, pg_role: primary, pg_exporter_port: 9633 }
      vars:
        pg_cluster: cb-sdw
        pg_mode: gpsql
        pg_shard: cb
        gp_role: segment
        pg_preflight_skip: true
        pg_packages: [ cloudberry, pgsql-common ]
        pg_exporter_config: pg_exporter_basic.yml
        pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'

Apply these role settings:

  • Use gp_role: master for the coordinator / master node. Applications usually connect to this node.
  • gp_role: segment nodes usually need pg_exporter to connect in utility mode for monitoring.

Client Access

Cloudberry uses the PostgreSQL wire protocol for application and BI access. Most PostgreSQL-compatible clients, drivers, and BI tools can connect without special configuration.

Use these access guidelines:

  • Applications and analytics queries should connect to the master / coordinator, not directly to segment nodes.
  • Segment nodes are better treated as data/compute shards and monitoring targets.
  • If you want a unified access endpoint, you can still use Pigsty’s HAProxy / PgBouncer / DNS service abstractions.

Extensions and Ecosystem

Cloudberry differs from native PostgreSQL with added extensions. For extension packages in Pigsty, consider these two categories:

  • Pure SQL objects or components with weak ABI coupling are usually easier to adapt.
  • Extensions that depend on PGXS or the kernel C ABI often need separate validation. They may also need recompilation with the Cloudberry version and toolchain.

If your workload requires postgis, vector extensions, FDWs, auditing, or custom C extensions, first validate them on the target Cloudberry version. Do not copy a native PostgreSQL extension list without this check.


Notes

  • Cloudberry currently has no dedicated Pigsty template. You should define its configuration manually with gpsql mode.
  • Pigsty currently provides packages, configuration, and monitoring. Use the official Cloudberry tools for MPP initialization and scale-out.
  • Cloudberry is an MPP distributed kernel. Native PostgreSQL operating assumptions may not apply to every Patroni / PgBouncer / PgBackRest node role.
  • If you need horizontal PostgreSQL scaling rather than a full MPP warehouse, Citus is usually the better first choice.

15.8 - AgensGraph

Use the AgensGraph (PG17) graph database kernel in Pigsty to get property graph and Cypher/SQL hybrid query capabilities within the PostgreSQL ecosystem.

AgensGraph is a property graph database kernel built on PostgreSQL. It supports openCypher queries and mixed Cypher/SQL workflows.


Overview

Pigsty integrates AgensGraph through pg_mode: agens while preserving most of the standard PostgreSQL operational model.

  • Kernel package: agensgraph
  • Mode identifier: pg_mode: agens
  • Current template version: AgensGraph 2.17.0
  • Current version string: PostgreSQL 17.10 (AgensGraph 2.17.0)
  • Built-in template: agens
  • Typical use cases: graph relationship analysis, path queries, knowledge graphs, and risk/association analysis layered onto relational data

AgensGraph uses the PostgreSQL wire protocol. Standard PostgreSQL clients, drivers, and connection pools can connect directly. AgensGraph adds graph objects, Cypher syntax, and the agtype data type to the database.


Installation

Use the built-in Pigsty template:

./configure -c agens
./deploy.yml

The agens template automatically enables pg_mode: agens and installs the agensgraph kernel package. After deployment, verify the kernel version:

psql -d meta -c "SELECT version();"

Configuration

Key configuration for AgensGraph in Pigsty:

all:
  vars:
    node_repo_modules: node,infra,pgsql
    pg_version: 17

  children:
    pg-meta:
      vars:
        pg_mode: agens
        pg_packages: [ agensgraph, pgsql-common ]

AgensGraph does not require a special preload stack, unlike pgEdge or Babelfish. Most standard Pigsty procedures for HA, backup, monitoring, access control, and IaC remain applicable. If graph traversal and complex path queries dominate the workload, examine work_mem, shared_buffers, and planner cost settings. Do not assume that the default OLTP settings are suitable.


Usage

After you connect to the database, usually create a graph first and set graph_path:

CREATE GRAPH g;
SET graph_path = g;

Create labels, vertices, and edges:

CREATE VLABEL person;
CREATE ELABEL knows;

CREATE (:person {name: 'Jack'});
CREATE (:person {name: 'Emily'})-[:knows]->(:person {name: 'Tom'});

Run graph queries and updates:

MATCH (:person {name: 'Emily'})-[:knows]->(v:person)
RETURN v.name;

MATCH (v:person {name: 'Jack'})
SET v.age = '24';

To call Cypher from within SQL, use the cypher() function:

SELECT *
FROM cypher('g', $$ MATCH (v:person) RETURN v.name $$) AS (name agtype);

Projects commonly combine relational tables, graph labels, and Cypher queries. Transactions, privileges, and backup procedures still follow PostgreSQL. Graph analysis uses AgensGraph graph objects and the cypher() interface.


Notes

  • AgensGraph currently uses the PG17-compatible line. Do not assume that PG18 extensions are available.
  • The default agens template uses one node for quick validation. Production deployments should use an HA topology.
  • Third-party PostgreSQL extensions may be incompatible with AgensGraph. Verify compatibility first.
  • Graph objects and relational objects can coexist in one database. For production, clear database or naming conventions usually help you distinguish them.
  • Tune memory and cost parameters for the size of your graph model. Check that the defaults are suitable before you use them.
  • For compatibility or semantic issues with the AgensGraph kernel, consult the official manual and upstream issues first.


Available Extensions

The AgensGraph kernel has 60 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
meta1.0Utility functions for agensgraph

15.9 - pgEdge

Use the pgEdge (PG15-18) kernel in Pigsty to build distributed PostgreSQL for edge scenarios on top of Spock multi-master logical replication.

pgEdge is a distributed PostgreSQL distribution for edge deployments. It uses Spock multi-master logical replication.


Overview

Pigsty integrates pgEdge through pg_mode: pgedge and delivers it through the standard PostgreSQL cluster workflow:

  • pgedge: a PG15, PG16, PG17, and PG18 compatible kernel; the template defaults to PG18
  • spock: Active-active multi-master logical replication
  • snowflake: Distributed unique sequences
  • lolor: Large object logical replication compatibility layer

The current Pigsty repository supplies the versioned kernel packages pgedge-15, pgedge-16, pgedge-17, and pgedge-18. The template defaults to pg_version: 18. The pgedge-$v kernel package includes the control files, SQL files, and shared libraries for spock, snowflake, and lolor. Thus, these extensions are no longer separate packages in pg_extensions. pgEdge supports the PostgreSQL wire protocol, so psql, JDBC/ODBC, DBeaver, and similar tools work as usual.

First validate the kernel on one node. Then expand to a replication topology with multiple nodes. The template configures the kernel, extensions, monitoring, backup, and access control. Design the multi-master topology for your workload consistency requirements and conflict strategy.


Installation

Use the built-in Pigsty template:

./configure -c pgedge
./deploy.yml

The template installs spock, snowflake, and lolor in the meta database. After deployment, verify the kernel and extensions:

psql -d meta -c "SELECT version();"
psql -d meta -c "SELECT extname, extversion FROM pg_extension WHERE extname IN ('spock','snowflake','lolor') ORDER BY 1;"

For the full template and parameters, see: pgedge config template.


Configuration

Key parameters in the pgedge template (matching conf/pgedge.yml):

pg_mode: pgedge
pg_version: 18
pg_packages: [ pgedge, pgsql-common ]
pg_libs: 'spock, lolor, pg_stat_statements, auto_explain'
pg_databases:
  - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [spock, snowflake, lolor] }

For a future multi-node multi-master topology, we recommend explicit settings for logical replication capacity and snowflake.node:

pg_parameters:
  wal_level: logical
  max_replication_slots: 16
  max_wal_senders: 16
  'snowflake.node': 1

snowflake.node must be unique on every writable node. Otherwise, distributed IDs will collide.


Usage

First validate the kernel on one node. Then expand to a Spock replication topology with multiple nodes.

If you need these capabilities in a business database as well, create the extensions first:

CREATE EXTENSION IF NOT EXISTS spock;
CREATE EXTENSION IF NOT EXISTS snowflake;
CREATE EXTENSION IF NOT EXISTS lolor;

Then use the Spock SQL API or the pgEdge CLI to create nodes, replication sets, and subscriptions. If your schema uses serial or identity, plan the snowflake sequence migration before you enable multi-master writes. Otherwise, primary key collisions between nodes are likely.


Notes

  • Configure pgEdge replication for each database. There is no instance-wide switch to enable multi-master replication for all databases.
  • Replicated tables should have a PRIMARY KEY or an appropriate REPLICA IDENTITY.
  • UNLOGGED and TEMPORARY tables do not participate in Spock logical replication.
  • Spock configuration and operations usually require superuser privileges. Production deployments should define privilege boundaries clearly.
  • If your workload requires large object replication, use lolor explicitly. Do not assume that native large objects will replicate correctly.
  • Before you configure multi-master replication across regions, evaluate network latency, conflict handling, and the write model.


Available Extensions

The pgEdge kernel has 63 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
lolor1.2.2Large Objects support for logical replication
snowflake2.5.0Snowflake style IDs for PostgreSQL
spock5.0.10PostgreSQL Logical Replication

15.10 - DocumentDB

DocumentDB and FerretDB provide MongoDB wire-protocol compatibility

Microsoft maintains DocumentDB, an open-source PostgreSQL document database extension. FerretDB is a stateless protocol translation proxy built on DocumentDB. Together, they provide a MongoDB wire-compatible endpoint for a standard PostgreSQL kernel. Applications can connect directly with MongoDB drivers. FerretDB translates their requests into PostgreSQL operations.

This stack is not a standalone PostgreSQL fork. The data layer uses native PostgreSQL 16–18, which the standard PGSQL module manages. PostgreSQL provides persistence, transactions, high availability, backup, monitoring, and access control. FerretDB runs as a Pigsty Docker APP and handles only protocol translation.

Pigsty is a FerretDB community partner. It provides binary packages for FerretDB and the DocumentDB extensions. Use the mongo configuration template to deploy the complete stack.


Quick Start

Use Pigsty’s standard installation flow with the mongo configuration template:

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c mongo    # Use the Mongo (DocumentDB + FerretDB) configuration template
./deploy.yml            # Install; change passwords in pigsty.yml first for production
./docker.yml -l pg-meta # Install Docker on the pg-meta node
./app.yml -l pg-meta    # Deploy the FerretDB Docker APP

By default, FerretDB listens on the local loopback address at port 27017. Connect with mongosh or any MongoDB-compatible client:

mongosh 'mongodb://mongod:[email protected]:27017/'

Configuration

Source: pigsty/conf/mongo.yml. See the Mongo configuration template documentation for the complete template.

The key PostgreSQL settings are the documentdb extension and its preload libraries, plus the backend superuser used by FerretDB:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - { name: mongod ,password: DBUser.Mongo ,superuser: true ,comment: FerretDB backend user }
    pg_databases:
      - { name: postgres, extensions: [ documentdb, postgis, vector, pg_cron, rum ]}
    pg_extensions: [ documentdb, postgis, pgvector, pg_cron, rum ]
    pg_libs: 'pg_documentdb, pg_documentdb_core, pg_documentdb_extended_rum, pg_cron, pg_stat_statements, auto_explain'

FerretDB runs as a Docker APP. Set its configuration overrides under apps.ferretdb.conf. The container connects to the local primary service on port 5436 through host.docker.internal:

docker_enabled: true
app: ferretdb
apps:
  ferretdb:
    conf:
      FERRETDB_IMAGE: ghcr.io/ferretdb/ferretdb:2.7.0
      FERRETDB_POSTGRESQL_URL: 'postgres://mongod:[email protected]:5436/postgres?pool_min_conns=1&pool_max_conns=20'
      FERRETDB_BIND_ADDR: 127.0.0.1
      FERRETDB_PORT: 27017
      FERRETDB_AUTH: true
      FERRETDB_TELEMETRY: disabled

High Availability

FerretDB is fully stateless, so its HA topology follows the standard PostgreSQL cluster pattern. The template includes a commented three-node pg-mongo example. Each node runs a FerretDB container on local port 27018. HAProxy routes access to these containers through the floating endpoint 10.10.10.4:27017 (mongo.pigsty).

Patroni and etcd manage PostgreSQL failover. The Mongo endpoint automatically recovers after the primary switches.


Notes

  • FerretDB enables authentication by default (FERRETDB_AUTH: true). It does not yet implement MongoDB authorization roles. PostgreSQL users and HBA rules control access.
  • Client-side MongoDB TLS is disabled by default. The Mongo endpoint is not exposed to the network. Change FERRETDB_BIND_ADDR only when remote access is required.
  • The backend cluster uses standard PostgreSQL parameters, playbooks, and dashboards. There is no independent FERRET module or mongo_* parameter group.
  • Repeat an authenticated CRUD smoke test after upgrading FerretDB or DocumentDB.

15.11 - Citus

Deploy high-availability Citus clusters with Pigsty. Scale PostgreSQL across shards and accelerate OLTP/OLAP queries.

Pigsty natively supports Citus. This is a distributed horizontal scaling extension based on the native PostgreSQL kernel.

Citus

Installation

Citus is a PostgreSQL extension plugin. Use the standard extension installation procedure to install and enable it on a native PostgreSQL cluster.

./pgsql.yml -t pg_extension -e '{"pg_extensions":["citus"]}'

Configuration

To define a Citus cluster, set these parameters:

  • pg_mode must be set to citus instead of the default pgsql
  • You must define the shard name pg_shard and shard number pg_group on each shard cluster
  • You must define pg_primary_db to specify the database managed by Patroni
  • You can run administration commands as postgres from pg_dbsu, instead of the default pg_admin_username. If you use this account, you must set pg_dbsu_password to a non-empty plaintext password.

You also need HBA rules that allow SSL access from localhost and other data nodes.

You can define each Citus cluster as a separate group, as with standard PostgreSQL clusters. The current complete template is conf/ha/citus.yml:

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # Global parameters for all Citus clusters
    pg_mode: citus                    # pgsql cluster mode must be set to: citus
    pg_shard: pg-citus                # citus horizontal shard name: pg-citus
    pg_primary_db: meta               # citus database name: meta
    pg_dbsu_password: DBUser.Postgres # If using dbsu, you need to configure a password for it
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

You can also specify identity parameters for all Citus cluster members within a single group, as shown in conf/ha/citus.yml:

#==========================================================#
# pg-citus: 10 node citus cluster (5 x primary-replica pair)
#==========================================================#
pg-citus: # citus group
  hosts:
    10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
  vars:
    pg_mode: citus                    # pgsql cluster mode: citus
    pg_shard: pg-citus                # citus shard name: pg-citus
    pg_primary_db: test               # primary database used by citus
    pg_dbsu_password: DBUser.Postgres # all dbsu password access for citus cluster
    pg_vip_enabled: true
    pg_vip_interface: auto
    pg_extensions: [ 'citus postgis timescaledb pgvector' ]
    pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain' # citus will be added by patroni automatically
    pg_users: [ { name: test ,password: test ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: test ,owner: test ,extensions: [ { name: citus }, { name: postgis } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 10.10.10.0/24 ,auth: trust ,title: 'trust citus cluster members'        }
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

Usage

Connect to any node as you would connect to a regular cluster:

pgbench -i postgres://test:test@pg-citus0/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus0/test

By default, changes to one shard affect only that cluster. These changes do not synchronize to other shards.

To distribute writes across all shards, use Citus API functions to mark tables as one of these types:

  • Distributed tables (automatic partitioning, requires specifying partition key)
  • Reference tables (full replication: does not require specifying partition key)

From Citus 11.2, any Citus database node can act as coordinator. Thus, any primary node can write:

psql -h pg-citus0 -d test -c "SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table('public.pgbench_accounts');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table('public.pgbench_branches');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_history');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_tellers');"

After distributing the tables, you can also access them on other nodes:

psql -h pg-citus1 -d test -c '\dt+'

For example, this full table scan shows a distributed execution plan:

vagrant@meta-1:~$ psql -h pg-citus3 -d test -c 'explain select * from pgbench_accounts'
                                               QUERY PLAN
---------------------------------------------------------------------------------------------------------
 Custom Scan (Citus Adaptive)  (cost=0.00..0.00 rows=100000 width=352)
   Task Count: 32
   Tasks Shown: One of 32
   ->  Task
         Node: host=10.10.10.52 port=5432 dbname=test
         ->  Seq Scan on pgbench_accounts_102008 pgbench_accounts  (cost=0.00..81.66 rows=3066 width=97)
(6 rows)

You can initiate writes from several different primary nodes:

pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus1/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus2/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus3/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus4/test

When a node fails, Patroni promotes the standby node. The standby automatically takes over through native high availability support.

test=# select * from  pg_dist_node;
 nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
      1 |       0 | 10.10.10.51 |     5432 | default  | t           | t        | primary  | default     | t              | f
      2 |       2 | 10.10.10.54 |     5432 | default  | t           | t        | primary  | default     | t              | t
      5 |       1 | 10.10.10.52 |     5432 | default  | t           | t        | primary  | default     | t              | t
      3 |       4 | 10.10.10.58 |     5432 | default  | t           | t        | primary  | default     | t              | t
      4 |       3 | 10.10.10.56 |     5432 | default  | t           | t        | primary  | default     | t              | t

15.12 - IvorySQL

Use HighGo’s open-source IvorySQL kernel to achieve Oracle syntax/PLSQL compatibility based on PostgreSQL clusters.

IvorySQL is an open-source PostgreSQL kernel fork that aims to provide Oracle compatibility.


Overview

The Pigsty PGSQL repository provides IvorySQL 5.4 packages compatible with PostgreSQL 18.4. Packages cover the currently supported EL, Debian, and Ubuntu platforms on both architectures. Online installation uses Pigsty’s pgsql repository. The Professional Edition also provides offline delivery for these platforms.

IvorySQL

Pigsty’s ivorysql package alias points to IvorySQL 5, which is compatible with PostgreSQL 18. Platform variables under roles/node_id/vars/ map the alias to package names. For example, EL uses ivorysql5, and Debian/Ubuntu uses ivorysql-5.

IvorySQL 3.3 was the last version to support EL7. It corresponds to PostgreSQL 16.3. IvorySQL 4.4 is the last version based on PostgreSQL 17.


Installation

Install with Pigsty’s built-in ivory configuration template:

./configure -c ivory
./deploy.yml

Configuration

Set these parameters for an IvorySQL database cluster:

#----------------------------------#
# Ivory SQL Configuration
#----------------------------------#
node_repo_modules: node,infra,pgsql       # use Pigsty node/infra/pgsql repos
pg_mode: ivory                    # IvorySQL Oracle Compatible Mode
pg_packages: [ ivorysql, pgsql-common ]
pg_libs: 'liboracle_parser, pg_stat_statements, auto_explain'
pg_extensions: [ ]                # do not install any vanilla postgresql extensions

For Oracle compatibility mode, dynamically load the liboracle_parser extension plugin.


Client Access

IvorySQL 5 is equivalent to PostgreSQL 18. Any client tool that supports the PostgreSQL wire protocol can connect to IvorySQL clusters.


Available Extensions

The IvorySQL kernel has 95 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
address_standardizer3.5.4Parses an address into its elements. Usually supports address normalization for geocoding.
address_standardizer_data_us3.5.4Address Standardizer US dataset example
age1.7.0AGE database extension
ddlx0.31DDL eXtractor functions
gb18030_20221.0support gb18030 2022 with extension
http1.7HTTP client for PostgreSQL, allows web page retrieval inside the database.
ivorysql_ora1.0Oracle Compatible extenison on Postgres Database
ora_btree_gin1.0support for indexing oracle datatypes in GIN
ora_btree_gist1.0support for oracle indexing common datatypes in GiST
pg_bigm1.2text similarity measurement and index searching based on bigrams
pg_cron1.6Job scheduler for PostgreSQL
pg_curl2.4PostgreSQL cURL allows most curl actions, including data transfer with URL syntax via HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP
pg_get_functiondef1.0Get function’s definition
pg_hint_plan1.8.0optimizer hints for PostgreSQL
pg_jieba1.1.1a parser for full-text search of Chinese
pg_partman5.3.1Extension to manage partitioned tables by time or ID
pg_show_plans2.1show query plans of all currently running SQL statements
pg_stat_monitor2.3Monitors PostgreSQL query performance. Based on the PostgreSQL contrib module pg_stat_statements. Provides aggregated statistics, client information, execution plans and plan details, and histogram information.
pg_textsearch0.1.0Full-text search with BM25 ranking
pgagent4.2A PostgreSQL job scheduler
pgaudit18.0provides auditing functionality
pgroonga4.0.4Super fast and all languages supported full text search index based on Groonga
pgroonga_database4.0.4PGroonga database management module
pgrouting3.8.0pgRouting Extension
plisql1.0PL/iSQL procedural language
plpgsql_check2.8extended check for plpgsql functions
postgis3.5.4PostGIS geometry and geography spatial types and functions
postgis_raster3.5.4PostGIS raster types and functions
postgis_sfcgal3.5.4PostGIS SFCGAL functions
postgis_tiger_geocoder3.5.4PostGIS tiger geocoder and reverse geocoder
postgis_topology3.5.4PostGIS topology spatial types and functions
redis_fdw1.0Foreign data wrapper for querying a Redis server
system_stats3.0EnterpriseDB system statistics for PostgreSQL
vector0.8.1vector data type and ivfflat and hnsw access methods
zhparser2.3a parser for full-text search of Chinese

Pigsty does not assume any warranty responsibility for use of the IvorySQL kernel. Contact the original vendor about issues or requirements for this kernel.

15.13 - PolarDB PG

Using Alibaba Cloud’s open-source PolarDB for PostgreSQL kernel to provide domestic innovation qualification support, with Oracle RAC-like user experience.

Overview

Use Pigsty and PolarDB to create PostgreSQL clusters with “domestic innovation qualification” credentials.

PolarDB for PostgreSQL now uses PostgreSQL 17 as its base. The Pigsty polar template, default path, and extension notes now use PG17. Any client tool that supports the PostgreSQL wire protocol can connect to PolarDB clusters.

Pigsty’s PGSQL repository provides PolarDB PG open-source installation packages. Pigsty does not download these packages to the local software repository during installation.

PolarDB for PostgreSQL

Installation

Use the built-in Pigsty template:

./configure -c polar
./deploy.yml

Change Summary

From Pigsty v4.4, the PolarDB PG kernel uses packages that Pigsty builds and maintains. The main changes are:

ItemOld docs / old defaultCurrent
Kernel baselinePostgreSQL 15PostgreSQL 17
Default PolarDB path/u01/polardb_pg/usr/polar-17
Supported architecturesx86_64x86_64, aarch64
Available extensionsOld docs said 61pg_available_extensions returns 93; 34 remain after filtering contrib
Replication user requirementreplicator must be SUPERUSERunchanged

Configuration

Set these parameters for PolarDB database clusters:

#----------------------------------#
# PGSQL & PolarDB
#----------------------------------#
pg_version: 17
pg_mode: polar
pg_packages: [ polardb, pgsql-common ]
pg_exporter_exclude_database: 'template0,template1,postgres,polardb_admin'
pg_default_roles:
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator   ,superuser: true  ,replication: true ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator } # <- superuser is required for replication
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

The default installation directory for the polar kernel is now /usr/polar-17. PolarDB PG still requires the replicator replication user to be SUPERUSER. Native PostgreSQL does not have this requirement.


Available Extensions

The PolarDB PG kernel has 93 available extensions. The following table lists the extra extensions, excluding bundled PG Contrib extensions:

ExtensionVersionDescription
hll2.18type for storing hyperloglog data
ip4r2.4
log_fdw1.4foreign-data wrapper for Postgres log file access
pase0.0.1ant ai similarity search
pg_bigm1.2text similarity measurement and index searching based on bigrams
pg_cron1.5Job scheduler for PostgreSQL
pg_cron_preload1.0polardb pg extend catalog
pg_hint_plan1.7.0optimizer hints for PostgreSQL
pg_jieba1.1.0a parser for full-text search of Chinese
pg_partman5.2.4Extension to manage partitioned tables by time or ID
pg_profile4.10PostgreSQL load profile repository and report builder
pg_repack1.5.1-1Reorganize tables in PostgreSQL databases with minimal locks
pg_similarity1.0support similarity queries
pg_squeeze1.9A tool to remove unused space from a relation.
pg_stat_kcache2.3.0Kernel statistics gathering
pgaudit17.1provides auditing functionality
pgtap1.3.3Unit testing for PostgreSQL
pldbgapi1.1server-side support for debugging PL/pgSQL functions
polar_advisor1.1polar_advisor
polar_feature_utils1.0PolarDB feature utilization
polar_io_stat1.0polar io stat in multi dimension
polar_monitor1.3monitor functions for PolarDB
polar_monitor_preload1.0examine the polardb information
polar_parameter_manager1.2Extension to select parameters for manger.
polar_proxy_utils1.0Extension to provide operations about proxy.
polar_resource_manager1.0a background process that forcibly frees user session process memory
polar_smgrperf1.0smgr perf test extension
polar_tde_utils1.0Internal extension for TDE
polar_vfs1.0polar virtual file system for different storage
polar_worker1.1polar_worker
prefix1.2.0Prefix Range module for PostgreSQL
roaringbitmap0.5support for Roaring Bitmaps
sequential_uuids1.0.3generator of sequential UUIDs
varbitx1.1varbit functions pack

15.14 - PolarDB Oracle

Using Alibaba Cloud’s commercial PolarDB for Oracle kernel (closed source, PG14, only available in special enterprise edition customization)

Use Pigsty to create PolarDB for Oracle clusters with “domestic innovation qualification” credentials.

The Security and Reliability Evaluation Results Announcement (No. 1, 2023) lists PolarDB v2.0 in Appendix 3, Centralized Database. It identifies PolarDB v2.0 as an autonomous, controllable, secure, and reliable domestic innovation database.

PolarDB for Oracle is an Oracle-compatible version based on PolarDB for PostgreSQL. Both use the same kernel. The --compatibility-mode parameter selects the mode.

We work with the Alibaba Cloud kernel team to provide a complete database solution with the PolarDB v2.0 kernel and Pigsty. For inquiries, contact sales. You can also purchase it on Alibaba Cloud Marketplace.

The PolarDB for Oracle kernel is currently only available on EL7 (CentOS 7) systems.

PolarDB for Oracle

Extensions

The PolarDB 2.0 kernel in Oracle compatibility mode currently includes these 188 extension plugins:

namedefault_versioncomment
cube1.5data type for multidimensional cubes
ip4r2.4NULL
adminpack2.1administrative functions for PostgreSQL
dict_xsyn1.0text search dictionary template for extended synonym processing
amcheck1.4functions for verifying relation integrity
autoinc1.0functions for autoincrementing fields
hstore1.8data type for storing sets of (key, value) pairs
bloom1.0bloom access method - signature file based index
earthdistance1.1calculate great-circle distances on the surface of the Earth
hstore_plperl1.0transform between hstore and plperl
bool_plperl1.0transform between bool and plperl
file_fdw1.0foreign-data wrapper for flat file access
bool_plperlu1.0transform between bool and plperlu
fuzzystrmatch1.1determine similarities and distance between strings
hstore_plperlu1.0transform between hstore and plperlu
btree_gin1.3support for indexing common datatypes in GIN
hstore_plpython2u1.0transform between hstore and plpython2u
btree_gist1.6support for indexing common datatypes in GiST
hll2.17type for storing hyperloglog data
hstore_plpython3u1.0transform between hstore and plpython3u
citext1.6data type for case-insensitive character strings
hstore_plpythonu1.0transform between hstore and plpythonu
hypopg1.3.1Hypothetical indexes for PostgreSQL
insert_username1.0functions for tracking who changed a table
dblink1.2connect to other PostgreSQL databases from within a database
decoderbufs0.1.0Logical decoding plugin that delivers WAL stream changes using a Protocol Buffer format
intagg1.1integer aggregator and enumerator (obsolete)
dict_int1.0text search dictionary template for integers
intarray1.5functions, operators, and index support for 1-D arrays of integers
isn1.2data types for international product numbering standards
jsonb_plperl1.0transform between jsonb and plperl
jsonb_plperlu1.0transform between jsonb and plperlu
jsonb_plpython2u1.0transform between jsonb and plpython2u
jsonb_plpython3u1.0transform between jsonb and plpython3u
jsonb_plpythonu1.0transform between jsonb and plpythonu
lo1.1Large Object maintenance
log_fdw1.0foreign-data wrapper for csvlog
ltree1.2data type for hierarchical tree-like structures
ltree_plpython2u1.0transform between ltree and plpython2u
ltree_plpython3u1.0transform between ltree and plpython3u
ltree_plpythonu1.0transform between ltree and plpythonu
moddatetime1.0functions for tracking last modification time
old_snapshot1.0utilities in support of old_snapshot_threshold
oracle_fdw1.2foreign data wrapper for Oracle access
oss_fdw1.1foreign-data wrapper for OSS access
pageinspect2.1inspect the contents of database pages at a low level
pase0.0.1ant ai similarity search
pg_bigm1.2text similarity measurement and index searching based on bigrams
pg_freespacemap1.2examine the free space map (FSM)
pg_hint_plan1.4controls execution plan with hinting phrases in comment of special form
pg_buffercache1.5examine the shared buffer cache
pg_prewarm1.2prewarm relation data
pg_repack1.4.8-1Reorganize tables in PostgreSQL databases with minimal locks
pg_sphere1.0spherical objects with useful functions, operators and index support
pg_cron1.5Job scheduler for PostgreSQL
pg_jieba1.1.0a parser for full-text search of Chinese
pg_stat_kcache2.2.1Kernel statistics gathering
pg_stat_statements1.9track planning and execution statistics of all SQL statements executed
pg_surgery1.0extension to perform surgery on a damaged relation
pg_trgm1.6text similarity measurement and index searching based on trigrams
pg_visibility1.2examine the visibility map (VM) and page-level visibility info
pg_wait_sampling1.1sampling based statistics of wait events
pgaudit1.6.2provides auditing functionality
pgcrypto1.3cryptographic functions
pgrowlocks1.2show row-level locking information
pgstattuple1.5show tuple-level statistics
pgtap1.2.0Unit testing for PostgreSQL
pldbgapi1.1server-side support for debugging PL/pgSQL functions
plperl1.0PL/Perl procedural language
plperlu1.0PL/PerlU untrusted procedural language
plpgsql1.0PL/pgSQL procedural language
plpython2u1.0PL/Python2U untrusted procedural language
plpythonu1.0PL/PythonU untrusted procedural language
plsql1.0Oracle compatible PL/SQL procedural language
pltcl1.0PL/Tcl procedural language
pltclu1.0PL/TclU untrusted procedural language
polar_bfile1.0The BFILE data type enables access to binary file LOBs that are stored in file systems outside Database
polar_bpe1.0polar_bpe
polar_builtin_cast1.1Internal extension for builtin casts
polar_builtin_funcs2.0implement polar builtin functions
polar_builtin_type1.5polar_builtin_type for PolarDB
polar_builtin_view1.5polar_builtin_view
polar_catalog1.2polardb pg extend catalog
polar_channel1.0polar_channel
polar_constraint1.0polar_constraint
polar_csn1.0polar_csn
polar_dba_views1.0polar_dba_views
polar_dbms_alert1.2implement polar_dbms_alert - supports asynchronous notification of database events.
polar_dbms_application_info1.0implement polar_dbms_application_info - record names of executing modules or transactions in the database.
polar_dbms_pipe1.1implements polar_dbms_pipe - package lets two or more sessions in the same instance communicate.
polar_dbms_aq1.2implement dbms_aq - provides an interface to Advanced Queuing.
polar_dbms_lob1.3implement dbms_lob - provides subprograms to operate on BLOBs, CLOBs, and NCLOBs.
polar_dbms_output1.2implement polar_dbms_output - enables you to send messages from stored procedures.
polar_dbms_lock1.0implement polar_dbms_lock - provides an interface to Oracle Lock Management services.
polar_dbms_aqadm1.3polar_dbms_aqadm - procedures to manage Advanced Queuing configuration and administration information.
polar_dbms_assert1.0implement polar_dbms_assert - provide an interface to validate properties of the input value.
polar_dbms_metadata1.0implement polar_dbms_metadata - provides a way for you to retrieve metadata from the database dictionary.
polar_dbms_random1.0implement polar_dbms_random - a built-in random number generator, not intended for cryptography
polar_dbms_crypto1.1implement dbms_crypto - provides an interface to encrypt and decrypt stored data.
polar_dbms_redact1.0implement polar_dbms_redact - provides an interface to mask data from queries by an application.
polar_dbms_debug1.1server-side support for debugging PL/SQL functions
polar_dbms_job1.0polar_dbms_job
polar_dbms_mview1.1implement polar_dbms_mview - enables to refresh materialized views.
polar_dbms_job_preload1.0polar_dbms_job_preload
polar_dbms_obfuscation_toolkit1.1implement polar_dbms_obfuscation_toolkit - enables an application to get data md5.
polar_dbms_rls1.1implement polar_dbms_rls - a fine-grained access control administrative built-in package
polar_multi_toast_utils1.0polar_multi_toast_utils
polar_dbms_session1.2implement polar_dbms_session - support to set preferences and security levels.
polar_odciconst1.0implement ODCIConst - Provide some built-in constants in Oracle.
polar_dbms_sql1.2implement polar_dbms_sql - provides an interface to execute dynamic SQL.
polar_osfs_toolkit1.0osfs library tools and functions extension
polar_dbms_stats14.0stabilize plans by fixing statistics
polar_monitor1.5monitor functions for PolarDB
polar_osfs_utils1.0osfs library utils extension
polar_dbms_utility1.3implement polar_dbms_utility - provides various utility subprograms.
polar_parameter_check1.0kernel extension for parameter validation
polar_dbms_xmldom1.0implement dbms_xmldom and dbms_xmlparser - support standard DOM interface and xml parser object
polar_parameter_manager1.1Extension to select parameters for manager.
polar_faults1.0.0simulate some database faults for end user or testing system.
polar_monitor_preload1.1examine the polardb information
polar_proxy_utils1.0Extension to provide operations about proxy.
polar_feature_utils1.2PolarDB feature utilization
polar_global_awr1.0PolarDB Global AWR Report
polar_publication1.0support polardb pg logical replication
polar_global_cache1.0polar_global_cache
polar_px1.0Parallel Execution extension
polar_serverless1.0polar serverless extension
polar_resource_manager1.0a background process that forcibly frees user session process memory
polar_sys_context1.1implement polar_sys_context - returns the value of parameter associated with the context namespace at the current instant.
polar_gpc1.3polar_gpc
polar_tde_utils1.0Internal extension for TDE
polar_gtt1.1polar_gtt
polar_utl_encode1.2implement polar_utl_encode - provides functions that encode RAW data into a standard encoded format
polar_htap1.1extension for PolarDB HTAP
polar_htap_db1.0extension for PolarDB HTAP database level operation
polar_io_stat1.0polar io stat in multi dimension
polar_utl_file1.0implement utl_file - support PL/SQL programs can read and write operating system text files
polar_ivm1.0polar_ivm
polar_sql_mapping1.2Record error sqls and mapping them to correct one
polar_stat_sql1.0Kernel statistics gathering, and sql plan nodes information gathering
tds_fdw2.0.2Foreign data wrapper for querying a TDS database (Sybase or Microsoft SQL Server)
xml21.1XPath querying and XSLT
polar_upgrade_catalogs1.1Upgrade catalogs for old version instance
polar_utl_i18n1.1polar_utl_i18n
polar_utl_raw1.0implement utl_raw - provides SQL functions for manipulating RAW datatypes.
timescaledb2.9.2Enables scalable inserts and complex queries for time-series data
polar_vfs1.0polar virtual file system for different storage
polar_worker1.0polar_worker
postgres_fdw1.1foreign-data wrapper for remote PostgreSQL servers
refint1.0functions for implementing referential integrity (obsolete)
roaringbitmap0.5support for Roaring Bitmaps
tsm_system_time1.0TABLESAMPLE method which accepts time in milliseconds as a limit
vector0.5.0vector data type and ivfflat and hnsw access methods
rum1.3RUM index access method
unaccent1.1text search dictionary that removes accents
seg1.4data type for representing line segments or floating-point intervals
sequential_uuids1.0.2generator of sequential UUIDs
uuid-ossp1.1generate universally unique identifiers (UUIDs)
smlar1.0compute similarity of any one-dimensional arrays
varbitx1.1varbit functions pack
sslinfo1.2information about SSL certificates
tablefunc1.0functions that manipulate whole tables, including crosstab
tcn1.0Triggered change notifications
zhparser1.0a parser for full-text search of Chinese
address_standardizer3.3.2Ganos PostGIS address standardizer
address_standardizer_data_us3.3.2Ganos PostGIS address standardizer data us
ganos_fdw6.0Ganos Spatial FDW extension for POLARDB
ganos_geometry6.0Ganos geometry lite extension for POLARDB
ganos_geometry_pyramid6.0Ganos Geometry Pyramid extension for POLARDB
ganos_geometry_sfcgal6.0Ganos geometry lite sfcgal extension for POLARDB
ganos_geomgrid6.0Ganos geometry grid extension for POLARDB
ganos_importer6.0Ganos Spatial importer extension for POLARDB
ganos_networking6.0Ganos networking
ganos_pointcloud6.0Ganos pointcloud extension For POLARDB
ganos_pointcloud_geometry6.0Ganos_pointcloud LIDAR data and ganos_geometry data for POLARDB
ganos_raster6.0Ganos raster extension for POLARDB
ganos_scene6.0Ganos scene extension for POLARDB
ganos_sfmesh6.0Ganos surface mesh extension for POLARDB
ganos_spatialref6.0Ganos spatial reference extension for POLARDB
ganos_trajectory6.0Ganos trajectory extension for POLARDB
ganos_vomesh6.0Ganos volume mesh extension for POLARDB
postgis_tiger_geocoder3.3.2Ganos PostGIS tiger geocoder
postgis_topology3.3.2Ganos PostGIS topology

15.15 - PostgresML

How to deploy PostgresML with Pigsty: ML, training, inference, Embedding, RAG inside DB.

PostgresML is a PostgreSQL extension that supports the latest large language models (LLM), vector operations, and classical machine learning. It also supports traditional Postgres application workloads.

PostgresML (pgml) is written in Rust. You can run standalone Docker images. This page is reference documentation and does not describe a docker-compose template.

PostgresML officially supports Ubuntu 22.04. We also maintain RPM versions for EL 8/9, for use without CUDA and NVIDIA-related features.

You need internet access on database nodes to download Python dependencies from PyPI and models from HuggingFace.

PostgresML is Deprecated

The company that developed PostgresML has ceased operations.


Configuration

PostgresML is an extension written in Rust. Pigsty maintains prebuilt packages for PG14-17 on EL8/EL9 and Debian/Ubuntu platforms.

Creating a New Cluster

PostgresML 2.10.0 is available for PostgreSQL 14-17. The following example uses PG17. For PG14-16, set pg_version to the applicable major version.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_version: 17
    pg_libs: 'pgml, pg_stat_statements, auto_explain'
    pg_extensions: [ pgml, pgvector, wal2json, pg_repack ]

Pigsty maps the pgml package alias to pgml_$v on EL and postgresql-$v-pgml on Debian/Ubuntu. You must also add pgml to pg_libs.

Enabling on an Existing Cluster

To enable pgml on an existing cluster, you can install it using Ansible’s package module:

ansible pg-meta -m package -b -a 'name=pgml_17'
# ansible el8,el9 -m package -b -a 'name=pgml_17'              # EL 8/9
# ansible u22,u24 -m package -b -a 'name=postgresql-17-pgml'   # Debian/Ubuntu

Python Dependencies

You must also install PostgresML’s Python dependencies on the cluster nodes. See the official tutorial: Installation Guide

Install Python and PIP

Ensure python3, pip, and venv are installed:

# Ubuntu 22.04 (python3.10), need to install pip and venv using apt
sudo apt install -y python3 python3-pip python3-venv

For EL 8 / EL9 and compatible distributions, you can use python3.11:

# EL 8/9, can upgrade the default pip and virtualenv
sudo yum install -y python3.11 python3.11-pip       # install latest python3.11
python3.11 -m pip install --upgrade pip virtualenv  # use python3.11 on EL8 / EL9
Using PyPI Mirrors

For users in mainland China, we recommend using Tsinghua University’s PyPI mirror.

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple    # set global mirror (recommended)
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple some-package        # use for single installation

Install Dependencies

Create a Python virtual environment. Use pip to install dependencies from requirements.txt and requirements-xformers.txt.

If you’re using EL 8/9, replace python3 with python3.11 in the following commands.

su - postgres;                          # create virtual environment as database superuser
mkdir -p /data/pgml; cd /data/pgml;     # create virtual environment directory
python3    -m venv /data/pgml           # create virtual environment directory (Ubuntu 22.04)
source /data/pgml/bin/activate          # activate virtual environment

# write Python dependencies and install with pip
cat > /data/pgml/requirements.txt <<EOF
accelerate==0.22.0
auto-gptq==0.4.2
bitsandbytes==0.41.1
catboost==1.2
ctransformers==0.2.27
datasets==2.14.5
deepspeed==0.10.3
huggingface-hub==0.17.1
InstructorEmbedding==1.0.1
lightgbm==4.1.0
orjson==3.9.7
pandas==2.1.0
rich==13.5.2
rouge==1.0.1
sacrebleu==2.3.1
sacremoses==0.0.53
scikit-learn==1.3.0
sentencepiece==0.1.99
sentence-transformers==2.2.2
tokenizers==0.13.3
torch==2.0.1
torchaudio==2.0.2
torchvision==0.15.2
tqdm==4.66.1
transformers==4.33.1
xgboost==2.0.0
langchain==0.0.287
einops==0.6.1
pynvml==11.5.0
EOF

# install dependencies using pip in the virtual environment
python3 -m pip install -r /data/pgml/requirements.txt
python3 -m pip install xformers==0.0.21 --no-dependencies

# additionally, 3 Python packages need to be installed globally using sudo!
sudo python3 -m pip install xgboost lightgbm scikit-learn

Enable PostgresML

Install the pgml extension and Python dependencies on all cluster nodes first. Then enable pgml on the PostgreSQL cluster.

Use patronictl to configure the cluster. Add pgml to shared_preload_libraries. Set pgml.venv to your virtual environment directory:

shared_preload_libraries: pgml, timescaledb, pg_stat_statements, auto_explain
pgml.venv: '/data/pgml'

Then restart the database cluster and create the extension using SQL commands:

CREATE EXTENSION vector;        -- also recommend installing pgvector!
CREATE EXTENSION pgml;          -- create PostgresML in the current database
SELECT pgml.version();          -- print PostgresML version information

If these commands succeed, the output should be similar to this:

# create extension pgml;
INFO:  Python version: 3.11.2 (main, Oct  5 2023, 16:06:03) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
INFO:  Scikit-learn 1.3.0, XGBoost 2.0.0, LightGBM 4.1.0, NumPy 1.26.1
CREATE EXTENSION

# SELECT pgml.version(); -- print PostgresML version information
 version
---------
 2.7.8

For more details, see the official PostgresML documentation: https://postgresml.org/docs/guides/use-cases/

15.16 - Greenplum

Deploy and monitor Greenplum clusters with Pigsty to build Massively Parallel Processing (MPP) PostgreSQL data warehouses.

Pigsty supports deployment of Greenplum clusters and the derivative distribution YMatrixDB. You can also add existing Greenplum deployments to Pigsty monitoring.


Overview

Only the professional/enterprise editions provide Greenplum / YMatrix cluster deployment. This capability is not currently open source.


Installation

Pigsty provides installation packages for Greenplum 6 (@el7) and Greenplum 7 (@el8). Open-source users can install and configure them manually.

# EL 7 Only (Greenplum6)
./node.yml -t node_install  -e '{"node_repo_modules":"pgsql","node_packages":["open-source-greenplum-db-6"]}'

# EL 8 Only (Greenplum7)
./node.yml -t node_install  -e '{"node_repo_modules":"pgsql","node_packages":["open-source-greenplum-db-7"]}'

Configuration

To define a Greenplum cluster, set pg_mode = gpsql. Also set the identity parameters pg_shard and gp_role.

#================================================================#
#                        GPSQL Clusters                          #
#================================================================#

#----------------------------------#
# cluster: mx-mdw (gp master)
#----------------------------------#
mx-mdw:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary , nodename: mx-mdw-1 }
  vars:
    gp_role: master          # this cluster is used as greenplum master
    pg_shard: mx             # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-mdw       # this master cluster name is mx-mdw
    pg_databases:
      - { name: matrixmgr , extensions: [ { name: matrixdbts } ] }
      - { name: meta }
    pg_users:
      - { name: meta , password: DBUser.Meta , pgbouncer: true }
      - { name: dbuser_monitor , password: DBUser.Monitor , roles: [ dbrole_readonly ], superuser: true }

    pgbouncer_enabled: true                # enable pgbouncer for greenplum master
    pgbouncer_exporter_enabled: false      # enable pgbouncer_exporter for greenplum master
    pg_exporter_params: 'host=127.0.0.1&sslmode=disable'  # use 127.0.0.1 as local monitor host

#----------------------------------#
# cluster: mx-sdw (gp master)
#----------------------------------#
mx-sdw:
  hosts:
    10.10.10.11:
      nodename: mx-sdw-1        # greenplum segment node
      pg_instances:             # greenplum segment instances
        6000: { pg_cluster: mx-seg1, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg2, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
    10.10.10.12:
      nodename: mx-sdw-2
      pg_instances:
        6000: { pg_cluster: mx-seg2, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633  }
        6001: { pg_cluster: mx-seg3, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634  }
    10.10.10.13:
      nodename: mx-sdw-3
      pg_instances:
        6000: { pg_cluster: mx-seg3, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg1, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
  vars:
    gp_role: segment               # these are nodes for gp segments
    pg_shard: mx                   # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-sdw             # these segment clusters name is mx-sdw
    pg_preflight_skip: true        # skip preflight check (since pg_seq & pg_role & pg_cluster not exists)
    pg_exporter_config: pg_exporter_basic.yml                             # use basic config to avoid segment server crash
    pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'  # use gp_role = utility to connect to segments

PG Exporter also requires extra connection parameters to collect metrics from Greenplum Segment instances.

15.17 - Neon

Use Neon’s open-source Serverless PostgreSQL kernel to build flexible, scale-to-zero, forkable PG services.

Neon separates storage from compute. It provides automatic scaling, scale-to-zero operation, and database branching.

Neon official website: https://neon.tech/

Neon binaries are currently too large for the open-source package set. This support remains in the pilot stage. If you need it, contact Pigsty sales.

16 - FAQ

Frequently asked questions about PostgreSQL

Why can’t my current user use the pg admin alias?

Since Pigsty v4.0, only members of admin on administration nodes can manage global Patroni and PostgreSQL clusters with pg.

The node.yml playbook creates the administration user dba with this permission. To grant the permission to your current user, explicitly add that user to the admin group:

sudo usermod -aG admin <username>

PGSQL Init Fails: Fail to wait for postgres/patroni primary

Several conditions can cause this error. Check the Ansible, systemd, Patroni, and PostgreSQL logs to find the cause.

  • Possibility 1: Cluster config error - find and fix the incorrect config items.
  • Possibility 2: A cluster with the same name exists, or the previous same-named cluster primary was improperly removed.
  • Possibility 3: Residual metadata from a same-named cluster remains in DCS. First inspect the exact keyspace with etcdctl get --prefix /pg/<cls>/; only after confirming the backup and full cluster name should you use etcdctl del --prefix /pg/<cls>/. The trailing / is the namespace boundary and must not be omitted, or a cluster whose name merely starts with the same text can also match. This is destructive; prefer the controlled decommissioning workflow.
  • Possibility 4: Your PostgreSQL or node-related RPM pkgs were not successfully installed.
  • Possibility 5: Your Watchdog kernel module was not properly enabled/loaded.
  • Possibility 6: The locale you specified during database init doesn’t exist (e.g., used en_US.UTF8 but English language pack or Locale support wasn’t installed).
  • If you encounter other causes, please submit an Issue or ask the community for help.

PGSQL Init Fails: Fail to wait for postgres/patroni replica

There are several possible causes:

Immediate failure: Possible causes include configuration errors, network problems, and corrupted DCS metadata. Check /pg/log to find the actual cause.

Failure after some time: The source instance may have corrupted data. See the FAQ about replica creation when data is corrupted.

Timeout after a long time: The wait for postgres replica task can time out after 30 minutes or more. This is common for large clusters. For example, replica creation for a cluster larger than 1TB may take hours.

In this case, replica creation continues in the background. Use pg list <cls> to check cluster status. Wait until the replica catches up with the primary. Then run the remaining tasks to complete replica initialization:

./pgsql.yml -t pg_hba,pg_reload,pg_backup,pgbouncer,pg_vip,pg_dns,pg_service,pg_exporter,pg_register -l <problematic_replica>

PGSQL Init Fails: ABORT due to pg_safeguard enabled

The target PostgreSQL instance has deletion protection enabled. Disable pg_safeguard to remove it.

If the deletion safeguard pg_safeguard is enabled, you cannot remove running PGSQL instances using bin/pgsql-rm or the pgsql-rm.yml playbook.

To disable pg_safeguard, set it to false in the inventory. Alternatively, pass -e pg_safeguard=false when you run the playbook.

./pgsql-rm.yml -e pg_safeguard=false -l <cls_to_remove>

How to Ensure No Data Loss During Failover?

Use the crit.yml param template, set pg_rpo to 0, or config the cluster for sync commit mode.

Consider using Sync Standby and Quorum Commit to ensure zero data loss during failover.

For more details, see the intro in Security Considerations - Availability.


How to Rescue When Disk is Full?

If the disk is full and shell commands cannot run, use rm -rf /pg/dummy to release emergency space.

The default pg_dummy_filesize is 64MB. For production, increase it to 8GB or more.

Pigsty creates /pg/dummy on the main PGSQL data disk. Delete this file to release emergency space.

This space lets you run shell scripts to reclaim more space. For example, you can remove unnecessary logs, WAL, old data, WAL archives, or backups.


How to Create a Replica When Cluster Data is Corrupted?

Pigsty sets clonefrom: true in each instance’s Patroni configuration. This marks the instance as a source for replica creation.

Corrupted data files on a source instance can cause replica creation to fail. Set clonefrom: false on that instance to prevent use of its data:

$ vi /etc/patroni/patroni.yml

tags:
  nofailover: false
  clonefrom: true      # ----------> change to false
  noloadbalance: false
  nosync: false
  version:  '18'
  spec: '4C.8G.50G'
  conf: 'oltp.yml'

$ systemctl reload patroni    # Reload Patroni config

What is the Perf Overhead of PostgreSQL Monitoring?

A normal PostgreSQL instance scrape takes about 200ms. The default scrape interval is 10 seconds. The overhead is usually negligible for a production database instance with multiple CPU cores.

Pigsty monitors in-database objects by default. For a database with hundreds of thousands of tables or indexes, a scrape may take several seconds.

You can change the Prometheus scrape frequency. Keep the scrape interval significantly longer than the time required for one scrape.


How to Monitor an Existing PostgreSQL Instance?

Detailed monitoring config instructions are provided in PGSQL Monitor.


How to Manually Remove PostgreSQL Monitoring Targets?

./pgsql-rm.yml -t rm_metrics -l <cls>     # Remove all instances of cluster 'cls' from victoria
bin/pgmon-rm <ins>     # Remove a single instance 'ins' monitoring object from Victoria, especially suitable for removing added external instances

17 - Misc

Miscellaneous Topics

17.1 - Service / Access

Separate read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

Separate read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

A service exposes database cluster capabilities to clients. It hides the details of the underlying cluster.

Services provide stable access in production, including after automatic failover in high-availability clusters. Single-node users usually do not need services.


Single-Node Users

Services are intended for production environments. Personal users and single-node clusters can connect directly through an instance name or IP address.

For example, Pigsty’s default single-node pg-meta.meta database can be connected directly using three different users:

psql postgres://dbuser_dba:[email protected]/meta     # Connect directly with DBA superuser
psql postgres://dbuser_meta:[email protected]/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

Production environments use primary-replica clusters with replication. Exactly one instance, the primary, leads the cluster and accepts writes. Replicas continuously fetch change logs from the primary to keep their data consistent. Replicas also handle read-only requests, which substantially reduces primary load in workloads with many reads. Thus, separating write requests from read-only requests is common.

For frequent short connections, Pgbouncer pools requests to reduce the cost of connection and backend process creation. Operations such as ETL and database changes need direct connections that bypass the pool. Failures in high-availability clusters can trigger failover and change the primary. Write traffic must follow the new primary automatically. Services represent these access requirements: separate reads and writes, pooled or direct connections, and automatic routing after failover.

Typically, database clusters must provide this most basic service:

  • Read-Write Service (primary): Can read and write to the database

For production database clusters, at least these two services should be provided:

  • Read-Write Service (primary): Write data: can only be carried by the primary.
  • Read-Only Service (replica): Read data: can be carried by replicas, or by the primary if there are no replicas

In addition, depending on specific business scenarios, there may be other services, such as:

  • Default Direct Service (default): Allows (admin) users to access the database directly, bypassing the connection pool
  • Offline Replica Service (offline): Dedicated replicas that do not handle online read-only traffic, used for ETL and analytical queries
  • Standby Replica Service (standby): Read-only service without replication lag, handled by sync standby/primary for read-only queries
  • Delayed Replica Service (delayed): Access old data from the same cluster at a previous point in time, handled by delayed replica

Default Services

By default, Pigsty provides four services for each PostgreSQL cluster:

ServicePortDescription
primary5433Production read-write, connects to primary connection pool (6432)
replica5434Production read-only, connects to replica connection pool (6432)
default5436Admin, ETL writes, direct access to primary (5432)
offline5438OLAP, ETL, personal users, interactive queries

Taking the default pg-meta cluster as an example, it provides four default services:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # pg-meta-primary : production read-write via primary pgbouncer(6432)
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # pg-meta-replica : production read-only via replica pgbouncer(6432)
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # pg-meta-default : direct connection via primary postgres(5432)
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : direct connection via offline postgres(5432)

You can see how these four services work from the sample cluster architecture diagram:

pigsty-ha.png

pg_dns_target controls the DNS target of pg-meta. The default, auto, selects the L2 VIP when enabled. Otherwise, it selects the IP address of the inventory primary. VIP is disabled by default. See Accessing Services.


Service Implementation

Pigsty implements services with HAProxy on nodes. Different ports identify different services.

Pigsty enables HAProxy on each managed node, including database nodes. Database nodes have primary or replica roles, but each node exposes the same services. With the correct service port, you can access the primary read-write service through a replica node. Thus, access to any cluster instance gives access to all cluster services.

This design is similar to Kubernetes NodePort services. Each Pigsty service has two core elements:

  1. Access endpoints exposed through NodePort (port number, where to access?)
  2. Target instances selected through Selectors (instance list, who carries the load?)

Pigsty provides services up to the cluster HAProxy endpoint. Clients can reach these load balancers in several ways. See Accessing Services.

Declare all services in configuration files. For example, pg_default_services defines the default PostgreSQL services:

pg_default_services:
- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

Define additional services in pg_services. Both pg_default_services and pg_services are arrays of service definition objects.


Defining Services

Pigsty allows you to define your own services:

  • pg_default_services: Services uniformly exposed by all PostgreSQL clusters, four by default.
  • pg_services: Additional PostgreSQL services, can be defined at global or cluster level as needed.
  • haproxy_services: Directly customize HAProxy service content, can be used for accessing other components

For PostgreSQL clusters, you usually need only the first two parameters. Each service generates a configuration file on every related HAProxy instance: /etc/haproxy/conf.d/<pg_cluster>-<service>.cfg. The following custom standby service provides read-only access without replication lag. Add its definition to pg_services:

- name: standby                   # Required, service name, final svc name uses `pg_cluster` as prefix, e.g.: pg-meta-standby
  port: 5435                      # Required, exposed service port (as kubernetes service node port mode)
  ip: "*"                         # Optional, IP address the service binds to, all IP addresses by default
  selector: "[]"                  # Required, service member selector, uses JMESPath to filter configuration manifest
  backup: "[? pg_role == `primary`]"  # Optional, service member selector (backup), instances selected here only carry the service when all default selector instances are down
  dest: default                   # Optional, target port, default|postgres|pgbouncer|<port_number>, defaults to 'default', meaning pg_default_service_dest decides
  check: /sync                    # Optional, health check URL path, defaults to /, here uses Patroni API: /sync, only sync standby and primary return 200 healthy status code
  maxconn: 5000                   # Optional, maximum number of allowed frontend connections, defaults to 5000
  balance: roundrobin             # Optional, haproxy load balancing algorithm (defaults to roundrobin, other options: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

For the three-node pg-test example cluster, Pigsty generates /etc/haproxy/conf.d/pg-test-standby.cfg from this definition:

#---------------------------------------------------------------------
# service: pg-test-standby @ 10.10.10.11:5435
#---------------------------------------------------------------------
# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12
# service backups   10.10.10.11
listen pg-test-standby
    bind *:5435            # <--- Binds port 5435 on all IP addresses
    mode tcp               # <--- Load balancer works on TCP protocol
    maxconn 5000           # <--- Maximum connections 5000, can be increased as needed
    balance roundrobin     # <--- Load balancing algorithm is rr round-robin, can also use leastconn
    option httpchk         # <--- Enable HTTP health check
    option http-keep-alive # <--- Keep HTTP connection
    http-check send meth OPTIONS uri /sync   # <---- Here uses /sync, Patroni health check API, only sync standby and primary return 200 healthy status code
    http-check expect status 200             # <---- Health check return code 200 means normal
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers: selector "[]" includes all three pg-test instances as pg-test-standby backends; /sync admits only the primary and synchronous standby.
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup  # <----- Only primary satisfies condition pg_role == `primary`, selected by backup selector
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100         #        Therefore serves as service fallback instance: normally doesn't handle requests, only handles read-only requests when all other replicas fail, thus maximally avoiding read-write service being affected by read-only service
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100         #

The selector: "[]" expression includes all three pg-test instances in the pg-test-standby backend list. However, the Patroni /sync health check returns HTTP 200 only for the primary and synchronous standby. Only those instances can handle requests. The primary also matches pg_role == primary, so the backup selector marks it as a backup server. It handles requests only when no synchronous standby can do so.


Primary Service

The Primary service provides cluster read-write access on port 5433. It is a critical production service. Its definition is:

- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  • selector: "[]" includes all cluster members in the Primary service.
  • Only the primary passes check: /primary and receives traffic.
  • dest: default makes pg_default_service_dest control the destination.
  • Pigsty replaces the dest value default with pg_default_service_dest, whose default is pgbouncer.
  • Thus, the default destination is the primary connection pool on pgbouncer_port, normally 6432.

Set pg_default_service_dest to postgres to bypass the connection pool. The Primary service then connects directly to pg_port, normally 5432. Use this setting when you do not need pooling.

Example: haproxy configuration for pg-test-primary
listen pg-test-primary
    bind *:5433         # <--- primary service defaults to port 5433
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary # <--- primary service defaults to Patroni RestAPI /primary health check
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

Patroni high availability ensures that no more than one instance passes the /primary health check at a time. Thus, the Primary service always routes traffic to the primary.

The Primary service also protects traffic routing during a split-brain condition. For example, kill -9 can stop primary Patroni without a watchdog. HAProxy sends traffic only when Patroni is alive and reports primary status. This protection is unavailable with a direct database connection.


Replica Service

The Replica service provides read-only access on port 5434. It is the next most important production service after Primary. Its definition is:

- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  • selector: "[]" includes all cluster members in the Replica service.
  • All instances can pass check: /read-only and receive traffic.
  • The backup selector [? pg_role == 'primary' || pg_role == 'offline' ] marks the primary and offline replicas as backups.
  • The primary or offline replicas receive traffic only when all normal replicas are down.
  • dest: default makes pg_default_service_dest control the destination.
  • Pigsty replaces the dest value default with pg_default_service_dest, whose default is pgbouncer, as for the Primary service.
  • Thus, the default destination is the replica connection pool on pgbouncer_port, normally 6432.
Example: haproxy configuration for pg-test-replica
listen pg-test-replica
    bind *:5434
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /read-only
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

The Replica service first sends read-only requests to available dedicated replicas. It sends them to the primary only when all replicas are down. For a two-node primary-replica cluster, this means it uses the replica while available and the primary when the replica fails.

The Replica service uses dedicated Offline instances only when all dedicated read-only instances are down. This separates fast online queries from slow offline queries to reduce interference.


Default Service

The Default service provides services on port 5436. It is a variant of the Primary service.

The Default service always bypasses the connection pool and connects directly to primary PostgreSQL. Use it for administration connections, ETL writes, and change data capture (CDC).

- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }

If pg_default_service_dest is postgres, Default and Primary have the same behavior. Only their names and ports differ. In this case, you can remove Default from the default services.

Example: haproxy configuration for pg-test-default
listen pg-test-default
    bind *:5436         # <--- Except for listening port/target port and service name, other configurations are exactly the same as primary service
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:5432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100

Offline Service

The Offline service listens on port 5438 and connects directly to PostgreSQL without the pool. Use it for slow queries, analytical queries, ETL reads, and personal interactive queries:

- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

Offline routes traffic directly to dedicated offline replicas. It can also use normal read-only instances with the pg_offline_query flag.

  • The selector includes replicas with pg_role = offline and normal read-only instances with pg_offline_query = true.
  • Dedicated offline replicas do not handle Replica service requests by default. This separates fast and slow requests. Flagged normal replicas do handle those requests.
  • The backup selector includes normal replicas without the offline flag. They handle Offline traffic when dedicated or flagged offline instances fail.
  • /replica returns HTTP 200 only for replicas. It rejects the primary, even when the primary is the only remaining cluster instance.
  • Neither selector includes the primary. Thus, Offline traffic never reaches or affects the primary.
Example: haproxy configuration for pg-test-offline
listen pg-test-offline
    bind *:5438
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /replica
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup

Offline provides restricted read-only access for personal interactive queries, slow analytical or ETL queries, and long transactions.

The Offline service needs special handling after a switchover. The HAProxy /replica health check automatically rejects the new primary. However, selector uses static pg_role and pg_offline_query inventory labels. In a two-node primary-replica cluster, only the replica may serve Offline queries. A switchover can then leave no eligible backend temporarily. Reloading an unchanged inventory does not add the old primary to the backend list.

First update the inventory labels or pg_offline_query for the new plan, then reload services. Alternatively, switch the primary back.

For a simple business model, you can remove Default and Offline. Use Primary and Replica to connect directly to the database.


Reload Services

Reload services after changes to cluster membership, service definitions, static selector labels, or relative weights. Patroni health checks handle normal Primary and Replica switchovers automatically. These switchovers need no separate reload.

bin/pgsql-svc <cls> [ip...]         # Reload services for lb cluster or lb instance
# ./pgsql.yml -t pg_service         # Actual ansible task for reloading services

Accessing Services

Pigsty provides services up to the cluster HAProxy endpoint. Clients can reach these load balancers in several ways.

The typical approach is to use DNS or VIP access, binding them to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL services in different ways.

Host

TypeExampleDescription
Cluster Domainpg-testAccess via cluster domain name (resolved by dnsmasq @ infra node)
Cluster VIP Address10.10.10.3Access via L2 VIP address managed by vip-manager, bound to primary node
Instance Hostnamepg-test-1Access via any instance hostname (resolved by dnsmasq @ infra node)
Instance IP Address10.10.10.11Access any instance’s IP address

Port

Pigsty uses different ports to distinguish pg services

PortServiceTypeDescription
5432postgresDatabaseDirect access to postgres server
6432pgbouncerMiddlewareAccess postgres via connection pool middleware
5433primaryServiceAccess primary pgbouncer (or postgres)
5434replicaServiceAccess replica pgbouncer (or postgres)
5436defaultServiceAccess primary postgres
5438offlineServiceAccess offline postgres

Combinations

# Access via cluster domain name (the examples below assume L2 VIP is enabled; without it, auto targets the inventory primary IP)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> Primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> Primary connection pool -> Primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary connection pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica connection pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> Offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://[email protected]:5432/test # L2 VIP -> Primary direct access
postgres://[email protected]:6432/test # L2 VIP -> Primary connection pool -> Primary
postgres://[email protected]:5433/test # L2 VIP -> HAProxy -> Primary connection pool -> Primary
postgres://[email protected]:5434/test # L2 VIP -> HAProxy -> Replica connection pool -> Replica
postgres://[email protected]:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for admin)
postgres://[email protected]:5438/test # L2 VIP -> HAProxy -> Offline direct connection (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database instance direct connection (single instance access)
postgres://test@pg-test-1:6432/test # DNS -> Connection pool -> Database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> Connection pool -> Database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> Connection pool -> Database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> Database direct connection
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> Database offline read/write

# Specify any cluster instance IP directly
postgres://[email protected]:5432/test # Database instance direct connection (direct instance specification, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection pool -> Database
postgres://[email protected]:5433/test # HAProxy -> Connection pool -> Database read/write
postgres://[email protected]:5434/test # HAProxy -> Connection pool -> Database read-only
postgres://[email protected]:5436/test # HAProxy -> Database direct connection
postgres://[email protected]:5438/test # HAProxy -> Database offline read/write

# Smart client: automatic read-write separation
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

Overriding Services

You can override service defaults in several ways. A common requirement is for Primary and Replica to bypass Pgbouncer and connect directly to PostgreSQL.

Set pg_default_service_dest to postgres. Services with svc.dest='default' then use postgres instead of the default pgbouncer destination.

If Primary already connects to PostgreSQL, Default is redundant. You can remove it.

If you do not need to distinguish personal interactive queries from slow analytical or ETL queries, you can remove Offline. Remove it from pg_default_services.

If replicas do not need to handle online read-only traffic, you can remove Replica from the default list.


Delegating Services

Pigsty exposes PostgreSQL services through HAProxy on nodes. All HAProxy instances in a cluster use the same service definitions.

You can instead delegate PostgreSQL services to a specific node group, such as a dedicated HAProxy load balancer cluster.

Override pg_default_services and set pg_service_provider to the proxy group name.

This example exposes the PostgreSQL Primary service on port 10013 through the proxy HAProxy node group.

pg_service_provider: proxy       # Use load balancer from `proxy` group on port 10013
pg_default_services:  [{ name: primary ,port: 10013 ,dest: postgres  ,check: /primary   ,selector: "[]" }]

Use a unique port for each delegated service in the proxy cluster.

An example of using a dedicated load balancer cluster is provided in the 20-node production environment simulation sandbox: conf/ha/simu.yml

17.2 - Access Control

Entry points for PostgreSQL access-control concepts, configuration, and administration.

Pigsty access-control documentation is organized by purpose:

dbrole_offline provides independent read-only object privileges. It does not restrict instance scope automatically. To allow access only on offline instances, set role: offline explicitly on the corresponding HBA rule. Verify the generated pg_hba.conf on both online and offline instances.

17.3 - User / Role

Users/roles refer to logical objects within a database cluster created using the SQL commands CREATE USER/ROLE.

In this context, users refer to logical objects within a database cluster created using the SQL commands CREATE USER/ROLE.

In PostgreSQL, users belong to the database cluster, rather than a specific database. Create business users before you create business databases.


Defining Users

Pigsty defines roles and users in database clusters through two configuration parameters:

  • pg_default_roles: Defines globally unified roles and users
  • pg_users: Defines business users and roles at the database cluster level

The first parameter defines roles and users shared across the environment. The second defines business roles and users for individual clusters. Both use arrays of user definition objects.

You can define multiple users and roles. Pigsty creates global objects first, then cluster-level objects, following array order within each group. Thus, later users can belong to roles defined earlier.

Here is the business user definition for the default cluster pg-meta in the Pigsty demo environment:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
      - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }

Each user/role definition is an object that may include the following fields. Using dbuser_meta as an example:

- name: dbuser_meta               # Required, `name` is the only mandatory field in user definition
  password: DBUser.Meta           # Optional, password can be scram-sha-256 hash string or plaintext
  login: true                     # Optional, can login by default
  superuser: false                # Optional, default is false, is this a superuser?
  createdb: false                 # Optional, default is false, can create databases?
  createrole: false               # Optional, default is false, can create roles?
  inherit: true                   # Optional, by default this role can use inherited privileges?
  replication: false              # Optional, default is false, can this role perform replication?
  bypassrls: false                # Optional, default is false, can this role bypass row-level security?
  pgbouncer: true                 # Optional, default is false, add this user to pgbouncer user list? (production users using connection pool should explicitly set to true)
  connlimit: -1                   # Optional, user connection limit, default -1 disables limit
  expire_in: 3650                 # Optional, this role expires: calculated from creation + n days (higher priority than expire_at)
  expire_at: '2030-12-31'         # Optional, when this role expires, use YYYY-MM-DD format string to specify a date (lower priority than expire_in)
  comment: pigsty admin user      # Optional, description and comment string for this user/role
  roles: [dbrole_admin]           # Optional, default roles are: dbrole_{admin,readonly,readwrite,offline}
  parameters: {}                  # Optional, use `ALTER ROLE SET` to configure role-level database parameters for this role
  pool_mode: transaction          # Optional, pgbouncer pool mode defaulting to transaction, user level
  pool_connlimit: 100             # Optional, user-level max pool connections; omitted inherits global default 100
  search_path: public             # Optional, key-value configuration parameters per postgresql documentation (e.g., use pigsty as default search_path)
  • The only required field is name. Use a valid username that is unique within the PostgreSQL cluster.
  • Roles do not need a password. Business users with login access usually need one.
  • password accepts plaintext, scram-sha-256, or md5 hash strings. Avoid plaintext passwords.
  • Pigsty creates users and roles in array order. Define roles and groups before their members.
  • login, superuser, createdb, createrole, inherit, replication, and bypassrls are Boolean flags.
  • pgbouncer is disabled by default. To add business users to the pool user list, explicitly set it to true.

ACL System

Pigsty provides a built-in access control / ACL model. Assign its default business roles to users as required:

  • dbrole_readwrite: Role with global read-write access (production accounts primarily used by business should have database read-write privileges)
  • dbrole_readonly: Role with global read-only access (if other businesses need read-only access, use this role)
  • dbrole_admin: Role with DDL privileges (business administrators, scenarios requiring table creation in applications)
  • dbrole_offline: Independent read-only role for ad hoc queries, ETL, and analytics; restrict its instance scope explicitly through HBA

To define your own ACL system, consider these parameters and templates:


Creating Users

During the cluster initialization PROVISION phase, Pigsty creates users and roles one at a time. It uses the definitions in pg_default_roles and pg_users. To create users on an existing cluster, use bin/pgsql-user. Add the definition to all.children.<cls>.pg_users, then run:

bin/pgsql-user <cls> <username>    # pgsql-user.yml -l <cls> -e username=<username>

The user creation playbook is always idempotent, unlike the database creation playbook. If the target user exists, Pigsty changes its attributes to match the configuration. Thus, you can usually repeat the playbook on existing clusters.

Please Use Playbooks to Create Users

Manual business user creation is not recommended, especially with the default Pgbouncer connection pool. If you create users manually, you must maintain the Pgbouncer user list and keep it consistent with PostgreSQL. The bin/pgsql-user tool and pgsql-user.yml playbook also add users to the Pgbouncer user list.


Modifying Users

The method for modifying PostgreSQL user attributes is the same as Creating Users.

First, change the required attributes in the user definition. Then run this command to apply them:

bin/pgsql-user <cls> <username>    # pgsql-user.yml -l <cls> -e username=<username>

User modification changes attributes with ALTER USER; it does not delete users. It also keeps existing privileges and groups. It uses GRANT to add new roles.


Pgbouncer Users

Pigsty enables Pgbouncer connection pooling and manages its users by default.

Pigsty adds all users in pg_users that explicitly have the pgbouncer: true flag to the pgbouncer user list.

Users in the Pgbouncer connection pool are listed in /etc/pgbouncer/userlist.txt:

"postgres" ""
"dbuser_wiki" "SCRAM-SHA-256$4096:+77dyhrPeFDT/TptHs7/7Q==$KeatuohpKIYzHPCt/tqBu85vI11o9mar/by0hHYM2W8=:X9gig4JtjoS8Y/o1vQsIX/gY1Fns8ynTXkbWOjUfbRQ="
"dbuser_view" "SCRAM-SHA-256$4096:DFoZHU/DXsHL8MJ8regdEw==$gx9sUGgpVpdSM4o6A2R9PKAUkAsRPLhLoBDLBUYtKS0=:MujSgKe6rxcIUMv4GnyXJmV0YNbf39uFRZv724+X1FE="
"dbuser_monitor" "SCRAM-SHA-256$4096:fwU97ZMO/KR0ScHO5+UuBg==$CrNsmGrx1DkIGrtrD1Wjexb/aygzqQdirTO1oBZROPY=:L8+dJ+fqlMQh7y4PmVR/gbAOvYWOr+KINjeMZ8LlFww="
"dbuser_meta" "SCRAM-SHA-256$4096:leB2RQPcw1OIiRnPnOMUEg==$eyC+NIMKeoTxshJu314+BmbMFpCcspzI3UFZ1RYfNyU=:fJgXcykVPvOfro2MWNkl5q38oz21nSl1dTtM65uYR1Q="
"dbuser_kong" "SCRAM-SHA-256$4096:bK8sLXIieMwFDz67/0dqXQ==$P/tCRgyKx9MC9LH3ErnKsnlOqgNd/nn2RyvThyiK6e4=:CDM8QZNHBdPf97ztusgnE7olaKDNHBN0WeAbP/nzu5A="
"dbuser_grafana" "SCRAM-SHA-256$4096:HjLdGaGmeIAGdWyn2gDt/Q==$jgoyOB8ugoce+Wqjr0EwFf8NaIEMtiTuQTg1iEJs9BM=:ed4HUFqLyB4YpRr+y25FBT7KnlFDnan6JPVT9imxzA4="
"dbuser_gitea" "SCRAM-SHA-256$4096:l1DBGCc4dtircZ8O8Fbzkw==$tpmGwgLuWPDog8IEKdsaDGtiPAxD16z09slvu+rHE74=:pYuFOSDuWSofpD9OZhG7oWvyAR0PQjJBffgHZLpLHds="
"dbuser_dba" "SCRAM-SHA-256$4096:zH8niABU7xmtblVUo2QFew==$Zj7/pq+ICZx7fDcXikiN7GLqkKFA+X5NsvAX6CMshF0=:pqevR2WpizjRecPIQjMZOm+Ap+x0kgPL2Iv5zHZs0+g="
"dbuser_bytebase" "SCRAM-SHA-256$4096:OMoTM9Zf8QcCCMD0svK5gg==$kMchqbf4iLK1U67pVOfGrERa/fY818AwqfBPhsTShNQ=:6HqWteN+AadrUnrgC0byr5A72noqnPugItQjOLFw0Wk="

User-level connection pool parameters are maintained in a separate file: /etc/pgbouncer/useropts.txt, for example:

dbuser_dba                  = pool_mode=session max_user_connections=16
dbuser_monitor              = pool_mode=session max_user_connections=8

When you create a database, Pigsty refreshes the Pgbouncer database list. An online reload applies the changes without affecting existing connections.

Pgbouncer runs as the same dbsu as PostgreSQL. The default is the postgres operating system user. As dbsu, use the pgb alias for Pgbouncer administration.

When you create users, Pigsty refreshes userlist.txt and useropts.txt automatically. An online reload applies these pool user changes, normally without affecting existing connections.

Use pgbouncer_auth_query to authenticate pool users with dynamic queries. This is an alternative when you do not want to manage pool users directly.

17.4 - Database

Database refers to the logical object created using the SQL command CREATE DATABASE within a database cluster.

In this context, Database refers to the logical object created using the SQL command CREATE DATABASE within a database cluster.

A PostgreSQL server can serve multiple databases at the same time. In Pigsty, define the required databases in the cluster configuration.

Pigsty customizes the default template database, template1. It creates default schemas, installs default extensions, and configures default privileges. New databases inherit these settings from template1 by default.

By default, Pigsty adds each business database to the Pgbouncer connection pool with a 1:1 mapping. pg_exporter automatically discovers all business databases and monitors their objects.


Define Database

Define business databases in the cluster parameter pg_databases. This parameter is an array of database definition objects. Pigsty creates databases in definition order. Thus, a later database can use an earlier database as its template.

Below is the database definition for the default pg-meta cluster in the Pigsty demo environment:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
      - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }

Each database definition is an object. The following meta definition shows the available fields:

- name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
  baseline: cmdb.sql              # optional, database sql baseline path (relative path among ansible search path, e.g. files/)
  pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
  schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
  extensions:                     # optional, additional extensions to be installed: array of extension objects
    - { name: postgis , schema: public }  # can specify which schema to install the extension in, or leave it unspecified (will install in the first schema of search_path)
    - { name: timescaledb }               # for example, some extensions create and use fixed schemas, so no schema specification is needed.
  comment: pigsty meta database   # optional, comment string for this database
  owner: postgres                 # optional, database owner, postgres by default
  template: template1             # optional, which template to use, template1 by default, target must be a template database
  encoding: UTF8                  # optional, database encoding, UTF8 by default (MUST same as template database)
  locale: C                       # optional, database locale, C by default (MUST same as template database)
  lc_collate: C                   # optional, database collate, C by default (MUST same as template database), no reason not to recommend changing.
  lc_ctype: C                     # optional, database ctype, C by default (MUST same as template database)
  tablespace: pg_default          # optional, default tablespace, 'pg_default' by default
  allowconn: true                 # optional, allow connection, true by default. false will disable connect at all
  revokeconn: false               # optional, revoke public connection privilege. false by default, when set to true, CONNECT privilege will be revoked from users other than owner and admin
  register_datasource: true       # optional, register this database to grafana datasources? true by default, explicitly set to false to skip registration
  connlimit: -1                   # optional, database connection limit, default -1 disable limit, set to positive integer will limit connections
  pool_auth_user: dbuser_meta     # optional, all connections to this pgbouncer database will be authenticated using this user (only useful when pgbouncer_auth_query is enabled)
  pool_mode: transaction          # optional, pgbouncer pool mode at database level, default transaction
  pool_size: 50                   # optional, pgbouncer pool size at database level, default 50
  pool_reserve: 30                # optional, pgbouncer reserve pool at database level, default 30
  pool_size_min: 0                # optional, pgbouncer pool size min at database level, default 0
  pool_connlimit: 100             # optional, max database connections at database level, default 100

The only required field is name. It should be a valid database name that is unique within the PostgreSQL cluster. Other parameters have defaults.

  • name: Database name, required.
  • baseline: SQL file path (Ansible search path, usually in files), used to initialize database content.
  • owner: Database owner, default is postgres
  • template: Template used when creating the database, default is template1
  • encoding: Database default character encoding, default is UTF8, default is consistent with the instance. It is recommended not to configure and modify.
  • locale: Database default locale, default is C, it is recommended not to configure, keep consistent with the instance.
  • lc_collate: Database default locale string collation, default is same as instance setting, it is recommended not to modify, must be consistent with template database. It is strongly recommended not to configure, or configure to C.
  • lc_ctype: Database default LOCALE, default is same as instance setting, it is recommended not to modify or set, must be consistent with template database. It is recommended to configure to C or en_US.UTF8.
  • allowconn: Whether to allow connection to the database, default is true, not recommended to modify.
  • revokeconn: Whether to revoke connection privilege to the database? Default is false. If true, PUBLIC CONNECT privilege on the database will be revoked. Only default users (dbsu|monitor|admin|replicator|owner) can connect. In addition, admin|owner will have GRANT OPTION, can grant connection privileges to other users.
  • tablespace: Tablespace associated with the database, default is pg_default.
  • connlimit: Database connection limit, default is -1, meaning no limit.
  • extensions: Object array, each object defines an extension in the database, and the schema in which it is installed.
  • parameters: KV object, each KV defines a parameter that needs to be modified for the database through ALTER DATABASE.
  • pgbouncer: Boolean option, whether to add this database to Pgbouncer. All databases will be added to Pgbouncer list unless explicitly specified as pgbouncer: false.
  • comment: Database comment information.
  • pool_auth_user: When pgbouncer_auth_query is enabled, all connections to this pgbouncer database will use the user specified here to execute authentication queries. You need to use a user with access to the pg_shadow table.
  • pool_mode: Database level pgbouncer pool mode, default is transaction, i.e., transaction pooling. If left empty, will use pgbouncer_poolmode parameter as default value.
  • pool_size: Database-level Pgbouncer default pool size, default 50.
  • pool_reserve: Database-level Pgbouncer reserve pool, default 30; when the regular pool is exhausted, at most this many burst connections can be added.
  • pool_size_min: Database level pgbouncer pool size min, default is 0
  • pool_connlimit: Database level pgbouncer connection pool max database connections, default is 100

By default, new databases use template1 as their template. During PG_PROVISION, Pigsty configures this template with extensions, schemas, and default privileges. New databases inherit this configuration unless you explicitly select another template.

For database access privileges, see Access Control: Database Isolation.


Create Database

Pigsty creates databases defined in pg_databases during cluster initialization. To create a database on an existing cluster, use bin/pgsql-db. Add the definition to all.children.<cls>.pg_databases, then run:

bin/pgsql-db <cls> <dbname>    # pgsql-db.yml -l <cls> -e dbname=<dbname>

Here are some considerations when creating a new database:

The database creation playbook is idempotent by default. However, a baseline script can change this behavior. Do not usually repeat the playbook on an existing database unless you know that its baseline SQL is also idempotent.

Manual database creation is not recommended, especially with the default Pgbouncer connection pool. If you create databases manually, you must maintain the Pgbouncer database list and keep it consistent with PostgreSQL. The pgsql-db tool and pgsql-db.yml playbook also add new databases to the Pgbouncer database list.

If the database owner differs from the default dbsu, postgres, verify that the owner user exists first. Always create users before you create databases.


Pgbouncer Database

By default, Pigsty enables one Pgbouncer connection pool for each PostgreSQL instance. The pool connects through the /var/run/postgresql Unix socket.

Connection pools can improve performance for short connections and reduce concurrency contention. They prevent excessive connections from overloading the database. They also provide more options during database migration.

By default, Pigsty adds all databases from pg_databases to the Pgbouncer database list. To exclude a database, set pgbouncer: false in its definition.

The Pgbouncer database list is in /etc/pgbouncer/database.txt. This file includes pool parameters from each database definition:

meta                        = host=/var/run/postgresql mode=session
grafana                     = host=/var/run/postgresql mode=transaction
bytebase                    = host=/var/run/postgresql auth_user=dbuser_meta
kong                        = host=/var/run/postgresql pool_size=32 reserve_pool=64
gitea                       = host=/var/run/postgresql min_pool_size=10
wiki                        = host=/var/run/postgresql
noco                        = host=/var/run/postgresql
mongo                       = host=/var/run/postgresql

When you create databases, Pigsty refreshes the Pgbouncer database list. An online configuration reload applies the changes, normally without affecting existing connections.

Pgbouncer runs as the same dbsu as PostgreSQL. The default is the postgres operating system user. As dbsu, use the pgb alias for Pgbouncer administration.

To route a managed Pgbouncer database to another node, edit /etc/pgbouncer/database.txt. Then reload the configuration and rebuild existing server connections:

# Change only mydb's backend target
sed -i -E '/^mydb[[:space:]]*=/ s#host=[^[:space:]]+#host=10.10.10.12#' /etc/pgbouncer/database.txt
pgb -c "RELOAD;"
pgb -c "RECONNECT mydb;"
pgb -c "WAIT_CLOSE mydb;"

The current pgb-route helper edits only /etc/pgbouncer/pgbouncer.ini. Pigsty stores managed database routes in the included database.txt file. Thus, this helper does not change managed routes.

17.5 - Authentication / HBA

Detailed explanation of Host-Based Authentication (HBA) in Pigsty.

Detailed explanation of Host-Based Authentication (HBA) in Pigsty.

Authentication supports Access Control and Default Privileges. PostgreSQL supports several authentication methods.

This page describes HBA, or Host-Based Authentication. HBA rules specify which users can access each database, from which locations, and with which authentication methods.


Client Authentication

Users must authenticate before they connect to a PostgreSQL database. By default, they use a password.

You can put the password in the connection string, but this is not secure. Alternatively, use the PGPASSWORD environment variable or a .pgpass file. For details, see psql and PostgreSQL Connection Strings.

psql 'host=<host> port=<port> dbname=<dbname> user=<username> password=<password>'
psql postgres://<username>:<password>@<host>:<port>/<dbname>
PGPASSWORD=<password>; psql -U <username> -h <host> -p <port> -d <dbname>

For example, to connect to Pigsty’s default meta database, you can use the following connection strings:

psql 'host=10.10.10.10 port=5432 dbname=meta user=dbuser_dba password=DBUser.DBA'
psql postgres://dbuser_dba:[email protected]:5432/meta
PGPASSWORD=DBUser.DBA; psql -U dbuser_dba -h 10.10.10.10 -p 5432 -d meta

By default, Pigsty enables server-side SSL encryption but does not verify client SSL certificates. To use client certificates, set PGSSLCERT and PGSSLKEY, or the sslkey and sslcert connection parameters.

psql 'postgres://dbuser_dba:[email protected]:5432/meta?sslkey=/path/to/dbuser_dba.key&sslcert=/path/to/dbuser_dba.crt'

Use cert.yml to sign client certificates with the local CA. Set the certificate CN to the username.


Defining HBA

In Pigsty, there are four parameters related to HBA rules:

Each parameter is an array of HBA rule objects. A rule object can use either of these forms:

1. Raw Form

The raw form of HBA is almost identical to the PostgreSQL pg_hba.conf format:

- title: allow intranet password access
  role: common
  rules:
    - host   all  all  10.0.0.0/8      md5
    - host   all  all  172.16.0.0/12   md5
    - host   all  all  192.168.0.0/16  md5

The rules field is an array of strings. Each string contains one raw HBA rule. The generated file uses title as a comment to explain the following rules.

The role field selects the instance roles to which the rule applies. Pigsty adds the rule when the instance pg_role matches role.

  • role: common: Add the rule to all instances.
  • role: primary: Add the rule only to primary instances.
  • role: replica: Add the rule only to replica instances.
  • role: offline: Add the rule to offline instances where pg_role = offline or pg_offline_query = true.

2. Alias Form

The alias form simplifies HBA rule maintenance. It replaces rules with the addr, auth, user, and db fields. The title and role fields still apply.

- addr: 'intra'    # world|intra|infra|admin|local|localhost|cluster|<cidr>
  auth: 'pwd'      # trust|pwd|ssl|cert|deny|<official auth method>
  user: 'all'      # all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>
  db: 'all'        # all|replication|....
  rules: []        # raw hba string precedence over above all
  title: allow intranet password access
  • addr: where - Which IP address ranges are affected by this rule?
    • world: All IP addresses
    • intra: All intranet IP address ranges: '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'
    • infra: IP addresses of Infra nodes
    • admin: IP addresses of admin_ip management nodes
    • local: Local Unix Socket
    • localhost: Local Unix Socket and TCP 127.0.0.1/32 loopback address
    • cluster: IP addresses of all members in the same PostgreSQL cluster
    • <cidr>: A specific CIDR address block or IP address
  • auth: how - What authentication method does this rule specify?
    • deny: Deny access
    • trust: Trust directly, no authentication required
    • pwd: Password authentication, uses md5 or scram-sha-256 authentication based on the pg_pwd_enc parameter
    • sha/scram-sha-256: Force use of scram-sha-256 password authentication.
    • md5: md5 password authentication, but can also be compatible with scram-sha-256 authentication, not recommended.
    • ssl: On top of password authentication pwd, require SSL to be enabled
    • ssl-md5: On top of password authentication md5, require SSL to be enabled
    • ssl-sha: On top of password authentication sha, require SSL to be enabled
    • os/ident: Use ident authentication with the operating system user identity
    • peer: Use peer authentication method, similar to os ident
    • cert: Use client SSL certificate-based authentication, certificate CN is the username
  • user: who: Which users are affected by this rule?
  • db: which: Which databases are affected by this rule?
    • all: All databases
    • replication: Allow replication connections (not specifying a specific database)
    • A specific database

3. Definition Location

Usually, global HBA rules belong in all.vars. To change the defaults, copy them from conf/ha/full.yml to all.vars, then edit them.

Cluster-specific HBA rules are defined in the database cluster-level configuration:

Here are some examples of cluster HBA rule definitions:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:
      - { user: dbuser_view ,db: all    ,addr: infra        ,auth: pwd  ,title: 'Allow dbuser_view password access to all databases from infrastructure nodes'}
      - { user: all         ,db: all    ,addr: 100.0.0.0/8  ,auth: pwd  ,title: 'Allow all users password access to all databases from K8S network'          }
      - { user: '${admin}'  ,db: world  ,addr: 0.0.0.0/0    ,auth: cert ,title: 'Allow admin user to login from anywhere with client certificate'       }

Reloading HBA

HBA rules are static configuration. Reload them after changes to apply the new rules. The default rules usually need no reload for role or membership changes because they do not depend on those values.

Some HBA rules restrict access by instance role or cluster membership. Adding or removing instances, or failover, can change where these rules apply. In these cases, reload HBA to apply the latest changes.

To reload postgres/pgbouncer hba rules:

bin/pgsql-hba <cls>                 # Reload hba rules for cluster `<cls>`
bin/pgsql-hba <cls> ip1 ip2...      # Reload hba rules for specific instances

The underlying Ansible playbook commands actually executed are:

./pgsql.yml -l <cls> -e pg_reload=true -t pg_hba,pg_reload
./pgsql.yml -l <cls> -e pg_reload=true -t pgbouncer_hba,pgbouncer_reload

Default HBA

Pigsty provides default HBA rules with sufficient security for most uses. The rules use the alias form to make their purpose clear.

pg_default_hba_rules:             # postgres global default HBA rules 
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost'}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' }
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' }
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password'}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'   }
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket'}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     }
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet'}
pgb_default_hba_rules:            # pgbouncer global default HBA rules 
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' }
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' }
Example: Rendered pg_hba.conf
#==============================================================#
# File      :   pg_hba.conf
# Desc      :   Postgres HBA Rules for pg-meta-1 [primary]
# Time      :   2023-01-11 15:19
# Host      :   pg-meta-1 @ 10.10.10.10:5432
# Path      :   /pg/data/pg_hba.conf
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Author    :   Ruohang Feng ([email protected])
# License   :   Apache-2.0
#==============================================================#

# addr alias
# local     : /var/run/postgresql
# admin     : 10.10.10.10
# infra     : 10.10.10.10
# intra     : 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

# user alias
# dbsu    :  postgres
# repl    :  replicator
# monitor :  dbuser_monitor
# admin   :  dbuser_dba

# dbsu access via local os user ident [default]
local    all                postgres                              ident

# dbsu replication from local os ident [default]
local    replication        postgres                              ident

# replicator replication from localhost [default]
local    replication        replicator                            scram-sha-256
host     replication        replicator         127.0.0.1/32       scram-sha-256

# replicator replication from intranet [default]
host     replication        replicator         10.0.0.0/8         scram-sha-256
host     replication        replicator         172.16.0.0/12      scram-sha-256
host     replication        replicator         192.168.0.0/16     scram-sha-256

# replicator postgres db from intranet [default]
host     postgres           replicator         10.0.0.0/8         scram-sha-256
host     postgres           replicator         172.16.0.0/12      scram-sha-256
host     postgres           replicator         192.168.0.0/16     scram-sha-256

# monitor from localhost with password [default]
local    all                dbuser_monitor                        scram-sha-256
host     all                dbuser_monitor     127.0.0.1/32       scram-sha-256

# monitor from infra host with password [default]
host     all                dbuser_monitor     10.10.10.10/32     scram-sha-256

# admin @ infra nodes with pwd & ssl [default]
hostssl  all                dbuser_dba         10.10.10.10/32     scram-sha-256

# admin @ everywhere with ssl & pwd [default]
hostssl  all                dbuser_dba         0.0.0.0/0          scram-sha-256

# pgbouncer read/write via local socket [default]
local    all                +dbrole_readonly                      scram-sha-256
host     all                +dbrole_readonly   127.0.0.1/32       scram-sha-256

# read/write biz user via password [default]
host     all                +dbrole_readonly   10.0.0.0/8         scram-sha-256
host     all                +dbrole_readonly   172.16.0.0/12      scram-sha-256
host     all                +dbrole_readonly   192.168.0.0/16     scram-sha-256

# allow etl offline tasks from intranet [default]
host     all                +dbrole_offline    10.0.0.0/8         scram-sha-256
host     all                +dbrole_offline    172.16.0.0/12      scram-sha-256
host     all                +dbrole_offline    192.168.0.0/16     scram-sha-256

# allow application database intranet access [common] [DISABLED]
#host    kong            dbuser_kong         10.0.0.0/8          md5
#host    bytebase        dbuser_bytebase     10.0.0.0/8          md5
#host    grafana         dbuser_grafana      10.0.0.0/8          md5
Example: Rendered pgb_hba.conf
#==============================================================#
# File      :   pgb_hba.conf
# Desc      :   Pgbouncer HBA Rules for pg-meta-1 [primary]
# Time      :   2023-01-11 15:28
# Host      :   pg-meta-1 @ 10.10.10.10:5432
# Path      :   /etc/pgbouncer/pgb_hba.conf
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Author    :   Ruohang Feng ([email protected])
# License   :   Apache-2.0
#==============================================================#

# PGBOUNCER HBA RULES FOR pg-meta-1 @ 10.10.10.10:6432
# ansible managed: 2023-01-11 14:30:58

# addr alias
# local     : /var/run/postgresql
# admin     : 10.10.10.10
# infra     : 10.10.10.10
# intra     : 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

# user alias
# dbsu    :  postgres
# repl    :  replicator
# monitor :  dbuser_monitor
# admin   :  dbuser_dba

# dbsu local admin access with os ident [default]
local    pgbouncer          postgres                              peer

# allow all user local access with pwd [default]
local    all                all                                   scram-sha-256
host     all                all                127.0.0.1/32       scram-sha-256

# monitor access via intranet with pwd [default]
host     pgbouncer          dbuser_monitor     10.0.0.0/8         scram-sha-256
host     pgbouncer          dbuser_monitor     172.16.0.0/12      scram-sha-256
host     pgbouncer          dbuser_monitor     192.168.0.0/16     scram-sha-256

# reject all other monitor access addr [default]
host     all                dbuser_monitor     0.0.0.0/0          reject

# admin access via intranet with pwd [default]
host     all                dbuser_dba         10.0.0.0/8         scram-sha-256
host     all                dbuser_dba         172.16.0.0/12      scram-sha-256
host     all                dbuser_dba         192.168.0.0/16     scram-sha-256

# reject all other admin access addr [default]
host     all                dbuser_dba         0.0.0.0/0          reject

# allow all user intra access with pwd [default]
host     all                all                10.0.0.0/8         scram-sha-256
host     all                all                172.16.0.0/12      scram-sha-256
host     all                all                192.168.0.0/16     scram-sha-256

Security Hardening

For higher security, use the hardened conf/ha/safe.yml template. It uses these default HBA rules:

pg_default_hba_rules:             # postgres host-based auth rules by default
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: ssl   ,title: 'replicator replication from localhost'}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: ssl   ,title: 'replicator replication from intranet' }
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: ssl   ,title: 'replicator postgres db from intranet' }
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: ssl   ,title: 'monitor from infra host with password'}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: cert  ,title: 'admin @ everywhere with ssl & cert'   }
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: ssl   ,title: 'pgbouncer read/write via local socket'}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: ssl   ,title: 'read/write biz user via password'     }
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: ssl   ,title: 'allow etl offline tasks from intranet'}
pgb_default_hba_rules:            # pgbouncer host-based authentication rules
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: ssl   ,title: 'monitor access via intranet with pwd' }
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: ssl   ,title: 'admin access via intranet with pwd'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: ssl   ,title: 'allow all user intra access with pwd' }

See Authentication for methods and default boundaries, and Security Considerations for production hardening.