kafgres

Overview

PGSTY targets PG16; requires preload and broker readiness before topic SQL; segment logs need separate replication and archiving.

Version

Build

You can build the RPM / DEB packages for kafgres using pig build:

pig build pkg kafgres         # build RPM / DEB packages

Install

You can install kafgres directly. First, make sure the PGDG and PIGSTY repositories are added and enabled:

pig repo add pgsql -u          # Add repo and update cache

Install the extension using pig or apt/yum/dnf:

pig install kafgres;          # Install for current active PG version

pig ext install -y kafgres -v 16  # PG 16

dnf install -y kafgres_16       # PG 16

apt install -y postgresql-16-kafgres   # PG 16

Preload:

shared_preload_libraries = 'kafgres';

Create Extension:

CREATE EXTENSION kafgres;

Usage

Sources:

• README 0.3.0

• Configuration 0.3.0

• 0.3.0 release

kafgres embeds a Kafka protocol broker in PostgreSQL. Upstream release 0.3.0 targets PostgreSQL 16 using pgrx 0.16.1. It needs superuser installation, shared preload and a restart. Its license is Elastic License 2.0.

Enable the broker

shared_preload_libraries = 'kafgres'
kafgres.database = 'postgres'
kafgres.bind_host = '127.0.0.1'
kafgres.advertised_host = '127.0.0.1'
kafgres.port = 9092

CREATE EXTENSION kafgres;
SELECT kafgres_create_topic('demo', 1);
BEGIN;
SELECT kafgres_produce('demo', 'key', 'value');
COMMIT;
SELECT * FROM kafgres_partition_offsets('demo');

Kafka clients connect to the configured broker port. SQL production participates in the caller’s transaction. Configure TLS, authentication and ACLs before exposing the listener beyond a trusted local environment.

Storage and CDC

kafgres.storage_engine defaults to segment; its log uses separate files and requires the extension’s replication and archive procedures. Configure kafgres.segment_archive_command and monitor kafgres_archive_status() before relying on segment retention and recovery. Ordinary PostgreSQL WAL/PITR does not cover the entire segment log. The table engine keeps its log in PostgreSQL tables; changing engines does not migrate existing records.

CDC additionally requires wal_level = logical; some PostgreSQL builds also require an output_plugin_libraries allowlist. Version 0.3.0 supports SQL CDC mappings with projection and filtering. Review the mapping and recovery procedures before deployment; the release artifacts target PostgreSQL 16, and Cargo feature names alone do not prove support for other majors.

Durability Settings

Version 0.3.0 defaults kafgres.fsync_before_ack to on and kafgres.relaxed_produce_commit to off. Relaxing the first can lose acknowledged segment records on a power failure; relaxing the second can lose the newest idempotent-producer state after a crash and permit duplicates after retries. These settings have narrower scope than transactional SQL production and do not apply uniformly to the table engine. Preserve the strict defaults until the durability tradeoff is deliberate.