Module: KAFKA

Kafka is a distributed event-streaming platform. Pigsty’s KAFKA module installs Apache Kafka 4.1+ dynamic KRaft clusters on managed nodes from RPM/DEB packages. It manages security, resources, lifecycle, and observability.

Module Capabilities

The KAFKA module currently provides:

• Native dynamic KRaft, without ZooKeeper or static controller.quorum.voters.
• Native roles combined, broker, and controller, with combined or separate control and data planes.
• Random Cluster ID and Controller Directory IDs for new clusters. A minimal bootstrap manifest freezes these values and blocks conflicting operations.
• Lifecycle selection from live health: cold start or repair, serial broker admission, dynamic controller join, or strict single-node rolling restart.
• Checks before and after each rolling step: controller majority, voter catch-up, offline partitions, under-min-ISR, and ISR catch-up.
• Member retirement and failed-node replacement through playbooks. kafka-rm.yml retires a strict subset, including dead nodes. Node replacement uses three commands.
• plaintext and production scram profiles. The latter provides TLS, SCRAM-SHA-512, controller mTLS, ACLs, and default-deny authorization.
• Declarative topics, credentials, ACLs, and quotas without implicit deletion. Protected internal credential and certificate rotation.
• JMX and protocol exporters, 19 recording rules, 15 alert rules, four Grafana dashboards, and VictoriaLogs storage.

Module Architecture

NODE provides node management, package repositories, and base monitoring. INFRA provides VictoriaMetrics, VictoriaLogs, Grafana, and Alertmanager. KAFKA depends on both.

Every Kafka JVM includes JMX Exporter and registers as job=kafka. kafka_exporter runs on the first two broker-capable nodes, ordered by kafka_seq. A single broker runs one exporter; pure controllers run none. Both exporters report the same logical cluster. Recording rules deduplicate their metrics before aggregation.

Documentation

First Time

Follow Quickstart to build a single-node development cluster, then a three-node TLS/SCRAM/ACL cluster. Continue with client access, parameter and resource changes, and deployment checks.

If you know Kafka and Pigsty, start with Cluster Config or Parameters.

Default Ports

All four ports must be different. Parameters can change each port. Restrict JMX and protocol exporter HTTP ports to the monitoring network through firewalls.

Current Boundaries

This role provides a core Kafka deployment baseline. A complete streaming platform or managed service requires additional components or procedures for:

• Existing partition reassignment and replica balancing after expansion. Playbooks manage member join, retirement, and replacement, but data movement needs an explicit plan.
• Increasing the frozen default.replication.factor after expansion. Kafka 4.3 requires explicit data migration and a static-configuration maintenance window.
• Changing existing topic replication factors, deleting topics, and deleting users.
• Migrating formatted clusters online from plaintext to scram.
• Kafka upgrades, feature-level finalization, backup, restore, and disaster exercises.
• Multiple listeners, NAT or public addresses, multiple client networks per broker, and tiered storage.
• Kafka Connect, Schema Registry, MirrorMaker 2, Cruise Control, and web interfaces.

Document these limits in production plans, approval processes, and exercises. Running the ordinary inventory again does not perform these operations.