Configuration

The KAFKA module expresses cluster intent through 15 persistent public parameters; everything else — topology, listeners, storage subdirectories, replication safety, authorization, and Exporter placement — is derived by the role in a single, consistent way. For a first deployment, start with the Quickstart; for the full field reference, see Parameters.

Pre-Deployment Checklist

Before filling in the inventory, confirm at least the following:

• The target hosts are already commissioned by NODE, the software repository is reachable, and inventory_hostname is directly routable by every Kafka member and client.
• A single operation will use -l to select precisely all members of one kafka_cluster — not a single node, a subset of members, or multiple clusters.
• kafka_seq is unique within the cluster, the controller count is odd, and the broker count matches your fault domains and capacity targets.
• Ports 9092, 9093, 9308, and 9404 do not collide, and Infra nodes can reach both metrics ports.
• kafka_data maps to a dedicated filesystem, sized for retention, write peaks, replication traffic, recovery time, and growth headroom.
• Production uses kafka_security: scram; node and admin clocks are synchronized, the Pigsty CA is available, and application passwords come from a secret source such as Vault.
• Topic partitions, replicas, min.insync.replicas, and retention policy — along with client acks, retries, and consumer recovery strategy — have been reviewed.
• Scale-out and scale-in, partition reassignment, upgrades, backup, restore, and controller membership changes each have their own runbook.

Roles and Topology

kafka_role accepts only three values:

kafka_role is declared independently per node, and defaults to combined when omitted. A cluster must contain at least one controller-capable node and at least one broker-capable node; an even number of controllers produces a warning, and production typically uses 3 controllers.

Single-Node Development Cluster

A single node serves as both broker and controller. It cannot tolerate a node failure and is suitable only for development, testing, and feature validation:

kf-dev:
  hosts:
    10.10.10.10: { kafka_seq: 1 }
  vars:
    kafka_cluster: kf-dev

The role derives RF=1 and minISR=1 from the initial broker count. Do not use a single-node topology or the default plaintext security profile directly in production.

Three-Node Combined Deployment

All three nodes serve as both broker and controller — a compact production starting point. Omit every role field to use the default combined:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1 }
    10.10.10.12: { kafka_seq: 2 }
    10.10.10.13: { kafka_seq: 3 }
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms4G -Xmx4G'
    kafka_security: scram
    kafka_parameters:
      num.partitions: 3
      num.network.threads: 6
      num.io.threads: 16
    kafka_topics:
      - name: order.events
        partitions: 12
        replication_factor: 3
        config:
          min.insync.replicas: 2
          cleanup.policy: delete

The initial three brokers automatically get the role-owned replication policy of RF=3 and minISR=2. You neither need nor are allowed to override the internal-topic RF, default.replication.factor, or min.insync.replicas in kafka_parameters. The 4G heap in the example is only illustrative; in production, balance the JVM heap, the operating-system page cache, and any other processes on the same host through load testing.

Separating Controllers and Brokers

Critical or larger clusters can separate the control plane from the data plane. Because explicit roles are present, every member must declare its role:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1, kafka_role: controller }
    10.10.10.12: { kafka_seq: 2, kafka_role: controller }
    10.10.10.13: { kafka_seq: 3, kafka_role: controller }
    10.10.10.21: { kafka_seq: 4, kafka_role: broker }
    10.10.10.22: { kafka_seq: 5, kafka_role: broker }
    10.10.10.23: { kafka_seq: 6, kafka_role: broker }
  vars:
    kafka_cluster: kf-main
    kafka_security: scram

A controller-only node does not listen on 9092 and does not run the protocol Exporter; it still exposes KRaft and JVM state through JMX. At most two kafka_exporter instances are placed on the broker-capable nodes with the lowest kafka_seq.

Dynamic KRaft and the Bootstrap Manifest

A new cluster uses the dynamic quorum directly: every node renders controller.quorum.bootstrap.servers, and no static controller.quorum.voters is generated. At the first format:

• The Cluster ID is generated randomly, not hashed from the cluster name;
• The Directory IDs of the initial controllers are generated randomly and frozen;
• Each node is formatted explicitly with either --initial-controllers or --no-initial-controllers mode;
• After startup, the role waits for the dynamic quorum to elect a leader, and verifies that every initial controller’s Directory ID is present in the live quorum.

The bootstrap-only facts are kept in the admin-side cache:

files/kafka/<kafka_cluster>/manifest.yml

At the same time, every cluster member keeps an authoritative copy at /etc/kafka/manifest.yml (a scram cluster also has /etc/kafka/secrets.yml). The manifest records only the cluster identity, the initial controller identities, the security profile, and the initial RF/minISR. The live cluster is always the authority on runtime facts:

• When the manifest conflicts with the live identity or security profile, ordinary playbooks fail closed;
• When an old manifest exists but all data disks are empty, reviving the old cluster is refused;
• When the admin-side cache is lost, it can be recovered automatically from any member’s node copy, without reformatting;
• When no manifest can be found on either the admin node or any member while storage is already formatted, the role fails closed and asks you to restore the manifest first;
• An already-formatted scram cluster likewise fails closed when neither the admin node nor any member holds the secret material.

Adding, replacing, or removing a controller is not an ordinary inventory operation. A new controller must be explicitly formatted against the existing cluster, started, and caught up, and then registered with Kafka add-controller; removal requires the corresponding remove-controller procedure. The role refuses to add an unregistered new controller to the voter set based on inventory alone.

Identity Parameters

An already-formatted node reads cluster.id and node.id from ${kafka_data}/metadata/meta.properties and cross-checks them against the manifest and the inventory; the initial controllers’ Directory IDs are compared against the live quorum after startup. An identity mismatch is a protective failure and must not be worked around by deleting meta.properties or wiping data.

Network and Listeners

The role exposes only ports — not bind addresses, advertised addresses, or the listener map:

The fixed listener conventions are as follows:

• The broker listener binds 0.0.0.0, and the controller listener binds inventory_hostname;
• The broker’s advertised.listeners uses inventory_hostname;
• The controller bootstrap address also uses inventory_hostname;
• plaintext: both BROKER and CONTROLLER use PLAINTEXT;
• scram: BROKER uses SASL_SSL + SCRAM-SHA-512, and CONTROLLER uses mutual TLS.

Clients must therefore be able to resolve and directly reach every broker’s inventory_hostname. The current v1 does not support NAT, public-address mapping, multiple client networks for the same broker, or arbitrary raw listener overrides; these scenarios cannot be assembled around through kafka_parameters.

Kafka’s standard access model is a smart client connecting directly to brokers: bootstrap.servers is configured with several seed addresses, and once the client fetches cluster metadata it connects directly to the partition leaders. HAProxy, a Keepalived VIP, or a cloud LB should not be used as the regular Kafka data-plane entry point, because they are unaware of Kafka metadata and partition leaders and cannot exempt clients from reaching every advertised.listeners address. A DNS or TCP LB can at most serve as an optional bootstrap discovery entry point; even then, the application network must still reach all brokers directly. See Quickstart: Connecting Application Clients for details.

The minimum required network flows:

The metrics ports are HTTP, and even when Kafka uses scram they should be restricted to the monitoring network by firewall.

Storage, Heap, and Rack

You set only the root directory:

kafka_data: /data/kafka

The role derives the topic data directory ${kafka_data}/data and the KRaft metadata directory ${kafka_data}/metadata in a fixed way. kafka_data must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg.

Production planning should account for at least retention, message peaks, replication traffic, partition/segment counts, disk latency and throughput, file descriptors, recovery time, JVM heap, and page cache. The current role generates only a single log.dirs; multi-disk JBOD, disk replacement, and automatic data migration require separate runbooks.

For deployments spanning fault domains, declare kafka_rack consistently on all broker-capable nodes:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }
10.10.10.22: { kafka_seq: 5, kafka_role: broker, kafka_rack: az-b }
10.10.10.23: { kafka_seq: 6, kafka_role: broker, kafka_rack: az-c }

Broker-capable nodes must either all set the rack or all omit it. Changing the rack triggers a safe rolling restart but does not automatically migrate existing replicas.

Replication Policy

At the first bootstrap, the policy is derived from the initial broker count:

replication_factor = min(3, broker_count)
min_insync_replicas = max(1, replication_factor - 1)

The initial default RF for future topics, the internal-topic RF, and the cluster minISR are all written into the manifest and frozen. After scaling out:

• default.replication.factor keeps its initial value; Kafka 4.3 does not allow changing it online through dynamic broker configuration;
• The RF of existing internal/business topics is not raised automatically;
• The role does not report “brokers have joined” as “data is balanced”;
• An RF change must use a reviewed kafka-reassign-partitions.sh plan; raising the static default additionally requires controller high availability or an explicit maintenance window, and takes effect through a full-cluster safe rolling restart.

Producer acks, idempotence, retries, batching, and compression are client policy, not parameters of the Kafka broker role.

kafka_parameters

kafka_parameters is the only broker-parameter escape hatch. It defaults to {} and is rendered only onto broker-capable nodes. It is suited to non-role-owned keys such as num.partitions, thread counts, buffers, retention, and segments.

The following patterns are owned by the role and must not be overridden:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

If any reserved key appears, the identity preflight fails outright before any file is written.

Security and Declarative Resources

kafka_security: scram is a complete production profile, not a set of switches to be combined at will. It automatically enables:

• Per-node certificates issued by the Pigsty CA;
• Mutual TLS on the controller listener;
• SASL_SSL + SCRAM-SHA-512 for broker/client and inter-broker traffic;
• StandardAuthorizer, deny-by-default, and role-owned admin/monitoring identities;
• Convergence of the minimal monitoring ACL before the protocol Exporter starts.

Application resources are declared through two domain objects:

kafka_security: scram
kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520
kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete

Resource convergence semantics: topic creation is idempotent, partitions only increase, and only explicitly declared config is updated; an RF change is refused with a prompt to run reassignment. A declared user’s password, ACLs, and the quota fields you provide converge idempotently. Removing a topic or user entry is not an implicit deletion procedure.

The security profile cannot be switched by an ordinary playbook after bootstrap. Internal credentials and certificates can use protected rotation, but an online migration from plaintext to scram still requires an explicit state machine that is planned for the future.

Packages and File Layout

The role installs java-runtime and kafka-stack through platform mappings. The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0; the actual versions still depend on the target platform’s repository and the installed packages.

These files are managed by the role. Persistent intent belongs in pigsty.yml. Do not edit generated files directly on the nodes, and do not copy passwords, private keys, or role-owned secret contents into inventory, logs, or tickets.