Module: Kafka

• 1: Quick Start
• 2: Configuration
• 3: Parameters
• 4: Administration
• 5: Playbook
• 6: Monitoring
• 7: Metrics
• 8: FAQ

flowchart LR
    admin["Pigsty admin node"] -->|"kafka.yml / exact cluster"| kafka["Kafka 4.x / dynamic KRaft"]
    kafka --> jmx["Each Kafka JVM / JMX :9404"]
    kafka --> exporter["Up to two brokers / kafka_exporter :9308"]
    kafka --> journal["Journald"]
    jmx --> vm["VictoriaMetrics"]
    exporter --> vm
    journal --> vector["Vector"] --> vl["VictoriaLogs"]
    vm --> grafana["Grafana"]
    vl --> grafana
    vm --> alert["Alertmanager"]

    style kafka fill:#70C1B3,stroke:#4f968b,color:#fff
    style vm fill:#E66B7A,stroke:#b84e5c,color:#fff
    style vl fill:#C98367,stroke:#9e634e,color:#fff
    style grafana fill:#F29C64,stroke:#c77845,color:#fff

1 - Quick Start

This tutorial starts from a minimal single-node cluster, walking through topic creation and message read/write; it then deploys a separate, secured three-node cluster with application users, ACLs, quotas, and production topics; finally, it demonstrates core parameter changes, client access, monitoring verification, and pre-launch checks.

Learning Path

Before You Begin

Unless noted otherwise, the following commands are run from the project directory on the Pigsty admin node:

cd ~/pigsty

Before you start, confirm that:

• pigsty.yml is the source of configuration for the current environment — back it up and review its existing contents first;
• each Kafka node’s inventory_hostname can be resolved and routed directly by every Kafka member and client;
• the admin node and the Kafka nodes have synchronized clocks;
• ports 9092, 9093, 9308, and 9404 are free of conflicts;
• /data/kafka maps to a dedicated data disk or directory that holds nothing else;
• every kafka.yml run uses -l to select exactly all members of one Kafka cluster;
• before any real change, you run --check first, review the output, and obtain approval for the change.

The inventory must keep the all.children hierarchy. Merge the groups below into your existing pigsty.yml; do not let the examples overwrite your existing all.vars, infra, etcd, pgsql, or other configuration.

Part 1: Deploy a Single-Node Kafka

1. Define the Cluster

Add the following kf-dev group to all.children. This node omits kafka_role, so it uses the default combined and serves as both broker and controller:

all:
  children:
    # keep your existing infra, etcd, pgsql, and other groups

    kf-dev:
      hosts:
        10.10.10.10: { kafka_seq: 1 }
      vars:
        kafka_cluster: kf-dev
        kafka_data: /data/kafka
        kafka_security: plaintext
        kafka_topics:
          - name: quickstart.events
            partitions: 1
            replication_factor: 1
            config:
              retention.ms: 86400000   # 1 day, tutorial only

This configuration yields:

• a random cluster ID;
• a single dynamic KRaft combined node;
• default RF=1 and minISR=1;
• a single-partition topic named quickstart.events;
• a JMX exporter on :9404 and a protocol exporter on :9308.

plaintext has no transport encryption, authentication, or ACLs, so use it only for local development or a trusted, isolated network.

2. Bring the Node Under Management

If this host has not yet been initialized by NODE, run check mode first:

./node.yml --check -l kf-dev

After reviewing the results and getting approval, bring the node under management:

./node.yml -l kf-dev

You can skip this step for a node that Pigsty already manages and whose package repository and time synchronization are healthy. For full NODE preparation and day-to-day management, see Node Administration.

3. Deploy Kafka

Run a check against the full cluster first:

./kafka.yml --check -l kf-dev

Confirm the target is exactly the full membership of kf-dev, review the data path, packages, ports, and configuration changes, then run:

./kafka.yml -l kf-dev

The role installs Java and kafka-stack, generates a random identity and bootstrap manifest, formats the KRaft storage, starts the service, creates the topics, and registers the monitoring targets.

4. Verify the Service and Quorum

Log in to the Kafka node and check the services:

systemctl is-active kafka kafka_exporter
journalctl -u kafka --since '-10 min' --no-pager

Run the role’s own health check:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties

The returned JSON should contain "healthy": true. Continue by checking the dynamic quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

You should see a valid LeaderId, a CurrentVoters list that includes this node, and quickstart.events with RF=1 and ISR=1.

5. Produce and Consume Messages

Start a console producer:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events

Type a few lines of messages, then press Ctrl-D to finish. Consume them from another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

At this point, the single-node deployment, topic convergence, and message read/write are complete. For further status checks, see Day-to-Day Administration.

Part 2: Deploy a Three-Node Production Baseline

The three-node example is a brand-new kf-main cluster built from three combined nodes. It can tolerate the loss of one controller; business topics use RF=3/minISR=2, and the scram production security profile is enabled.

1. Define the Secured Cluster and Its Resources

Add the following group to your existing all.children:

all:
  children:
    # keep your existing groups

    kf-main:
      hosts:
        10.10.10.11: { kafka_seq: 1 }
        10.10.10.12: { kafka_seq: 2 }
        10.10.10.13: { kafka_seq: 3 }
      vars:
        kafka_cluster: kf-main
        kafka_data: /data/kafka
        kafka_heap_opts: '-Xms4G -Xmx4G'
        kafka_security: scram

        kafka_parameters:
          num.partitions: 12
          num.network.threads: 6
          num.io.threads: 16
          log.retention.hours: 168
          log.segment.bytes: 1073741824

        kafka_users:
          - name: quickstart-app
            password: "{{ vault_kafka_quickstart_password }}"
            acls:
              - resource: topic
                name: quickstart.
                pattern: prefixed
                operations: [Read, Write, Describe]
              - resource: group
                name: quickstart.
                pattern: prefixed
                operations: [Read]
              - resource: cluster
                name: kafka-cluster
                operations: [Describe, IdempotentWrite]
            quota:
              producer_byte_rate: 10485760
              consumer_byte_rate: 20971520

        kafka_topics:
          - name: quickstart.events
            partitions: 12
            replication_factor: 3
            config:
              min.insync.replicas: 2
              cleanup.policy: delete
              retention.ms: 604800000

vault_kafka_quickstart_password must be supplied by your existing Ansible Vault, KMS, or another secret-injection mechanism, and must be at least 12 characters. Never commit a real password directly to Git, logs, or tickets.

The key semantics of this configuration:

• all three nodes omit kafka_role, so they consistently use combined;
• the new cluster is bootstrapped directly as dynamic KRaft;
• scram enables node TLS, controller mTLS, SCRAM-SHA-512, ACLs, and deny-by-default all at once;
• with three brokers, the initial replication policy is automatically derived as RF=3 and minISR=2;
• quickstart.events is created explicitly with 12 partitions and 3 replicas;
• quickstart-app can read and write quickstart.* topics, read quickstart.* groups, and use the idempotent producer;
• at most two brokers run kafka_exporter, while all three Kafka JVMs run the JMX exporter.

If the three brokers truly reside in different failure domains, you may add kafka_rack: az-a/az-b/az-c to all nodes respectively. Do not use fictitious rack labels to manufacture a disaster-recovery guarantee that does not exist; for the detailed rules, see Cluster Configuration: Rack.

2. Bring Under Management and Deploy

If the nodes are not yet under management:

./node.yml --check -l kf-main
./node.yml -l kf-main

When deploying Kafka, you must select all three members:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

You cannot use -l 10.10.10.11 alone, nor select kf-dev,kf-main at once. The role rejects a missing, partial, or cross-cluster limit.

3. Verify the Health of All Three Nodes

From the admin node, check the three Kafka services:

ansible kf-main -b -m command -a 'systemctl is-active kafka'

Run the full health check on any broker:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties

Query the quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

Before launch, you should see one active controller, three current voters, and three available brokers; every quickstart.events partition has three replicas with ISR=3, and there are no offline, under-replicated, or under-min-ISR partitions.

Part 3: Connect Application Clients

1. Distribute the CA Public Certificate

Securely copy the public CA certificate from the admin node to the application host:

files/pki/ca/ca.crt  ->  /etc/kafka-client/pigsty-ca.crt

ca.crt is a public certificate that is safe to distribute. Never copy, expose, or distribute files/pki/ca/ca.key. On the application host, the CA file should be owned by root and set read-only. An application host already managed by Pigsty needs no copy: the NODE module has installed the same CA at /etc/pki/ca.crt, which the client can reference directly.

2. Create the Client Configuration

On the application host, create /etc/kafka-client/client.properties:

bootstrap.servers=10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="quickstart-app" password="<secret-from-vault>";

ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka-client/pigsty-ca.crt
ssl.endpoint.identification.algorithm=https

The Kafka Java client supports SASL_SSL + SCRAM and a PEM truststore. A real application should inject the password from a secret manager at runtime rather than committing a file containing the password to the repository. For the complete set of fields, see the Kafka 4.3 SASL/SCRAM and producer configuration references.

3. Why Applications Should Connect Directly to Multiple Brokers

The Kafka client is itself a cluster-aware, intelligent client. bootstrap.servers is used only to obtain the initial metadata; once connected, the client uses that metadata to connect directly to the leader broker of each partition and refreshes its routing whenever a leader changes. The standard practice in production is therefore to:

• configure at least two, and usually three, broker addresses in bootstrap.servers, located in different failure domains;
• allow the application to reach 9092 on all brokers, and ensure that each broker’s advertised inventory_hostname is resolvable and routable;
• let producers and consumers rely on the Kafka client’s own retry, metadata-refresh, idempotence, and consumer-group protocols;
• not place HAProxy, a Keepalived VIP, a layer-4 load balancer, or a layer-7 reverse proxy in front of the Kafka data plane.

A single VIP or LB cannot substitute for the broker addresses in Kafka’s metadata, nor can it transparently forward a connection to the correct partition leader; it also adds complexity around long-lived connection state, fault localization, and capacity planning. If your platform must provide a unified discovery entry point, you can use a DNS name or a TCP load balancer as a bootstrap-only entry point, but each broker’s advertised.listeners must still return an address that the client can reach directly, and the application must not be permitted to reach only the LB.

Scenarios that span NAT, the public internet, Kubernetes, or multiple networks usually require a dedicated externally reachable address/port and an additional listener for each broker. The current module fixes the inventory address as advertised.listeners and does not support arbitrary listener mappings of this kind; do not use a single HAProxy VIP to paper over an address model that does not hold.

4. Authenticate and Read/Write with the Application Identity

On an application host with the Kafka 4.3 CLI installed, run:

kafka-console-producer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events

When consuming, use a group prefix permitted by the ACL:

kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

A production application should also explicitly review its client semantics:

Client-side acks, retries, idempotence, batching, compression, and offset strategy are application configuration and should not be written into the broker’s kafka_parameters.

Part 4: Change Core Parameters

Always express persistent intent for Kafka by editing pigsty.yml, never by editing /etc/kafka/server.properties directly. Common intents map as follows:

Example: Adjust the Heap and Broker Default Parameters

Suppose that after load testing you decide to raise the heap to 6G, increase the thread counts, and change the default retention for new topics to 72 hours:

kf-main:
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms6G -Xmx6G'
    kafka_parameters:
      num.partitions: 12
      num.network.threads: 8
      num.io.threads: 24
      log.retention.hours: 72
      log.segment.bytes: 1073741824

Do not copy 6G/8/24 verbatim; these values must be determined by load-testing against your CPU, memory, connection count, message size, partition count, disk, and page cache.

Example: Add Partitions and Shorten Topic Retention

Increase quickstart.events from 12 partitions to 24 and change the retention to three days:

kafka_topics:
  - name: quickstart.events
    partitions: 24
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 259200000

Partitions cannot be reduced. When replication_factor differs from what is live, the role refuses ordinary convergence and requires an explicit partition reassignment; it never relocates existing replicas automatically.

Apply the Change

Whether you change static parameters or dynamic resources, run the full state machine:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Do not run -t kafka_config alone. The role decides automatically: a static change triggers a strict node-by-node rolling restart; when only dynamic resources such as topics or users change, Kafka is not restarted.

The following keys belong to the role itself and must not be placed in kafka_parameters:

kafka_parameters:
  min.insync.replicas: 2          # wrong: owned by the role
  default.replication.factor: 3   # wrong: owned by the role
  listeners: ...                  # wrong: owned by the role

For all 15 public parameters, their defaults, and the reserved keys, see the Parameter Reference.

Part 5: Key Pre-Launch Checks

Topology and Data Safety

• Production uses at least three brokers and an odd number of controllers; for critical or large clusters, consider a separated 3-controller + N-broker topology;
• topic RF, minISR, and the producer’s acks form a consistent failure model;
• kafka_rack expresses only real failure domains, and replica placement has been verified;
• data-disk capacity, throughput, latency, retention, peak write rate, and recovery time have been load-tested;
• there is an explicit reassignment plan for after a new broker joins, since existing topics’ RF is not raised automatically;
• the procedures for Kafka data backup/rebuild, broker replacement, controller membership, and disaster recovery are all defined.

Security and Network

• a new production cluster uses kafka_security: scram from bootstrap onward;
• application passwords are injected by Vault/KMS/a secret manager and never reach Git or logs;
• only the CA public certificate is distributed to clients, never the CA private key;
• clients can resolve and reach the inventory_hostname of every broker directly;
• 9092/9093 are open only to the principals that need them, and 9308/9404 only to the monitoring network;
• a review checklist for application principals, topic/group/cluster ACLs, and quotas is in place;
• a protected rotation of internal credentials and certificates is scheduled.

Operations and Monitoring

• /usr/local/bin/pigsty-kafka-health cluster reports healthy;
• the dynamic quorum has exactly one leader, and every expected controller is in the current voters;
• there are no offline, under-replicated, or under-min-ISR partitions;
• produce/consume verification is done over the real application network with a real principal;
• the Kafka Overview, Kafka Instance, and Kafka Node dashboards show healthy data;
• alert routing, log search, capacity thresholds, on-call responsibilities, and rollback conditions are confirmed;
• upgrades, feature-level changes, topic deletion, user deletion, and cluster teardown each have a separate approval process.

For detailed alerts and PromQL, see Monitoring and Alerting; for metric semantics, see Metric Definitions.

Documentation Index and Next Steps

We recommend continuing along the following path:

One recommended reading path is: Quick Start → Cluster Configuration → Parameter Reference → Playbook → Day-to-Day Administration → Monitoring and Alerting → FAQ.

2 - Configuration

The KAFKA module expresses cluster intent through 15 persistent public parameters; everything else — topology, listeners, storage subdirectories, replication safety, authorization, and Exporter placement — is derived by the role in a single, consistent way. For a first deployment, start with the Quickstart; for the full field reference, see Parameters.

Pre-Deployment Checklist

Before filling in the inventory, confirm at least the following:

• The target hosts are already commissioned by NODE, the software repository is reachable, and inventory_hostname is directly routable by every Kafka member and client.
• A single operation will use -l to select precisely all members of one kafka_cluster — not a single node, a subset of members, or multiple clusters.
• kafka_seq is unique within the cluster, the controller count is odd, and the broker count matches your fault domains and capacity targets.
• Ports 9092, 9093, 9308, and 9404 do not collide, and Infra nodes can reach both metrics ports.
• kafka_data maps to a dedicated filesystem, sized for retention, write peaks, replication traffic, recovery time, and growth headroom.
• Production uses kafka_security: scram; node and admin clocks are synchronized, the Pigsty CA is available, and application passwords come from a secret source such as Vault.
• Topic partitions, replicas, min.insync.replicas, and retention policy — along with client acks, retries, and consumer recovery strategy — have been reviewed.
• Scale-out and scale-in, partition reassignment, upgrades, backup, restore, and controller membership changes each have their own runbook.

Roles and Topology

kafka_role accepts only three values:

kafka_role is declared independently per node, and defaults to combined when omitted. A cluster must contain at least one controller-capable node and at least one broker-capable node; an even number of controllers produces a warning, and production typically uses 3 controllers.

Single-Node Development Cluster

A single node serves as both broker and controller. It cannot tolerate a node failure and is suitable only for development, testing, and feature validation:

kf-dev:
  hosts:
    10.10.10.10: { kafka_seq: 1 }
  vars:
    kafka_cluster: kf-dev

The role derives RF=1 and minISR=1 from the initial broker count. Do not use a single-node topology or the default plaintext security profile directly in production.

Three-Node Combined Deployment

All three nodes serve as both broker and controller — a compact production starting point. Omit every role field to use the default combined:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1 }
    10.10.10.12: { kafka_seq: 2 }
    10.10.10.13: { kafka_seq: 3 }
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms4G -Xmx4G'
    kafka_security: scram
    kafka_parameters:
      num.partitions: 3
      num.network.threads: 6
      num.io.threads: 16
    kafka_topics:
      - name: order.events
        partitions: 12
        replication_factor: 3
        config:
          min.insync.replicas: 2
          cleanup.policy: delete

The initial three brokers automatically get the role-owned replication policy of RF=3 and minISR=2. You neither need nor are allowed to override the internal-topic RF, default.replication.factor, or min.insync.replicas in kafka_parameters. The 4G heap in the example is only illustrative; in production, balance the JVM heap, the operating-system page cache, and any other processes on the same host through load testing.

Separating Controllers and Brokers

Critical or larger clusters can separate the control plane from the data plane. Because explicit roles are present, every member must declare its role:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1, kafka_role: controller }
    10.10.10.12: { kafka_seq: 2, kafka_role: controller }
    10.10.10.13: { kafka_seq: 3, kafka_role: controller }
    10.10.10.21: { kafka_seq: 4, kafka_role: broker }
    10.10.10.22: { kafka_seq: 5, kafka_role: broker }
    10.10.10.23: { kafka_seq: 6, kafka_role: broker }
  vars:
    kafka_cluster: kf-main
    kafka_security: scram

A controller-only node does not listen on 9092 and does not run the protocol Exporter; it still exposes KRaft and JVM state through JMX. At most two kafka_exporter instances are placed on the broker-capable nodes with the lowest kafka_seq.

Dynamic KRaft and the Bootstrap Manifest

A new cluster uses the dynamic quorum directly: every node renders controller.quorum.bootstrap.servers, and no static controller.quorum.voters is generated. At the first format:

• The Cluster ID is generated randomly, not hashed from the cluster name;
• The Directory IDs of the initial controllers are generated randomly and frozen;
• Each node is formatted explicitly with either --initial-controllers or --no-initial-controllers mode;
• After startup, the role waits for the dynamic quorum to elect a leader, and verifies that every initial controller’s Directory ID is present in the live quorum.

The bootstrap-only facts are kept in the admin-side cache:

files/kafka/<kafka_cluster>/manifest.yml

At the same time, every cluster member keeps an authoritative copy at /etc/kafka/manifest.yml (a scram cluster also has /etc/kafka/secrets.yml). The manifest records only the cluster identity, the initial controller identities, the security profile, and the initial RF/minISR. The live cluster is always the authority on runtime facts:

• When the manifest conflicts with the live identity or security profile, ordinary playbooks fail closed;
• When an old manifest exists but all data disks are empty, reviving the old cluster is refused;
• When the admin-side cache is lost, it can be recovered automatically from any member’s node copy, without reformatting;
• When no manifest can be found on either the admin node or any member while storage is already formatted, the role fails closed and asks you to restore the manifest first;
• An already-formatted scram cluster likewise fails closed when neither the admin node nor any member holds the secret material.

Adding, replacing, or removing a controller is not an ordinary inventory operation. A new controller must be explicitly formatted against the existing cluster, started, and caught up, and then registered with Kafka add-controller; removal requires the corresponding remove-controller procedure. The role refuses to add an unregistered new controller to the voter set based on inventory alone.

Identity Parameters

An already-formatted node reads cluster.id and node.id from ${kafka_data}/metadata/meta.properties and cross-checks them against the manifest and the inventory; the initial controllers’ Directory IDs are compared against the live quorum after startup. An identity mismatch is a protective failure and must not be worked around by deleting meta.properties or wiping data.

Network and Listeners

The role exposes only ports — not bind addresses, advertised addresses, or the listener map:

The fixed listener conventions are as follows:

• The broker listener binds 0.0.0.0, and the controller listener binds inventory_hostname;
• The broker’s advertised.listeners uses inventory_hostname;
• The controller bootstrap address also uses inventory_hostname;
• plaintext: both BROKER and CONTROLLER use PLAINTEXT;
• scram: BROKER uses SASL_SSL + SCRAM-SHA-512, and CONTROLLER uses mutual TLS.

Clients must therefore be able to resolve and directly reach every broker’s inventory_hostname. The current v1 does not support NAT, public-address mapping, multiple client networks for the same broker, or arbitrary raw listener overrides; these scenarios cannot be assembled around through kafka_parameters.

Kafka’s standard access model is a smart client connecting directly to brokers: bootstrap.servers is configured with several seed addresses, and once the client fetches cluster metadata it connects directly to the partition leaders. HAProxy, a Keepalived VIP, or a cloud LB should not be used as the regular Kafka data-plane entry point, because they are unaware of Kafka metadata and partition leaders and cannot exempt clients from reaching every advertised.listeners address. A DNS or TCP LB can at most serve as an optional bootstrap discovery entry point; even then, the application network must still reach all brokers directly. See Quickstart: Connecting Application Clients for details.

The minimum required network flows:

The metrics ports are HTTP, and even when Kafka uses scram they should be restricted to the monitoring network by firewall.

Storage, Heap, and Rack

You set only the root directory:

kafka_data: /data/kafka

The role derives the topic data directory ${kafka_data}/data and the KRaft metadata directory ${kafka_data}/metadata in a fixed way. kafka_data must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg.

Production planning should account for at least retention, message peaks, replication traffic, partition/segment counts, disk latency and throughput, file descriptors, recovery time, JVM heap, and page cache. The current role generates only a single log.dirs; multi-disk JBOD, disk replacement, and automatic data migration require separate runbooks.

For deployments spanning fault domains, declare kafka_rack consistently on all broker-capable nodes:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }
10.10.10.22: { kafka_seq: 5, kafka_role: broker, kafka_rack: az-b }
10.10.10.23: { kafka_seq: 6, kafka_role: broker, kafka_rack: az-c }

Broker-capable nodes must either all set the rack or all omit it. Changing the rack triggers a safe rolling restart but does not automatically migrate existing replicas.

Replication Policy

At the first bootstrap, the policy is derived from the initial broker count:

replication_factor = min(3, broker_count)
min_insync_replicas = max(1, replication_factor - 1)

The initial default RF for future topics, the internal-topic RF, and the cluster minISR are all written into the manifest and frozen. After scaling out:

• default.replication.factor keeps its initial value; Kafka 4.3 does not allow changing it online through dynamic broker configuration;
• The RF of existing internal/business topics is not raised automatically;
• The role does not report “brokers have joined” as “data is balanced”;
• An RF change must use a reviewed kafka-reassign-partitions.sh plan; raising the static default additionally requires controller high availability or an explicit maintenance window, and takes effect through a full-cluster safe rolling restart.

Producer acks, idempotence, retries, batching, and compression are client policy, not parameters of the Kafka broker role.

kafka_parameters

kafka_parameters is the only broker-parameter escape hatch. It defaults to {} and is rendered only onto broker-capable nodes. It is suited to non-role-owned keys such as num.partitions, thread counts, buffers, retention, and segments.

The following patterns are owned by the role and must not be overridden:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

If any reserved key appears, the identity preflight fails outright before any file is written.

Security and Declarative Resources

kafka_security: scram is a complete production profile, not a set of switches to be combined at will. It automatically enables:

• Per-node certificates issued by the Pigsty CA;
• Mutual TLS on the controller listener;
• SASL_SSL + SCRAM-SHA-512 for broker/client and inter-broker traffic;
• StandardAuthorizer, deny-by-default, and role-owned admin/monitoring identities;
• Convergence of the minimal monitoring ACL before the protocol Exporter starts.

Application resources are declared through two domain objects:

kafka_security: scram
kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520
kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete

Resource convergence semantics: topic creation is idempotent, partitions only increase, and only explicitly declared config is updated; an RF change is refused with a prompt to run reassignment. A declared user’s password, ACLs, and the quota fields you provide converge idempotently. Removing a topic or user entry is not an implicit deletion procedure.

The security profile cannot be switched by an ordinary playbook after bootstrap. Internal credentials and certificates can use protected rotation, but an online migration from plaintext to scram still requires an explicit state machine that is planned for the future.

Packages and File Layout

The role installs java-runtime and kafka-stack through platform mappings. The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0; the actual versions still depend on the target platform’s repository and the installed packages.

These files are managed by the role. Persistent intent belongs in pigsty.yml. Do not edit generated files directly on the nodes, and do not copy passwords, private keys, or role-owned secret contents into inventory, logs, or tickets.

3 - Parameters

The KAFKA role deliberately exposes only 15 persistent parameters. Details such as topology, listeners, security implementation, storage subdirectories, replication safety, and Exporter placement are derived by the role in a single, consistent way, and cannot be overridden as additional persistent variables.

Parameter Overview

kafka_cluster and kafka_seq must be defined; kafka_role has a real default. The cluster’s roles are either all omitted or all declared explicitly.

Identity and Topology

kafka_cluster

The required cluster identity. It must start with a letter or digit and contain only letters, digits, underscores, and hyphens:

kafka_cluster: kf-main

It is used to discover the complete cluster membership, generate instance names, and locate the bootstrap manifest. Every kafka.yml lifecycle operation must use a precise -l to select all members of this cluster.

kafka_seq

A required non-negative integer, unique within the same kafka_cluster, which becomes the KRaft node.id directly:

10.10.10.11: { kafka_seq: 1 }

The instance name is derived as ${kafka_cluster}-${kafka_seq}. Once a node has been formatted, do not change or reuse a sequence number that still has associated data.

kafka_role

Defaults to combined, and accepts only:

When all cluster members omit it, they consistently use combined; as soon as any member sets it explicitly, all members must set it explicitly. No legacy role aliases are provided.

kafka_cluster_id

Unset by default, used only to assert the identity of an existing cluster during takeover or recovery. It must be a 22-character Kafka UUID:

kafka_cluster_id: MkU3OEVBNTcwNTJENDM2Qk

Do not set it for an ordinary new cluster. The role generates the Cluster ID randomly and writes it into files/kafka/<kafka_cluster>/manifest.yml. This parameter does not relabel existing data; it fails closed when it conflicts with the manifest or meta.properties.

kafka_rack

An optional broker fault-domain label, rendered as broker.rack:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }

All broker-capable nodes must either all declare it or all omit it. Controller-only nodes do not use this value. Changing the rack is a static change that goes through a strict rolling restart, but does not reassign existing replicas.

Storage, JVM, and Network

kafka_data

The data root directory, defaulting to /data/kafka:

kafka_data: /data/kafka

The role derives ${kafka_data}/data and ${kafka_data}/metadata in a fixed way. This path must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg. kafka-rm.yml deletes the entire root directory by default, so do not mix other services or business files into it.

kafka_heap_opts

The Kafka JVM heap, defaulting to:

kafka_heap_opts: '-Xms1G -Xmx1G'

In production, set it according to load and memory load testing. Typically keep Xms and Xmx equal, and leave enough memory for the operating-system page cache and other processes.

kafka_port

The broker/client listener port, defaulting to 9092, and listening only on broker-capable nodes. plaintext mode uses PLAINTEXT; scram mode uses SASL_SSL + SCRAM-SHA-512.

kafka_controller_port

The KRaft controller listener port, defaulting to 9093 (the conventional Kafka KRaft port), and listening only on controller-capable nodes. When sharing a node with other services, verify yourself that the ports do not collide; the role does not automatically detect cross-service port usage.

The four public ports must all differ from one another. The broker listener binds 0.0.0.0, while the controller listener, the broker advertised address, and the controller bootstrap address all use inventory_hostname in a fixed way, with no separate address parameters.

kafka_parameters

Defaults to {}. It is the only Kafka broker-parameter escape hatch, rendered only onto broker-capable nodes:

kafka_parameters:
  num.partitions: 12
  num.network.threads: 6
  num.io.threads: 16
  log.retention.hours: 168
  log.segment.bytes: 1073741824

The following keys or patterns are owned by the role and cannot be overridden through this mapping:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

Identity, listeners, security, storage, and replication policy must remain single-authority; when a reserved key is included, the preflight fails outright.

Observability

kafka_jmx_exporter_port

The JMX Exporter HTTP port, defaulting to 9404. The role injects the JMX Exporter Java agent unconditionally into every Kafka JVM and registers it as job=kafka; there is no separate toggle parameter. The lifecycle health gate uses the role-owned Kafka CLI/metadata channel and does not depend on JMX. Infra monitoring nodes must be able to reach this port; the endpoint does not automatically enable HTTPS because of kafka_security: scram, so it should be protected by the monitoring network and firewall.

kafka_exporter_port

The HTTP port of the protocol-level kafka_exporter, defaulting to 9308. The role configures, starts, and registers it only on the first two broker-capable nodes after sorting by kafka_seq; a single-broker cluster runs only one. The monitoring target file is refreshed on every full run according to the current placement, but a stale Exporter service on a node that was previously selected is not stopped automatically by an ordinary playbook.

The Kafka protocol version, TLS/SCRAM parameters, and replica placement used by the Exporter are all internal role conventions, with no additional public toggles or options parameters.

Security and Resources

kafka_security

Defaults to plaintext, and accepts only:

scram simultaneously configures the Pigsty CA-issued node certificates, the role-owned admin/monitoring/internal identities, and the ordering in which TLS/SCRAM and ACLs are enabled. The security profile is written into the bootstrap manifest; once the cluster is formatted, an ordinary rerun can neither switch plaintext to scram nor switch back.

kafka_users

Defaults to [], and may only be declared in scram mode. Each object accepts only name, password, acls, and quota:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
      - resource: transactional_id
        name: order.
        pattern: prefixed
        operations: [Write, Describe]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

Constraints:

• name is unique within the list; password is required, must be at least 12 characters, and should reference a secret management system;
• ACL resource is one of topic, group, transactional_id, cluster;
• pattern is literal (default) or prefixed;
• Operations are Read, Write, Create, Delete, Alter, Describe, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite;
• Quota keys are producer_byte_rate, consumer_byte_rate, request_percentage, controller_mutation_rate.

The role converges the SCRAM password, the complete ACL set, and the explicitly provided quota fields for a declared user. Removing a user entry does not implicitly delete the principal or credentials; deletion/revocation requires a separate, audited operation.

kafka_topics

Defaults to []. Each object accepts only name, partitions, replication_factor, and config:

kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 604800000

name is unique within the list; both partitions and RF must be at least 1, and RF cannot exceed the current broker count. The convergence semantics are:

• The topic is created idempotently when it does not exist;
• Partitions may only increase; decreasing fails;
• When RF differs from the live state, ordinary convergence is refused and an explicit reassignment is required;
• Only the keys declared in config are updated;
• Removing a topic from the list never deletes the topic.

Transient Protected Operational Variables

The following variables are used only via the command-line -e for one-off operational actions. They are not part of the 15 persistent API parameters and should not be written into pigsty.yml:

The two rotation actions are mutually exclusive, and must target a precise, complete cluster. kafka-rm.yml deletes the data directory and the Manifest/Secret/PKI cache on the admin node by default, so before running it you must explicitly confirm the target cluster and your backup/rebuild intent. For the commands and full semantics, see Playbooks.

4 - Administration

The KAFKA module installs Kafka under /opt/kafka, manages the service with Systemd, and keeps its persistent intent in pigsty.yml. The files generated on the nodes are not meant to be edited by hand.

All of the Kafka CLI examples below use the role-generated /etc/kafka/admin.properties. Even when the current profile is plaintext, keep --command-config on every command: that way the command structure stays the same when you switch the admin channel to scram. Replace <broker>:9092 with a reachable inventory_hostname and port.

Status Check

On any Kafka node, check the service and its recent logs:

systemctl status kafka
systemctl is-enabled kafka
journalctl -u kafka --since '-30 min' --no-pager

The protocol exporter runs on at most two broker-capable nodes, the ones with the lowest kafka_seq. On the selected nodes, also check:

systemctl status kafka_exporter
journalctl -u kafka_exporter --since '-30 min' --no-pager

Inspect the listeners and metrics endpoints:

ss -lntp | grep -E ':9092|:9093|:9308|:9404'
curl -fsS http://<kafka-ip>:9404/metrics | grep -E '^(jmx_scrape_error|kafka_server_raft_state|kafka_server_broker_messages_in_total)'
curl -fsS http://<exporter-ip>:9308/metrics | grep -E '^(kafka_brokers|kafka_topic_partitions)'

kafka_up and kafka_exporter_up are recording metrics computed on the VictoriaMetrics side; they may not appear on the raw endpoints. A healthy JMX endpoint should include jmx_scrape_error 0.0, JVM metrics, and the kafka_ metrics that match the node’s role.

Role-Owned Health Check

The role’s lifecycle gates do not rely on JMX. Instead, they use the same admin channel to check the dynamic quorum, offline partitions, under-replicated partitions, and under-min-ISR partitions:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties

The gate passes only when the returned JSON reports healthy: true. This is fine for read-only diagnostics, but it is not a substitute for end-to-end business validation.

KRaft Quorum Status

Query the dynamic quorum from any available broker:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

Focus on:

• LeaderId is present and points at the expected controller;
• CurrentVoters matches the live dynamic quorum;
• MaxFollowerLag and MaxFollowerLagTimeMs are not growing steadily;
• The dashboard shows exactly one active controller.

To confirm the dynamic quorum (KIP-853) feature level, run /opt/kafka/bin/kafka-features.sh ... describe and inspect kraft.version.

Inspect controller replication status:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --replication

If there is no leader, a member lags persistently, or the voter set differs from what you expect, stop all topology changes, preserve the logs, manifest, and meta.properties, and then analyze the network, node/directory identity, and any explicit controller membership operations. Do not force-rewrite the quorum by re-running an ordinary manifest.

Declarative Topics

Production topics should be declared in kafka_topics in pigsty.yml:

kafka_topics:
  - name: orders
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      retention.ms: 604800000

Converge against the full cluster target:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role creates topics idempotently, only adds partitions, and only changes the config keys you declared. A change in replication factor fails and demands an explicit partition reassignment; removing an entry from the manifest does not delete the topic.

Read-only topic inspection:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic orders

Temporary or externally managed topics can be created with the Kafka CLI, but they are not written back to pigsty.yml. Never let both declarative management and manual management own the same topic. Deleting a topic is a deletion of business data: it requires its own approval, exact-name confirmation, and a recovery plan, so this document does not provide a general delete command.

Declarative Users, ACLs, and Quotas

When kafka_security: scram, application identities should be managed through kafka_users:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: orders
        operations: [Read, Write, Describe]
      - resource: group
        name: order-worker
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

The full playbook idempotently converges the password, that user’s ACL set, and the quota fields you explicitly provide. Do not commit passwords in plaintext to the repository or emit them to logs. Removing a user entry does not automatically delete the principal or its credentials; deleting or fully revoking access requires a separate, audited process.

Message Read/Write Validation

Use a test topic for end-to-end validation. The console producer and consumer use the same client config file:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke

Consume in another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke \
  --from-beginning \
  --group ops-smoke-check

Production acceptance should run from the real client network and cover DNS/advertised.listeners, certificate validation, ACLs, producer ACKs, consumer commits, and end-to-end latency, not just the broker’s local loopback path.

Consumer Groups

List and describe consumer groups:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group order-worker

Judge lag against consumption rate and your business SLO: a brief backlog may just be batch behavior, whereas a backlog that keeps growing while the consumption rate stays below the production rate means the consumer cannot catch up. Resetting offsets can cause reprocessing or skipped messages, so it requires separate approval, exact group/topic confirmation, and a replay plan.

Persistent Configuration Changes

After editing pigsty.yml, always run against the full cluster:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role picks its path automatically, based on live health and the static fingerprint:

• Cluster unhealthy or stopped: start only the stopped controllers, and once the quorum recovers and catches up, start the brokers; if static changes also exist, the members that were already online then enter a strict rolling restart;
• Adding pure brokers to a healthy cluster: format, start, and confirm registration one node at a time;
• Static changes on a healthy cluster: a strict node-by-node rolling restart, applying — before and after each node restart — the controller zero-lag/recently-caught-up, quorum, offline-partition, under-min-ISR, and ISR-catch-up gates;
• No static changes: Kafka is not restarted.

Do not bypass the full state machine with -t kafka_config. Dynamic topic/user/ACL/quota convergence lives in the kafka_provision resource-convergence stage; whether a static change triggers a restart is the role’s decision.

Scaling and Topology Changes

Adding Pure Brokers

A healthy cluster can take on new kafka_role: broker nodes. Assign each new node a kafka_seq that has never collided, update the full inventory, and then still target the full cluster:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role admits each newly formatted pure broker one at a time and verifies that the broker is registered and not fenced. You cannot -l only the new nodes; nor can you join a new combined or controller node as if it were an ordinary broker.

Adding a broker does not migrate existing partitions. You must separately generate, review, and monitor a kafka-reassign-partitions.sh plan, control the disk/network load, and prepare a rollback. “The service is registered” is not the same as “scaling is complete.”

The replication policy does not scale up with the broker count, either. In particular, Kafka 4.3’s default.replication.factor cannot be changed dynamically: after scaling from 1 broker to 3, it remains the RF=1 set at initial build, and any future topic that does not specify RF explicitly is still created with RF=1. First complete the reassignment of existing partitions, then plan for controller high availability or a maintenance window, and finally let the new static default take effect through a safe full-cluster rolling restart. Do not bypass the downtime gates just to change a default.

Adding, Replacing, or Removing Controllers

The cluster is a dynamic quorum from the start, but controller membership is still an explicit Kafka administrative action:

1. Verify the current leader, voters, directory IDs, and majority;
2. Explicitly format the new controller against the existing cluster ID;
3. Start it and confirm it has caught up;
4. Run and verify add-controller;
5. To remove one, run the matching remove-controller, confirm the new majority, and only then decommission the node.

An ordinary inventory change does not perform these steps automatically; the role rejects any unformatted controller that was not registered in the initial manifest. Use Kafka’s official membership-change procedure and a rehearsed, standalone runbook for this operation.

Changing Addresses or Ports

The role always uses inventory_hostname as the broker’s advertised address and the controller’s bootstrap address. Changing an inventory address, kafka_port, or kafka_controller_port affects client metadata, broker communication, or the quorum, and counts as a high-risk static change; you must check DNS, certificate SANs, routing, firewalls, bootstrap addresses, monitoring targets, and all cluster members in lockstep.

Rotating Security Material

A formatted, healthy scram cluster supports two mutually exclusive protected actions: internal credential rotation and certificate rotation. Both require an exact full-cluster target and a matching kafka_rotate_confirm confirmation string, and it is recommended to run --check first. Certificates are re-issued by the same Pigsty CA, old and new certificates trust each other, and the rotation takes effect node by node through a strict rolling restart.

For the exact commands and failure semantics, see Playbook: Protected Rotation. The security profile itself is a bootstrap-only property; these actions do not imply support for online migration from plaintext to scram.

Data Protection and Recovery

Kafka’s data protection relies on replicas across failure domains, correct min-ISR, producer ACKs, and a rehearsed recovery procedure. The current role does not provide Kafka data backup, automatic broker drain, KRaft recovery orchestration, or cross-region disaster recovery.

When a disk or node fails:

1. First look at the Kafka Overview/Node dashboards, the quorum, ISR, offline partitions, and under-min-ISR partitions;
2. Preserve the evidence: journalctl -u kafka, node metrics, the manifest, server.properties, and meta.properties;
3. Confirm the node’s role, node.id, cluster ID, directory ID, and the availability of the remaining replicas;
4. Do not run kafka-rm.yml or delete meta.properties until you have a clear recovery/replacement plan;
5. Use a standalone runbook for reformatting, identity replacement, reassignment, or controller membership operations.

Log Diagnostics

journalctl -u kafka -f
journalctl -u kafka_exporter -f
journalctl SYSLOG_IDENTIFIER=kafka --since today
journalctl SYSLOG_IDENTIFIER=kafka_exporter --since today

VictoriaLogs/Grafana queries:

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter

The usual diagnostic order is: service logs → listening ports → admin-channel health → dynamic quorum → broker/partition/ISR → client addresses and certificates/ACLs → consumer lag. For the detailed panels and their alert mappings, see Monitoring & Alerting.

5 - Playbook

The KAFKA module ships two playbooks: kafka.yml deploys an Apache Kafka 4.x dynamic KRaft cluster and converges its security, resource, and monitoring state; kafka-rm.yml tears down a cluster or removes a member.

Basic Usage

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Check mode validates the public API, the full cluster, roles, racks, ports, the manifest, and any inspectable file changes, but it skips formatting, service startup, and live health acceptance. A successful --check is therefore not a guarantee of a successful runtime.

Execution Stages

kafka.yml is itself a thin wrapper: a single play runs the node_id and kafka roles in sequence, mirroring the structure of pgsql.yml. Inside the role, the lifecycle is split into six task stages; all cross-node ordering (parallel bootstrap, one-broker-at-a-time admission, strict node-by-node rolling) is handled centrally by the launch stage:

The play uses any_errors_fatal: true. When a stage fails, dangerous forward progress stops; once you fix the cause, you can re-run against the full cluster, and the role recovers from live state and the persistent fingerprint instead of blindly reformatting.

Lifecycle Paths

The config stage uses the role’s own admin channel to judge cluster health and select exactly one downstream path:

Cold Start, First Deployment, or Repair

When the cluster is stopped or the health predicate does not hold, it enters converge:

1. Start all controller-capable nodes;
2. Wait for the controller listener and a dynamic quorum leader;
3. Verify that the initial controller directory IDs are still in the live quorum;
4. Start the pure brokers;
5. Wait for the broker listener and require the full cluster to be healthy;
6. Persist the static fingerprint only after the config has been proven to run successfully.

JMX plays no part in the lifecycle gates: the decisions for startup, admission, and rolling are made entirely on the role’s own Kafka CLI/metadata admin channel.

Adding Pure Brokers to a Healthy Cluster

The role only auto-admits newly formatted kafka_role: broker nodes. It handles one new broker at a time and requires it to be registered and not fenced before continuing. Adding a controller or combined node cannot take this shortcut.

Admission only proves that the service has joined; existing partitions are not migrated onto the new broker automatically, so you must run an explicit reassignment separately.

Static Changes on a Healthy Cluster

When the rendered static fingerprint changes, the strict rolling restart handles one node at a time:

• Before restarting, it checks the controller majority, that all voters have zero lag and recently completed catch-up, offline partitions, under-replicated, under-min-ISR, and the effective ISR of each partition once the target is excluded;
• After restarting, it requires the target controller to be back as a voter and re-caught-up, the target broker to be registered and not fenced, and its replicas to re-enter the ISR;
• Any failed gate immediately stops the remaining nodes.

If a fault to repair and a static change coexist, converge only starts the stopped members and does not restart the still-online members in parallel; once the quorum recovers and catches up, the static changes that have not yet been loaded proceed into the strict rolling restart.

If the static fingerprint is unchanged, Kafka is not restarted. Dynamic resource changes still take effect online during the resource-convergence stage.

Task Tags

An ordinary configuration change should run the full kafka.yml and let the role choose its own lifecycle path. Stage tags are meant primarily for development, diagnostics, and controlled repair; you cannot bypass the full state machine with -t kafka_config or by limiting to a single node.

Identity, Formatting, and Manifest

Before writing any config, the role validates that:

• The limit contains exactly all members of one Kafka cluster;
• kafka_seq is unique, and the role sequences are either all omitted or all explicit;
• There is at least one controller and one broker;
• Racks are either all present or all absent across the broker-capable nodes;
• Ports are valid and non-conflicting, and the role-owned keys are not overridden by kafka_parameters;
• The manifest, security profile, meta.properties, and the live cluster identity are consistent.

A new cluster randomly generates the cluster ID and the initial controller directory IDs and formats each node in explicit dynamic mode. When ${kafka_data}/metadata/meta.properties already exists, it only validates the cluster ID, node ID, and directory ID; it does not reformat automatically.

The bootstrap manifest lives at:

files/kafka/<kafka_cluster>/manifest.yml

Each cluster member also keeps an authoritative copy at /etc/kafka/manifest.yml (a scram cluster additionally has /etc/kafka/secrets.yml). The live cluster is the authoritative runtime fact, but an ordinary playbook will not silently rewrite either side on conflict:

• When the admin-node cache is lost, it is restored automatically from any member’s node copy, without reformatting;
• When neither the admin node nor any member has a manifest but the storage is already formatted, it fails closed and asks you to restore the manifest first;
• When a manifest exists but all data disks are empty, it fails closed;
• When the cluster ID, security profile, or controller identity conflicts, it fails closed;
• A non-initial controller newly added in the inventory does not automatically become a voter.

Do not delete meta.properties, the manifest, or the secrets to bypass these protections.

Static Fingerprint and Recoverable Re-runs

The role computes an expected fingerprint over the static files that affect the Kafka process, and writes /etc/kafka/.pigsty-applied-static.sha256 only after one of the following holds:

• Converge has successfully started and passed the global health check;
• The strict rolling restart has restarted this node, let it catch up, and passed the post-restart gates.

If the run is interrupted, changes that have not been proven to take effect are not recorded as “applied.” The next full re-run can still recognize the pending static restart.

Resource Convergence and Monitoring Registration

Once the cluster is fully healthy, the resource-convergence and monitoring stages run in order:

1. Converge the role-owned dynamic cluster min-ISR;
2. Idempotently process the credentials, ACLs, and declared quotas of kafka_users;
3. Idempotently process the creation, partition growth, and explicit config of kafka_topics;
4. Check internal-topic RF drift, but do not reassign automatically;
5. Configure and start the protocol exporter on the first two broker-capable nodes, ordered by kafka_seq;
6. Refresh the file-discovery targets on all infra nodes.

Each instance maps to one target file, and both the JMX target and the (selected nodes’) protocol-exporter target live under the same kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

The target files are refreshed on every full run to match the current exporter placement; target deletion is handled by the deregistration step of kafka-rm.yml.

Protected Rotation

The rotation variables are one-shot extra-vars and should not be written into pigsty.yml. The two actions are mutually exclusive and only one may run at a time; the prerequisites are that all members are formatted, the cluster is healthy, the security profile is scram, the role-owned secret material exists, and kafka_rotate_confirm matches the cluster name exactly.

Internal Credential Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

The role uses active/standby internal identities: it first updates the inactive credential through the live admin channel, then atomically switches the local protected record, and enters the normal strict rolling restart. The old active identity is kept as the next round’s standby, so a re-run after an interruption is recoverable.

Certificate Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

The role discards the node certificates already issued in the admin-node cache, re-issues a private key and certificate for each node from the same Pigsty CA, updates the PEM certificate bundle on the nodes, and enters the strict rolling restart. Because the old and new certificates are issued by the same CA and trust each other, no staged trust swap is needed; if the health precheck fails, the rotation does not begin and the existing certificates on the nodes are left unchanged.

Cluster Teardown

Cluster removal is not in kafka.yml; it uses the separate kafka-rm.yml playbook:

./kafka-rm.yml -l kf-main                          # Remove the cluster: deregister monitoring, stop services, delete config; deletes data and manifest by default
./kafka-rm.yml -l kf-main -e kafka_rm_data=false   # Keep the on-disk data, remove only services and config
./kafka-rm.yml -l kf-main -e kafka_rm_pkg=true     # Also uninstall the kafka-stack packages (the shared Java runtime is not removed)

The execution order is: deregister the VictoriaMetrics targets (kafka_deregister) → stop and disable the kafka/kafka_exporter services (kafka) → delete the config, Systemd units, and helper scripts (kafka_config) → delete the data directories and the admin-node manifest/secrets/PKI cache (kafka_data, controlled by kafka_rm_data) → optionally uninstall the packages (kafka_pkg, controlled by kafka_rm_pkg).

The safeguard switch is kafka_safeguard: when set to true (on the command line or in the inventory), the playbook aborts immediately and deletes nothing.

kafka-rm.yml can also select a single member for scale-in (for example, -l 10.10.10.13). But removing a controller-capable member is a quorum membership change: you must first complete the explicit remove-controller procedure and confirm the new majority. An identity conflict, an exporter anomaly, or an ordinary startup failure is not a reason to delete data.

Playbook Boundaries

kafka.yml currently does not perform automatically:

• The controller add-controller / remove-controller membership procedures;
• Reassignment of existing partitions after a new broker joins, decommission drain, and data balancing;
• Changing the RF of an existing topic, deleting a topic, or deleting a user;
• Online migration between plaintext and scram for a formatted cluster;
• Kafka version upgrades, feature-level finalization, cross-version migration, and rollback;
• Kafka data backup, recovery orchestration, and disaster drills;
• Ecosystem components such as Connect, Schema Registry, MirrorMaker, and Cruise Control.

These operations require their own production runbooks. For day-to-day read-only checks and resource management, see Administration.

6 - Monitoring

Pigsty gives the KAFKA module a unified observability stack that combines metrics, logs, dashboards, and alerts. Monitoring covers both the Kafka JVM internals and the Kafka protocol view, so you never end up seeing only that the process is alive without visibility into partitions, ISR, and consumer lag, nor seeing only cluster metadata without visibility into the JVM, request queues, and KRaft controller health.

Scrape Architecture

The KAFKA module uses two complementary exporters:

On each Infra node, the role generates one file-discovery target per instance. The JMX target and the protocol exporter target (on the selected nodes) both live in the same file, under the same kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

A single-broker cluster runs only one protocol exporter; a multi-broker cluster runs at most two. Controller-only nodes register only the JMX target; brokers that were not selected and controller-only nodes have no protocol exporter target, which is expected behavior. The target file is refreshed to match the current placement on every full run; deletion of an instance target is handled by the deregistration step in kafka-rm.yml.

Label Model

Both target types are registered under the same job=kafka scrape job, and are distinguished by the presence or absence of the role label.

JMX Target

Protocol Exporter Target

A protocol exporter target carries only cls, ins, ip, and instance (10.10.10.11:9308); it has no role or node_id labels. The recording rules on the vmagent side use this to distinguish the two availability types: kafka_up is up{job="kafka",role=~".+"}, and kafka_exporter_up is up{job="kafka",role=""}.

The exporter queries the entire Kafka cluster through a broker, so the two exporters of the same cluster may return an identical view of topics, partitions, and consumer groups. The cluster-level recording rules first deduplicate across exporter instances, then aggregate the logical cluster rates. In scram mode, the TLS/SCRAM parameters the exporter needs to connect to Kafka are generated automatically from the role’s own monitoring identity.

Grafana Dashboards

Pigsty ships three complementary dashboards:

Kafka Overview

Cluster and global overview. cls=All is the overview across all Kafka clusters; once you select a specific cls, the same dashboard becomes the overview for that Kafka cluster, rather than a separate set of panels.

Main content:

• Inventory of clusters, brokers, topics, partitions, and consumer groups
• Broker availability, exporter health, and cluster workload
• Leaderless, under-replicated, ISR deficit, and non-preferred replica
• Topic offset progress, consumer commit progress, and total lag
• Consumer group members, lag ranking, and topic/group drill-down
• Kafka/exporter log volume, firing alerts, and log detail

Common variables: cls, members, topic, group, topk.

Kafka Instance

Use the ins variable to select a broker instance running kafka_exporter, and view cluster metadata and consumer state from the protocol perspective, correlated with host resources.

Main content:

• Exporter availability, identity, runtime, and scrape cost
• Broker directory, topic inventory, partition topology
• Leader distribution, ISR deficit, leaderless, and non-preferred
• Topic offset span, append/commit progress
• Consumer group inventory, members, offsets, and lag
• Node CPU/memory, disk I/O, network, filesystem, and logs

Common variables: ins, cls, ip, topic, consumergroup, topk.

Kafka Node

Use the ins variable to select any Kafka JVM, including controller-only nodes, and view JMX, broker, and KRaft internal state.

Main content:

• JVM heap, GC, threads, buffer pool, CPU, FD, and uptime
• Broker message/network/replication throughput and ISR churn
• Request rate, errors, P95/P99 latency, queues, and handler/network idle
• Under-replicated, under min ISR, offline replica/log directory
• KRaft member state, metadata log, propagation lag, and snapshot
• Active controller, fenced broker, offline partition, and event latency
• JMX scrape quality, host pressure, and Kafka logs

Common variables: cls, ins, ip.

Choosing a Dashboard

Recording Rules

The Kafka rule file lives at /infra/rules/kafka.yml. The main recorded metrics are:

Rates derived from offset changes represent progress, not client request counts. Log truncation, offset rollback, or an exporter restart can produce a transient negative change; the rules use clamp_min(..., 0) to keep only forward progress.

Alert Rules

An unclean leader election can mean data loss. Immediately preserve the controller/broker logs, confirm the affected topics and replicas, and only then decide on a recovery action.

Common PromQL

Check scrape targets:

kafka_up
kafka_exporter_up
up{job="kafka"}

Check the replication health of a cluster:

sum by (cls) (kafka_server_replica_manager_under_replicated_partitions{job="kafka"})
sum by (cls) (kafka_server_replica_manager_under_min_isr_partitions{job="kafka"})
max by (cls) (kafka_controller_offline_partition_count{job="kafka"})

Check consumer lag:

topk(20, kafka_consumergroup_lag_sum{cls="kf-main"})

Check request saturation and latency:

kafka_server_request_handler_idle_ratio{job="kafka",cls="kf-main"}
max by (ins,request,quantile) (
  kafka_network_request_total_time_seconds{job="kafka",cls="kf-main",quantile=~"0.95|0.99"}
)

Log Queries

Kafka services write stdout and stderr to Journald; the node’s Vector Journald source forwards them to VictoriaLogs, all under job:syslog.

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter
ip:10.10.10.11 job:syslog (unit:kafka OR app:kafka)

The log panel on the Kafka Node dashboard uses similar queries and shows time, level, systemd unit, and message. When diagnosing, align the logs with the KRaft, ISR, request queue, GC, disk I/O, and network metrics from the same time window.

Verifying the Monitoring Chain

Verify the raw endpoints on a Kafka node:

curl -fsS http://<kafka-ip>:9404/metrics | grep '^jmx_scrape_error'
curl -fsS http://127.0.0.1:9308/metrics | grep '^kafka_brokers'

Check file discovery on an Infra node (one file per instance; the file for a selected node contains both the JMX and the protocol exporter targets):

ls -l /infra/targets/kafka/
cat /infra/targets/kafka/kf-main-1.yml

Then query up{job="kafka"} in VictoriaMetrics (or the recorded metrics kafka_up and kafka_exporter_up). After a failed scrape, custom exporter metrics may briefly retain stale samples, so endpoint liveness should be judged by Prometheus’s native up. If the raw endpoints are fine but the recorded metrics are missing, check file discovery, the VictoriaMetrics target, network reachability, rule loading, and labels, in that order. If the JMX HTTP endpoint is fine but jmx_scrape_error is 1, check the Kafka logs and the MBean matching in /etc/kafka/jmx_exporter.yml.

For complete metric semantics, see Metric Definitions.

7 - Metrics

The KAFKA module uses two kinds of metric sources, both registered under the same job=kafka scrape job: the JMX target (with a role label) collects the internal state of each JVM, while the protocol exporter target (without a role label) collects the state of the logical cluster, topics, partitions, and consumer groups over the Kafka protocol. The protocol exporter is placed only on the one or two broker-capable nodes with the smallest kafka_seq, and a single-broker cluster runs just one.

The JMX configuration is an allow-list: it exports only the JVM baseline and a bounded set of broker, replication, request-path, and KRaft metrics. High-cardinality per-client and per-partition JMX MBeans are deliberately excluded; partition detail is supplied by the protocol exporter.

Common Labels

The job for both target types is kafka; whether a series carries the role label is what distinguishes the two.

Some metrics also carry dimensions such as topic, partition, broker, consumergroup, request, version, error, quantile, state, or operation.

Availability and Scrape Metrics

Protocol Exporter Metrics

The following metrics come from the protocol exporter target. Multiple exporters of the same cluster see the same logical cluster state, so any direct cluster aggregation must deduplicate by semantics rather than simply summing across all ins.

Broker and Topic

current_offset - oldest_offset estimates the currently retained offset span, but an offset count is not a byte count, and for a compacted topic it is not an exact message count either.

Consumer Group

Ephemeral consumers that never commit an offset, clients that use external offset storage, and groups that have not yet consumed a topic will not necessarily produce these time series.

Exporter Itself

JMX: JVM Baseline

excludeJvmMetrics: false makes the JMX Exporter expose the standard JVM/process metrics. The Kafka Node dashboard mainly uses:

JMX: Broker Traffic

These are broker-wide totals with no topic dimension, which keeps the JMX series count from growing with the number of topics. Topic-level offsets and progress come from the protocol exporter.

JMX: Replication and Storage

Under Replicated means the replicas are not all in sync. Under Min ISR is more serious: the write-availability or durability condition has fallen below the configured minimum ISR. At Min ISR has not crossed the line yet, but there is no remaining replica headroom.

JMX: Request Path

When investigating high latency, look at request volume, error codes, P95/P99, both queues, handler/processor idle, GC, CPU, disk I/O, and network together. A low idle ratio alone is not enough to pinpoint where the bottleneck is.

JMX: KRaft and Broker Metadata

log_end_offset - high_watermark helps gauge metadata commit lag; also factor in the member role, current leader, epoch, and controller event latency.

JMX: Controller

These MBeans exist only in Kafka processes that carry the controller role:

A healthy cluster should have exactly one active controller. Any increase in offline_partition_count, metadata_errors_total, or unclean_leader_elections_total should be treated as a priority.

Recording Rule Metrics

Offset Progress

JVM and Broker

Cardinality and Interpretation Notes

• Do not directly sum multiple kafka_exporter results for the same cls; they may be duplicate views of the same cluster.
• kafka_topic_partition_current_offset is an offset, not an exact count of bytes, requests, or business events.
• Consumer lag covers only groups that are visible in Kafka and have committed an offset.
• A controller-only node lacking broker metrics and protocol exporter metrics is a normal role difference; a broker that was not selected lacking protocol exporter metrics is a normal result of placement.
• When a given MBean does not exist in a specific Kafka version/role, the corresponding JMX series will not appear either; interpret this together with role.
• Per-client/per-partition JMX metrics are excluded by the allow-list to avoid unpredictable time-series cardinality.

For how to use the dashboards and alerts, see Monitoring.

8 - FAQ

How mature is the current KAFKA module?

The current role implements a production-grade v1 baseline: dynamic KRaft, full cluster guardrails, cold-start/repair, broker-only serial admission, strict rolling restart, TLS/SCRAM/ACL, declarative convergence of topics/users, internal credential and certificate rotation, and the full monitoring pipeline.

It is not a managed Kafka product. Production still requires kafka_security: scram, an odd number of controllers, sufficient brokers/RF/minISR, plus your own capacity planning, reassignment, controller membership, upgrade, backup, restore, and failure drills. The default plaintext is only suitable for development or a trusted, isolated network.

Why is there no ZooKeeper and no controller.quorum.voters?

This module targets Kafka 4.x and uses native dynamic KRaft, with no ZooKeeper installed and no static quorum created. All members render controller.quorum.bootstrap.servers; new clusters are formatted explicitly with --initial-controllers/--no-initial-controllers, and after startup the role verifies that the directory IDs of the initial controllers have joined the live quorum.

The initial controller identity is written into the bootstrap manifest. Later controller additions and removals still require the explicit add-controller/remove-controller management procedures; editing the inventory alone is not enough.

What is the difference between combined, broker, and controller?

• combined: acts as both broker and controller, listening on 9092 and 9093; this is the default;
• broker: pure data plane, listening only on 9092;
• controller: pure control plane, listening only on 9093.

The cluster roles must either be omitted entirely and consistently use combined, or be declared explicitly for every member. The old role aliases are no longer provided.

Will controller port 9093 collide with Alertmanager?

The controller uses Kafka KRaft’s conventional port 9093 by default. Alertmanager on a Pigsty Infra node also uses 9093 by default: the two collide only when Kafka and Infra share the same node. In that case, adjust kafka_controller_port for that cluster; the role only enforces that the four Kafka ports 9092, 9093, 9308, and 9404 differ from one another, and does not automatically detect port conflicts with other services.

The service is up, but a remote client cannot connect?

A broker’s advertised.listeners always uses inventory_hostname. After connecting to the bootstrap server, a client must also resolve and reach every broker address returned in the metadata.

Check in order:

grep '^advertised.listeners' /etc/kafka/server.properties
ss -lntp | grep ':9092'
getent hosts <inventory-hostname>

A scram client must additionally check the CA, SASL mechanism, username/password, and ACLs. The current v1 does not offer custom advertised addresses, multiple listeners, or NAT/public mapping; if a client cannot route directly to inventory_hostname, that network model is outside the current core contract and cannot be worked around by overriding the raw listener via kafka_parameters.

Why does it report a Cluster ID, Node ID, or Directory ID mismatch?

The role cross-checks the bootstrap manifest, ${kafka_data}/metadata/meta.properties, the inventory, and the live dynamic quorum. Common causes include:

• kafka_cluster or kafka_seq was changed;
• another cluster’s data disk was mounted on the current node;
• an incorrect kafka_cluster_id was given during restore/takeover;
• the controller data directory or directory ID does not match the live voter records;
• the wrong target cluster was selected, or a stale manifest was used.

This is a protective failure. Do not delete meta.properties, the manifest, or run kafka-rm.yml. First confirm data ownership, the remaining replicas, the true Cluster/Node/Directory identity, and the recovery target.

What happens if the manifest is lost or only an old one remains?

Every cluster member keeps an authoritative copy of the manifest at /etc/kafka/manifest.yml (a scram cluster also has /etc/kafka/secrets.yml). When the files/kafka/<cluster>/ cache on the admin node is lost, the role automatically restores it from any member’s node copy and does not reformat. Only when both the admin node and all member copies are lost, while the storage has already been formatted, does the role fail closed and prompt you to restore the manifest first; a formatted scram cluster likewise fails closed when the secret material cannot be found on either side.

Conversely, if the manifest exists but all Kafka data disks are empty, the role fails closed to avoid accidentally reviving a vanished cluster under an old identity. If you genuinely intend to rebuild, you must first run kafka-rm.yml and follow an explicit rebuild procedure.

Why are some keys in kafka_parameters rejected?

Identity, the dynamic quorum, listeners, storage, replication, rack, and security must be managed uniformly by the role. The reserved patterns include:

process.roles, node.id, controller.quorum.*,
listeners, advertised.listeners, listener.security.protocol.map,
inter.broker.listener.name, controller.listener.names,
log.dirs, metadata.log.dir, broker.rack,
min.insync.replicas, default.replication.factor,
offsets.topic.replication.factor,
transaction.state.log.*, share.coordinator.state.topic.*,
authorizer.class.name, super.users, allow.everyone.if.no.acl.found,
sasl.*, ssl.*, listener.*

Use the corresponding 15 public parameters; there are no variables for advertised addresses, path subdirectories, the listener map, or exporter options.

How do I enable TLS, SCRAM, and ACLs?

Set on a new cluster:

kafka_security: scram

This enables the Pigsty CA node certificates, controller mTLS, broker/client SASL_SSL + SCRAM-SHA-512, StandardAuthorizer, and default-deny all at once. Application users declare their passwords, ACLs, and optional quotas via kafka_users.

The security mode is a bootstrap-only property. A formatted cluster cannot switch online from plaintext to scram via ordinary playbooks; that requires a separate migration state machine. A healthy scram cluster can rotate internal credentials or certificates through protected actions.

Do kafka_topics and kafka_users delete resources?

No. Removing an entry from the inventory never implicitly deletes a topic or a user.

Topics are created idempotently, partitions only increase, and only the declared configs are updated; an RF change requires an explicit reassignment. A declared user has its password, complete ACL set, and the given quota fields converged. Topic deletion, user deletion, and full privilege revocation are all separate, audited operations.

What is the difference between the JMX Exporter and kafka_exporter?

The JMX Exporter is injected into every Kafka JVM and collects JVM, broker, replication, request-path, and KRaft internal metrics, registered as a job=kafka target with a role label.

kafka_exporter queries the logical cluster, topics, partitions, offsets, consumer groups, and lag over the Kafka protocol, registered as a target under the same job=kafka but without a role label. The role runs it only on the first two broker-capable nodes ordered by kafka_seq; a single-broker cluster runs one, and pure controllers run none.

The two are complementary. The lifecycle health gate uses the role’s own Kafka CLI/metadata channel and does not depend on either exporter.

Why does a particular broker or pure controller have no kafka_exporter?

This is the expected derived placement. The protocol exporter returns a view of the entire logical cluster, not node metrics; capping it at two replicas avoids a monitoring single point of failure while keeping the cost of duplicate scraping in check.

Check the current targets (one file per instance; the files of selected nodes contain the :9308 protocol exporter target):

ls -l /infra/targets/kafka/
grep 9308 /infra/targets/kafka/*.yml

A full run refreshes each instance’s target file according to the current placement, so you should not run against a single node just to register labels. Note: if the exporter placement moves due to a topology change, the old kafka_exporter service on a formerly selected node is not stopped automatically by ordinary playbooks and must be cleaned up manually or via kafka-rm.yml.

Why is the JMX endpoint reachable but jmx_scrape_error=1?

HTTP reachability only means the Java agent is loaded; jmx_scrape_error=1 means the MBean scrape failed this round:

journalctl -u kafka --since '-30 min' --no-pager
curl -fsS http://<kafka-ip>:9404/metrics | head -n 40

Check whether /etc/kafka/jmx_exporter.yml matches the currently installed Kafka/JMX Exporter packages, and whether the JVM has passed startDelaySeconds. Real startup acceptance requires jmx_scrape_error 0.0, JVM metrics, and at least one kafka_ metric matching the role.

Why is there no Consumer Lag data?

Common causes: the consumer does not use a group, does not commit offsets to Kafka, stores offsets in an external system, the group has not yet consumed the target topic, or the protocol exporter has a TLS/SCRAM/ACL/network problem.

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group <group>

Then check kafka_exporter_up, the exporter logs, the dashboard variables, and the raw kafka_consumergroup_* metrics. Endpoint liveness is judged by Prometheus’s native up; do not substitute a custom metric that may briefly linger after a scrape failure.

Why can’t the cluster metrics from two kafka_exporters be added together?

Both exporters query the same logical cluster and may return the same topic/partition/consumer-group state; summing them directly double-counts. Pigsty’s kafka:cls:* recording rules first deduplicate across the exporter replicas, then aggregate to the cluster.

Should applications go through HAProxy, a Keepalived VIP, or an LB?

Usually not. Kafka producers/consumers are cluster-aware smart clients: they first connect to any available seed in bootstrap.servers to fetch metadata, then connect directly to each partition leader. A production configuration should provide at least two, and usually three, broker seed addresses, and should let the application reach every address a broker advertises directly.

A VIP or a generic TCP LB understands neither partition leaders nor rewrites the broker addresses in Kafka metadata; placing one in the data plane usually only adds long-connection state, an extra point of failure, and troubleshooting complexity. If a corporate platform mandates a single discovery entry point, DNS or a TCP LB may serve bootstrap only, but advertised.listeners must still return a client-reachable address for each broker, and the LB must not become the only network path.

Exposure across NAT, the public internet, multiple networks, or Kubernetes usually requires a separate external address per broker and additional listeners. The current module always advertises the inventory address and does not support such mapping. See Quickstart: why applications should connect directly to multiple brokers and Cluster Config: network and listeners.

Can I just add a broker?

A healthy cluster supports adding a pure broker. After updating the full inventory, you still need the exact full-cluster limit:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role formats, starts, and verifies the registration of the new broker one at a time. You cannot limit the run to the new node only. After it joins, existing partitions are not migrated automatically; you must run and monitor reassignment separately. “The broker is registered” does not mean “capacity is already balanced.”

Can I just add or remove a controller?

Not by inventory alone. Although the cluster uses a dynamic quorum, a new controller must be explicitly formatted, started, and caught up against the existing Cluster ID, and then have add-controller applied; removal requires remove-controller, majority verification, and a node decommissioning procedure.

The role refuses to automatically treat a new controller in the inventory as a voter. Use Kafka 4.3 KRaft membership changes together with a separate, rehearsed runbook.

Which parameter controls the package version?

The role uses package_map['java-runtime'] and package_map['kafka-stack']; there is no kafka_version, scala_version, or exporter version parameter. The actual versions are determined by the Pigsty repository for the target platform and the installed packages.

The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0. An upgrade still requires a separate review of compatibility, backup/rollback, rolling order, and feature level; you cannot simply swap the packages.

How do I safely wipe Kafka data?

kafka.yml never performs cleanup; cluster teardown uses the separate kafka-rm.yml playbook. By default (kafka_rm_data=true) it permanently deletes the data/KRaft metadata, the node security state, the monitoring targets, and the manifest/secret/PKI cache on the admin node; setting kafka_safeguard=true forcibly aborts any deletion.

The playbook has no extra gate such as a confirmation string, so before running it you must manually confirm the exact -l target, a recoverable backup or a clear intent to rebuild, and the business-decommissioned status. For the full semantics, see Playbook: cluster teardown.