Module: KAFKA

• 1: Quick Start
• 2: Configuration
• 3: Parameters
• 4: Administration
• 5: Playbook
• 6: Monitoring
• 7: Metrics
• 8: FAQ

flowchart LR
    admin["Pigsty admin node"] -->|"kafka.yml / exact cluster"| kafka["Kafka 4.1+ / dynamic KRaft"]
    kafka --> jmx["Each Kafka JVM / JMX :9404"]
    kafka --> exporter["Up to two brokers / kafka_exporter :9308"]
    kafka --> journal["Journald"]
    jmx --> vm["VictoriaMetrics"]
    exporter --> vm
    journal --> vector["Vector"] --> vl["VictoriaLogs"]
    vm --> grafana["Grafana"]
    vl --> grafana
    vm --> alert["Alertmanager"]

    style kafka fill:#70C1B3,stroke:#4f968b,color:#fff
    style vm fill:#E66B7A,stroke:#b84e5c,color:#fff
    style vl fill:#C98367,stroke:#9e634e,color:#fff
    style grafana fill:#F29C64,stroke:#c77845,color:#fff

1 - Quick Start

First build a single-node cluster, create a topic, and read and write messages. Then build a separate secure three-node cluster with users, ACLs, quotas, and production topics. Finally, change parameters, connect clients, verify monitoring, and perform pre-launch checks.

Learning Path

Before You Begin

Unless stated otherwise, run commands from the project directory on the Pigsty administration node:

cd ~/pigsty

Before you start, confirm that:

• Confirm pigsty.yml describes this environment. Back it up and review existing contents.
• Ensure all members and clients can resolve and route directly to each inventory_hostname.
• Synchronize administration and Kafka node clocks.
• Check 9092, 9093, 9308, and 9404 for conflicts.
• Use /data/kafka exclusively for Kafka data.
• Select exactly all members of one cluster with -l on every kafka.yml run.
• Before changes, run --check, review results, and obtain change approval.

Retain the all.children inventory structure. Merge these groups into pigsty.yml. Do not overwrite existing all.vars, infra, etcd, pgsql, or other configuration.

Part 1: Deploy a Single-Node Kafka

1. Define the Cluster

Add kf-dev under all.children. Omitting kafka_role selects combined, so this node runs broker and controller:

all:
  children:
    # keep your existing infra, etcd, pgsql, and other groups

    kf-dev:
      hosts:
        10.10.10.10: { kafka_seq: 1 }
      vars:
        kafka_cluster: kf-dev
        kafka_data: /data/kafka
        kafka_security: plaintext
        kafka_topics:
          - name: quickstart.events
            partitions: 1
            replication_factor: 1
            config:
              retention.ms: 86400000   # 1 day, tutorial only

This configuration yields:

• a random cluster ID;
• a single dynamic KRaft combined node;
• default RF=1 and minISR=1;
• a single-partition topic named quickstart.events;
• a JMX exporter on :9404 and a protocol exporter on :9308.

plaintext provides no encryption, authentication, or ACLs. Use it only for local development or trusted isolated networks.

2. Bring the Node Under Management

If this host has not yet been initialized by NODE, run check mode first:

./node.yml --check -l kf-dev

After reviewing the results and getting approval, bring the node under management:

./node.yml -l kf-dev

Skip this step if Pigsty already manages the node and its repository access and time synchronization work. See Node Administration for preparation and operations.

3. Deploy Kafka

Run a check against the full cluster first:

./kafka.yml --check -l kf-dev

Verify that the target includes exactly every kf-dev member. Review data paths, packages, ports, and configuration changes, then run:

./kafka.yml -l kf-dev

The role installs Java and kafka-stack, generates random identity and manifest, formats KRaft storage, starts Kafka, creates topics, and registers monitoring.

4. Verify the Service and Quorum

Log in to the Kafka node and check the services:

systemctl is-active kafka kafka_exporter
journalctl -u kafka --since '-10 min' --no-pager

Run the role’s own health check:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties

JSON should include "healthy": true. Then check quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

Expect a valid LeaderId, this node in CurrentVoters, and quickstart.events with RF=1 and ISR=1.

5. Produce and Consume Messages

Start a console producer:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events

Enter several messages and press Ctrl-D. Read them from another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

Single-node deployment, topic convergence, and message read/write are complete. See Day-to-Day Administration for more checks.

Part 2: Deploy a Three-Node Production Baseline

The new kf-main cluster has three combined nodes and tolerates one controller failure. Business topics use RF=3/minISR=2. Security uses scram.

1. Define the Secured Cluster and Its Resources

Add the following group to your existing all.children:

all:
  children:
    # keep your existing groups

    kf-main:
      hosts:
        10.10.10.11: { kafka_seq: 1 }
        10.10.10.12: { kafka_seq: 2 }
        10.10.10.13: { kafka_seq: 3 }
      vars:
        kafka_cluster: kf-main
        kafka_data: /data/kafka
        kafka_heap_opts: '-Xms4G -Xmx4G'
        kafka_security: scram

        kafka_parameters:
          num.partitions: 12
          num.network.threads: 6
          num.io.threads: 16
          log.retention.hours: 168
          log.segment.bytes: 1073741824

        kafka_users:
          - name: quickstart-app
            password: "{{ vault_kafka_quickstart_password }}"
            acls:
              - resource: topic
                name: quickstart.
                pattern: prefixed
                operations: [Read, Write, Describe]
              - resource: group
                name: quickstart.
                pattern: prefixed
                operations: [Read]
              - resource: cluster
                name: kafka-cluster
                operations: [Describe, IdempotentWrite]
            quota:
              producer_byte_rate: 10485760
              consumer_byte_rate: 20971520

        kafka_topics:
          - name: quickstart.events
            partitions: 12
            replication_factor: 3
            config:
              min.insync.replicas: 2
              cleanup.policy: delete
              retention.ms: 604800000

Supply vault_kafka_quickstart_password through Ansible Vault, KMS, or another secret injection method. It must contain at least 12 characters. Never put real passwords in Git, logs, or tickets.

This configuration has these effects:

• All nodes omit kafka_role and use combined.
• Bootstrap uses dynamic KRaft directly.
• scram enables node TLS, controller mTLS, SCRAM-SHA-512, ACLs, and default denial together.
• Three brokers set initial RF=3 and minISR=2.
• quickstart.events explicitly has 12 partitions and three replicas.
• quickstart-app can read/write quickstart.* topics, read quickstart.* groups, and use idempotent production.
• At most two brokers run kafka_exporter. All three JVMs run JMX Exporter.

If brokers occupy separate failure domains, set kafka_rack: az-a/az-b/az-c respectively on all nodes. Fictitious labels cannot provide disaster protection. See Rack configuration.

2. Bring Under Management and Deploy

If the nodes are not yet under management:

./node.yml --check -l kf-main
./node.yml -l kf-main

When deploying Kafka, you must select all three members:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Do not select only -l 10.10.10.11. Partial clusters are rejected. You can select several complete clusters with -l kf-dev,kf-main, or all clusters without a limit.

3. Verify the Health of All Three Nodes

From the administration node, check all three Kafka services:

ansible kf-main -b -m command -a 'systemctl is-active kafka'

Run the full health check on any broker:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties

Query the quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

Before launch, verify one active controller, three voters, and three available brokers. Every quickstart.events partition must have three replicas and ISR=3. No partitions should be offline, under-replicated, or under-min-ISR.

Part 3: Connect Application Clients

1. Distribute the CA Public Certificate

Securely copy the public CA certificate from the admin node to the application host:

files/pki/ca/ca.crt  ->  /etc/kafka-client/pigsty-ca.crt

ca.crt is public and safe to distribute. Never copy, expose, or distribute files/pki/ca/ca.key. On application hosts, make the CA file root-owned and read-only. Pigsty-managed hosts already have it at /etc/pki/ca.crt; clients can use that path directly.

2. Create the Client Configuration

On the application host, create /etc/kafka-client/client.properties:

bootstrap.servers=10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="quickstart-app" password="<secret-from-vault>";

ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka-client/pigsty-ca.crt
ssl.endpoint.identification.algorithm=https

Kafka Java clients support SASL_SSL, SCRAM, and PEM truststores. Applications should inject passwords from secret management at runtime. Do not commit password files to repositories. See Kafka 4.3 SASL/SCRAM and producer settings for all fields.

3. Why Applications Should Connect Directly to Multiple Brokers

Clients use bootstrap.servers only for initial metadata. They then connect directly to partition leaders and refresh routing after leadership changes. In production:

• Set at least two, usually three, bootstrap.servers addresses in different failure domains.
• Permit application access to 9092 on all brokers. Ensure each advertised inventory_hostname resolves and routes correctly.
• Use client retry, metadata refresh, idempotence, and consumer-group protocols.
• Do not put HAProxy, Keepalived VIP, L4 load balancers, or L7 reverse proxies in the Kafka data plane.

A VIP or load balancer cannot replace metadata broker addresses or transparently select partition leaders. It adds connection-state, diagnostic, and capacity-planning complexity. A DNS name or TCP load balancer can provide bootstrap discovery if required. However, every advertised.listeners address must remain directly reachable, not only the load balancer. NAT, public internet, Kubernetes, and multiple networks need per-broker external addresses and additional listeners. The module fixes advertised addresses to inventory addresses and does not support these mappings.

4. Authenticate and Read/Write with the Application Identity

On an application host with the Kafka 4.3 CLI installed, run:

kafka-console-producer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events

When consuming, use a group prefix permitted by the ACL:

kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

Explicitly review these production client settings:

Set acks, retries, idempotence, batching, compression, and offset strategy in applications. Do not put them in broker kafka_parameters.

Part 4: Change Core Parameters

Declare persistent changes in pigsty.yml. Never edit /etc/kafka/server.properties directly. Use these parameters:

Example: Adjust the Heap and Broker Default Parameters

This example raises heap to 6G, increases threads, and sets new-topic default retention to 72 hours after load testing:

kf-main:
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms6G -Xmx6G'
    kafka_parameters:
      num.partitions: 12
      num.network.threads: 8
      num.io.threads: 24
      log.retention.hours: 72
      log.segment.bytes: 1073741824

Do not copy 6G/8/24 without testing. Choose values from CPU, memory, connections, message size, partitions, disk, and page-cache measurements.

Example: Add Partitions and Shorten Topic Retention

Increase quickstart.events from 12 partitions to 24 and change the retention to three days:

kafka_topics:
  - name: quickstart.events
    partitions: 24
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 259200000

Partitions cannot decrease. If replication_factor differs from live state, ordinary convergence fails and requires explicit reassignment. The role never relocates replicas automatically.

Apply the Change

Whether you change static parameters or dynamic resources, run the full state machine:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Do not run only -t kafka_config. The role rolls nodes for static changes. Topic or user changes alone do not restart Kafka.

The following keys belong to the role itself and must not be placed in kafka_parameters:

kafka_parameters:
  min.insync.replicas: 2          # wrong: owned by the role
  default.replication.factor: 3   # wrong: owned by the role
  listeners: ...                  # wrong: owned by the role

For all 15 public parameters, their defaults, and the reserved keys, see the Parameter Reference.

Part 5: Key Pre-Launch Checks

Topology and Data Safety

• Use at least three production brokers and an odd controller count. Consider three separate controllers plus N brokers for critical or large clusters.
• Align topic RF, minISR, and producer acks with one failure model.
• Set kafka_rack only for real failure domains and verify replica placement.
• Load-test capacity, throughput, latency, retention, peak writes, and recovery time.
• Plan reassignment after broker admission. Existing topic RF does not increase automatically.
• Define backup/rebuild and disaster recovery procedures. Rehearse failed-node replacement and retirement.

Security and Network

• Bootstrap new production clusters with kafka_security: scram.
• Inject passwords through Vault, KMS, or secret management. Keep them out of Git and logs.
• Distribute only the public CA certificate, never its private key.
• Verify direct client resolution and routing to every inventory_hostname.
• Restrict 9092/9093 to authorized participants and 9308/9404 to monitoring.
• Review principals, topic/group/cluster ACLs, and quotas through a checklist.
• Schedule protected rotation of internal credentials and certificates.

Operations and Monitoring

• Verify /usr/local/bin/pigsty-kafka-health cluster reports health.
• Verify exactly one quorum leader and all expected controllers in current voters.
• Confirm no offline, under-replicated, or under-min-ISR partitions.
• Test production and consumption from the real application network with a real principal.
• Check healthy data in Kafka Overview, Kafka Instance, Kafka Topic, and Kafka Consumer.
• Confirm alert routing, log search, capacity thresholds, on-call duties, and rollback conditions.
• Define separate approvals for upgrades, feature changes, topic/user deletion, and cluster teardown.

For detailed alerts and PromQL, see Monitoring and Alerting; for metric semantics, see Metric Definitions.

Documentation Index and Next Steps

Use these pages for the next task:

Recommended reading order: Quick Start → Cluster Configuration → Parameter Reference → Day-to-Day Administration → Playbook → Monitoring and Alerting → FAQ.

2 - Configuration

KAFKA uses 15 persistent public parameters. The role derives topology, listeners, storage subdirectories, replication safety, authorization, and exporter placement consistently. Start with Quickstart. See Parameters for the field reference.

Pre-Deployment Checklist

Before writing the inventory, confirm these requirements:

• NODE manages the hosts, and the software repository is reachable. Every member and client can route directly to inventory_hostname.
• Each operation uses -l to select exactly all members of one kafka_cluster. Do not select a node, subset, or multiple clusters.
• kafka_seq is unique per cluster. Controller count is odd. Broker count meets failure-domain and capacity requirements.
• 9092, 9093, 9308, and 9404 do not conflict. INFRA nodes can reach both metrics ports.
• kafka_data uses a dedicated filesystem. Size it for retention, peak writes, replication, recovery time, and growth.
• Production uses kafka_security: scram. Synchronize node and administration clocks. Provide the Pigsty CA and application passwords from a secret source such as Vault.
• Review partitions, replica counts, min.insync.replicas, retention, client acks, retries, and consumer recovery.
• Prepare separate procedures for expansion, shrink, reassignment, upgrades, backup, restore, and controller membership changes.

Roles and Topology

kafka_role accepts only three values:

Either omit kafka_role on every member to use combined, or explicitly set it on every member. Identity checks reject mixed declarations. A cluster needs at least one controller-capable and one broker-capable node. Even controller counts trigger a warning. Production usually uses three controllers.

Single-Node Development Cluster

A single node runs both broker and controller. It cannot survive node failure. Use it only for development, testing, and feature validation:

kf-dev:
  hosts:
    10.10.10.10: { kafka_seq: 1 }
  vars:
    kafka_cluster: kf-dev

The initial broker count sets RF=1 and minISR=1. Do not use this single-node topology or default plaintext profile directly in production.

Three-Node Combined Deployment

Three combined broker/controller nodes provide a compact production starting point. Omit all role fields to use combined:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1 }
    10.10.10.12: { kafka_seq: 2 }
    10.10.10.13: { kafka_seq: 3 }
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms4G -Xmx4G'
    kafka_security: scram
    kafka_parameters:
      num.partitions: 3
      num.network.threads: 6
      num.io.threads: 16
    kafka_topics:
      - name: order.events
        partitions: 12
        replication_factor: 3
        config:
          min.insync.replicas: 2
          cleanup.policy: delete

Three initial brokers set RF=3 and minISR=2. The role owns this policy. Do not override internal-topic RF, default.replication.factor, or min.insync.replicas in kafka_parameters. The example 4G heap is illustrative. Use load tests to balance JVM heap, OS page cache, and other host processes.

Separating Controllers and Brokers

For critical or larger clusters, separate control and data planes. Explicit roles require a role declaration on every member:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1, kafka_role: controller }
    10.10.10.12: { kafka_seq: 2, kafka_role: controller }
    10.10.10.13: { kafka_seq: 3, kafka_role: controller }
    10.10.10.21: { kafka_seq: 4, kafka_role: broker }
    10.10.10.22: { kafka_seq: 5, kafka_role: broker }
    10.10.10.23: { kafka_seq: 6, kafka_role: broker }
  vars:
    kafka_cluster: kf-main
    kafka_security: scram

Controller-only nodes have no 9092 listener or protocol exporter. JMX still exposes KRaft and JVM metrics. At most two kafka_exporter instances run on broker-capable nodes with the lowest kafka_seq.

Dynamic KRaft and the Bootstrap Manifest

New clusters use dynamic quorum. Each node renders controller.quorum.bootstrap.servers, without static controller.quorum.voters. During first formatting:

• Generate a random Cluster ID rather than hashing the cluster name.
• Generate and freeze random initial controller Directory IDs.
• Format each node explicitly with --initial-controllers or --no-initial-controllers.
• After bootstrap startup, wait for leader election. Verify that every initial controller Directory ID entered live quorum.

The bootstrap-only facts live on every cluster member:

/etc/kafka/manifest.yml

Each scram member also stores /etc/kafka/secrets.yml. Every run retrieves the manifest and secrets from any member copy. The administration node keeps no Kafka state. Certificates reside in shared PKI at files/pki/kafka/, with CSRs in files/pki/csr/. Missing certificates are signed again by the Pigsty CA. The manifest stores only cluster identity, initial controller identities, security profile, and initial RF/minISR.

Live cluster state is authoritative for runtime facts:

• Manifest conflicts with live identity or security profile block ordinary playbooks.
• An old manifest with all data disks empty cannot revive the old cluster.
• Formatted storage without any member manifest blocks execution. Restore a manifest copy on any member first.
• A formatted scram cluster without secret material on any member also blocks execution.

The manifest records bootstrap facts. After commissioning, live Raft state controls membership. Playbooks join new combined/controller members by fresh formatting, observer catch-up, and add-controller promotion. A strict-subset kafka-rm.yml run retires members with remove-controller and broker unregistration. See Expand Cluster and Shrink Cluster.

Identity Parameters

A formatted node reads cluster.id and node.id from ${kafka_data}/metadata/meta.properties. It checks them against the manifest and inventory. After startup, it checks initial controller Directory IDs against live quorum. Identity mismatches stop execution for protection. Never bypass this by deleting meta.properties or erasing data.

Network and Listeners

Public parameters expose ports only. They do not expose bind addresses, advertised addresses, or listener maps:

The listener conventions are fixed:

• Brokers bind 0.0.0.0; controllers bind inventory_hostname.
• Broker advertised.listeners uses inventory_hostname.
• Controller bootstrap addresses also use inventory_hostname.
• plaintext uses PLAINTEXT for BROKER and CONTROLLER.
• scram uses SASL_SSL + SCRAM-SHA-512 for BROKER and mutual TLS for CONTROLLER.

Clients must resolve and directly reach every broker’s inventory_hostname. Version 1 does not support NAT, public-address mapping, multiple client networks per broker, or arbitrary listener overrides. kafka_parameters cannot bypass these limits.

Kafka clients connect directly to brokers. Configure several seeds in bootstrap.servers. After fetching metadata, clients connect to partition leaders. Do not use HAProxy, Keepalived VIP, or a cloud load balancer as the regular data-plane entry. They cannot interpret Kafka metadata or remove the need to reach every advertised.listeners address.

DNS or a TCP load balancer can optionally provide bootstrap discovery only. Clients must still reach every broker directly. See Connecting Application Clients.

Permit these minimum network flows:

Metrics use HTTP even with scram. Restrict these ports to the monitoring network through firewalls.

Storage, Heap, and Rack

You set only the root directory:

kafka_data: /data/kafka

The role uses ${kafka_data}/data for topics and ${kafka_data}/metadata for KRaft. Set kafka_data to a dedicated absolute path. Forbidden paths are /, /data, /var, /etc, /opt, /usr, /home, /root, and /pg.

Plan production capacity for retention, peak messages, replication, partition/segment counts, disk latency and throughput, file descriptors, recovery time, heap, and page cache. The role generates one log.dirs. Multi-disk JBOD, disk replacement, and automatic data migration require separate procedures.

For deployments spanning fault domains, declare kafka_rack consistently on all broker-capable nodes:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }
10.10.10.22: { kafka_seq: 5, kafka_role: broker, kafka_rack: az-b }
10.10.10.23: { kafka_seq: 6, kafka_role: broker, kafka_rack: az-c }

Either set racks on all broker-capable nodes or omit them on all. Rack changes trigger safe rolling restarts without moving existing replicas.

Replication Policy

At the first bootstrap, the policy is derived from the initial broker count:

replication_factor = min(3, broker_count)
min_insync_replicas = max(1, replication_factor - 1)

The manifest freezes initial topic-default RF, internal-topic RF, and cluster minISR. After expansion:

• default.replication.factor retains its initial value. Kafka 4.3 cannot update it through dynamic broker configuration.
• Existing internal and business topics retain their RF.
• Broker admission does not imply balanced data.
• RF changes require a reviewed kafka-reassign-partitions.sh plan. Increasing the static default also requires controller HA or an explicit maintenance window. A safe full-cluster rolling restart applies it.

Configure producer acks, idempotence, retries, batching, and compression on clients. They are not broker-role parameters.

kafka_parameters

kafka_parameters is the only custom broker configuration interface. It defaults to {} and renders only on broker-capable nodes. Use keys that the role does not own, such as num.partitions, threads, buffers, retention, and segments.

The following patterns are owned by the role and must not be overridden:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

Any reserved key causes identity preflight to fail before writing files.

Security and Declarative Resources

kafka_security: scram selects a complete production profile. Its controls cannot be combined individually. It enables:

• Pigsty CA certificates for each node.
• Mutual TLS on controllers.
• SASL_SSL + SCRAM-SHA-512 for broker/client and inter-broker traffic.
• StandardAuthorizer, default denial, and role-managed administration and monitoring identities.
• Minimum monitoring ACLs before starting the protocol exporter.

Application resources are declared through two domain objects:

kafka_security: scram
kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520
kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete

Topic creation is idempotent. Partitions can only increase, and only declared settings change. RF changes fail with a reassignment instruction. User passwords, ACLs, and specified quota fields converge idempotently. Removing topic or user entries does not delete those resources.

Ordinary playbooks cannot change the security profile after bootstrap. Protected rotation changes internal credentials or certificates. Online plaintext to scram migration still needs a separate state machine planned for future work.

Packages and File Layout

Platform mappings install java-runtime and kafka-stack. The payload verified on 2026-07-16 contained Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0. Actual versions depend on target-platform repositories and installed packages.

The role manages these files. Put persistent settings in pigsty.yml, not generated node files. Never copy passwords, private keys, or role-managed secrets into inventory, logs, or tickets.

3 - Parameters

KAFKA exposes 15 persistent parameters. The role derives topology, listeners, security, storage subdirectories, replication safety, and exporter placement consistently. Additional persistent variables cannot override these derived settings.

Parameter Overview

Set kafka_cluster and kafka_seq. kafka_role has a default. Omit it on every member or declare it on every member.

Identity and Topology

kafka_cluster

Required cluster identity. Start with a letter or digit. Use only letters, digits, underscores, and hyphens:

kafka_cluster: kf-main

This identifies all cluster members, generates instance names, and locates the manifest. Every kafka.yml lifecycle operation must select all cluster members with precise -l.

kafka_seq

Required non-negative integer, unique within kafka_cluster. It directly becomes KRaft node.id:

10.10.10.11: { kafka_seq: 1 }

The instance name is ${kafka_cluster}-${kafka_seq}. After formatting, do not change or reuse a sequence number that still has data.

kafka_role

Defaults to combined, and accepts only:

If all members omit the role, they use combined. If one declares it, all must declare it. Legacy aliases are unavailable.

kafka_cluster_id

Unset by default. Use it only to assert an existing cluster identity for takeover or recovery. It requires a 22-character Kafka UUID:

kafka_cluster_id: MkU3OEVBNTcwNTJENDM2Qk

Omit it for ordinary new clusters. The role generates a random Cluster ID and writes /etc/kafka/manifest.yml on every member. This parameter cannot relabel data. Conflicts with the manifest or meta.properties block execution.

kafka_rack

An optional broker fault-domain label, rendered as broker.rack:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }

Set this on all broker-capable nodes or none. Controller-only nodes ignore it. Changes require a strict rolling restart but do not reassign replicas.

Storage, JVM, and Network

kafka_data

The data root directory, defaulting to /data/kafka:

kafka_data: /data/kafka

The role derives ${kafka_data}/data and ${kafka_data}/metadata. Use a dedicated absolute path. Forbidden roots are /, /data, /var, /etc, /opt, /usr, /home, /root, and /pg. kafka-rm.yml deletes the whole root by default. Do not store other service or business files there.

kafka_heap_opts

The Kafka JVM heap, defaulting to:

kafka_heap_opts: '-Xms1G -Xmx1G'

Set production heap size through load and memory tests. Usually keep Xms equal to Xmx. Reserve memory for OS page cache and other processes.

kafka_port

Broker/client port, default 9092. Only broker-capable nodes listen here. plaintext uses PLAINTEXT; scram uses SASL_SSL + SCRAM-SHA-512.

kafka_controller_port

Controller port, default 9093, the conventional KRaft port. Only controller-capable nodes listen here. Check for conflicts with other services yourself; the role does not detect them.

All four public ports must differ. Brokers bind 0.0.0.0. Controller listeners, broker advertised addresses, and controller bootstrap addresses use inventory_hostname. No separate address parameters exist.

kafka_parameters

Default {}. This is the only custom broker parameter mapping, rendered only on broker-capable nodes:

kafka_parameters:
  num.partitions: 12
  num.network.threads: 6
  num.io.threads: 16
  log.retention.hours: 168
  log.segment.bytes: 1073741824

The following keys or patterns are owned by the role and cannot be overridden through this mapping:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

The role owns identity, listeners, security, storage, and replication policy. Reserved keys cause preflight failure.

Observability

kafka_jmx_exporter_port

JMX HTTP port, default 9404. Every Kafka JVM receives the agent and registers as job=kafka; no toggle exists. Lifecycle checks use a separate Kafka CLI/metadata channel, not JMX. INFRA nodes must reach this port. kafka_security: scram does not enable HTTPS here. Protect the endpoint with the monitoring network and firewall.

kafka_exporter_port

Protocol kafka_exporter HTTP port, default 9308. The role starts and registers exporters on the first two broker-capable nodes by kafka_seq. A single broker runs one. Full runs refresh discovery targets for current placement. Ordinary playbooks do not stop stale exporter services on previously selected nodes.

Exporter protocol version, TLS/SCRAM settings, and placement are internal conventions. No additional public toggle or options parameters exist.

Security and Resources

kafka_security

Defaults to plaintext, and accepts only:

scram configures Pigsty CA certificates, internal/administration/monitoring identities, and TLS/SCRAM/ACL activation order together. The manifest records this profile. After formatting, ordinary reruns cannot switch between plaintext and scram in either direction.

Certificates use shared cert_validity, default 7300d. KAFKA has no separate validity parameter.

kafka_users

Default []; permitted only in scram mode. Supply a list of mappings with only name, password, acls, and quota. Non-mappings or unknown top-level keys fail before resource convergence:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
      - resource: transactional_id
        name: order.
        pattern: prefixed
        operations: [Write, Describe]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

Constraints:

• name must be unique in the list. Required password must contain at least 12 characters and should reference secret management.
• ACL resource: topic, group, transactional_id, or cluster.
• pattern: literal (default) or prefixed.
• Operations: Read, Write, Create, Delete, Alter, Describe, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite.
• Quotas: producer_byte_rate, consumer_byte_rate, request_percentage, controller_mutation_rate.

The role applies each declared user’s SCRAM password, full ACL set, and specified quota fields. Removing entries does not delete principals or credentials. Deletion and revocation require separate audited operations.

kafka_topics

Default []. Supply mappings with only name, partitions, replication_factor, and config. Non-mappings or unknown top-level keys fail before resource convergence:

kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 604800000

Identity prechecks validate only unique name values. Kafka checks partitions and RF at creation: both must be at least 1; RF cannot exceed current broker count. These errors appear during resource convergence, not --check. The rules are:

• Create missing topics idempotently.
• Increase partitions only; decreases fail.
• Reject RF differences from live state and require explicit reassignment.
• Change only keys declared in config.
• Never delete a topic because its list entry disappeared.

Transient Protected Operational Variables

Use these variables only with command-line -e for individual operations. They are outside the 15 persistent parameters. Do not put them in pigsty.yml:

Rotation actions are mutually exclusive and require one exact complete cluster. kafka-rm.yml deletes data and node-local /etc/kafka recovery state by default. kafka_rm_data=false retains both. Before removal, confirm the target and backup or rebuild intent. See Playbooks for commands and details.

kafka_safeguard

For kafka-rm.yml only. Default false. When true, abort before deregistration, retirement, service shutdown, or deletion. This boolean switch does not test whether the cluster is alive.

kafka_rm_data

For kafka-rm.yml only. Default true. Delete all of kafka_data and /etc/kafka, including the manifest, credentials, and recovery state needed to readopt retained storage. false keeps both paths. It still deregisters targets, stops services, and removes runtime integration configuration.

kafka_rm_pkg

For kafka-rm.yml only. Default false. When true, uninstall mapped kafka-stack packages: Kafka, Kafka Exporter, and JMX Exporter. Never uninstall the shared Java runtime.

4 - Administration

KAFKA installs Kafka in /opt/kafka and manages it with Systemd. Declare persistent settings in pigsty.yml. Do not edit generated node files manually.

These CLI examples use the generated /etc/kafka/admin.properties. Keep --command-config even with plaintext. Commands then retain the same structure if the administration channel changes to scram. Replace <broker>:9092 with a reachable inventory_hostname and port.

Quick Reference

See Configuration for cluster definitions and parameters, Playbooks for execution behavior, and Monitoring for observability.

Status Check

Check the service and recent logs on any Kafka node:

systemctl status kafka
systemctl is-enabled kafka
journalctl -u kafka --since '-30 min' --no-pager

The protocol exporter runs on at most two broker-capable nodes with the lowest kafka_seq. On those nodes, also check:

systemctl status kafka_exporter
journalctl -u kafka_exporter --since '-30 min' --no-pager

Check listeners and metric endpoints:

ss -lntp | grep -E ':9092|:9093|:9308|:9404'
curl -fsS http://<kafka-ip>:9404/metrics | grep -E '^(jmx_scrape_error|kafka_server_raft_state|kafka_server_broker_messages_in_total)'
curl -fsS http://<exporter-ip>:9308/metrics | grep -E '^(kafka_brokers|kafka_topic_partitions)'

kafka_up and kafka_exporter_up are VictoriaMetrics recording metrics. They may not appear at raw endpoints. JMX should expose jmx_scrape_error 0.0, JVM metrics, and role-appropriate kafka_ metrics.

Health Check

Lifecycle checks use the administration channel, not JMX. They check dynamic quorum, unavailable partitions, under-replication, and under-min-ISR:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties

The check passes only if JSON returns healthy: true. Use it for read-only diagnostics. It does not replace end-to-end business validation.

Every playbook run executes the parser regression suite, pigsty-kafka-health selftest, immediately after installation. If it fails, stop changes and investigate. The health result is then untrustworthy.

KRaft Quorum Status

Query the dynamic quorum from any available broker:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

Verify these conditions:

• LeaderId exists and identifies an expected controller.
• CurrentVoters contains the expected members. A joining node first appears in CurrentObservers.
• MaxFollowerLag and MaxFollowerLagTimeMs do not grow continuously.
• Dashboards show exactly one active controller.

To confirm the dynamic quorum (KIP-853) feature level, inspect kraft.version with /opt/kafka/bin/kafka-features.sh ... describe.

Inspect controller replication state:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --replication

Stop other changes if no leader exists, a member persistently lags, or voters differ from expectations. Preserve logs, the manifest, and meta.properties before analysis. Remove dead voters through shrink or replacement. Never rewrite quorum state manually.

Manage Topics

Production topics should be declared in kafka_topics in pigsty.yml:

kafka_topics:
  - name: orders
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      retention.ms: 604800000

Converge after editing the declaration:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role creates topics idempotently, increases partitions only, and changes only declared configuration keys. RF changes fail and require explicit partition reassignment. Removing an inventory entry does not delete the topic.

Read-only topic inspection:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic orders

You can create ad hoc or externally managed topics with the CLI. These are not written back to pigsty.yml. Never manage the same topic both manually and declaratively. Topic deletion deletes business data. It requires separate approval, exact-name confirmation, and a recovery plan. This page therefore gives no generic deletion command.

Manage Users and ACLs

With kafka_security: scram, application identities should be managed through kafka_users:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: orders
        operations: [Read, Write, Describe]
      - resource: group
        name: order-worker
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

A full run idempotently applies the password, user ACL set, and explicitly declared quota fields. Never commit plaintext passwords or print them in logs. Removing an entry does not delete the principal or credentials. Deletion and full revocation need a separately reviewed procedure.

Verify Read/Write

Validate end-to-end operation with a test topic. The console producer and consumer use the same client configuration:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke

Consume in another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke \
  --from-beginning \
  --group ops-smoke-check

Run production acceptance from the actual client network. Check DNS, advertised.listeners, certificates, ACLs, producer ACKs, consumer commits, and end-to-end latency. Broker-local tests alone are insufficient.

Manage Consumer Groups

List and inspect consumer groups:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group order-worker

Assess lag against consumption rate and business SLOs. A short backlog can be normal batch processing. Sustained growth with consumption below production means the group cannot catch up. Offset resets can duplicate or skip messages. They require separate approval, exact group/topic confirmation, and a replay plan.

Config Cluster

After editing pigsty.yml, run against complete clusters:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Live health and the static fingerprint determine the execution path:

• Unhealthy or stopped: Start only stopped controllers. Restore quorum and wait for catch-up, then start brokers. If static settings also changed, roll still-online members afterward.
• Controller join pending: Join each node as an observer, wait for catch-up, then promote with add-controller. Process one node at a time.
• Healthy with new pure brokers: Format, start, and verify registration one node at a time.
• Healthy with static changes: Restart one node at a time. Before and after each restart, check controller majority, voter catch-up, offline partitions, under-min-ISR, and ISR catch-up.
• No static changes: Do not restart Kafka.

Do not bypass the full state machine with -t kafka_config. kafka_provision applies topics, users, ACLs, and quotas. The role determines whether static changes require restarts.

Expand Cluster

Add new kafka_role: broker, combined, or controller members to a healthy cluster through the inventory. Assign each a never-used kafka_seq. A host can belong to only one Kafka cluster. Ensure that Pigsty manages the node. Run against complete clusters:

./node.yml  --check -l 10.10.10.14    # manage the new node
./kafka.yml --check -l kf-main        # dry run first
./kafka.yml -l kf-main                # admit / join new members one at a time

The role processes one new node at a time according to its type:

• Pure broker: Format, start, and verify registration and an unfenced state (admit).
• Combined/controller: Format with --no-initial-controllers. Start as an observer, catch up on metadata, and promote with add-controller. Verify voter membership and full cluster health (join).

quorum-join-hosts and broker-admission-hosts list processed nodes at completion. Note these behaviors:

• A new controller-capable node changes controller.quorum.bootstrap.servers on all members. Existing nodes then undergo one strict rolling restart with health checks.
• An even controller count triggers a warning. It adds no fault tolerance; keep the count odd.

Joining does not move existing partitions to the new broker. Separately generate, review, and monitor a kafka-reassign-partitions.sh plan. Limit disk and network load, and prepare rollback. Registration alone does not complete expansion.

More brokers do not automatically increase the replication policy. Kafka 4.3 cannot change default.replication.factor dynamically. After expansion from one broker to three, the initial RF=1 remains. New topics without explicit RF still use RF=1. First reassign existing partitions. Then plan controller HA or a maintenance window.

Apply the new static default with a safe full-cluster rolling restart. Do not bypass downtime checks to change this default.

Shrink Cluster

kafka-rm.yml retires a strict subset of cluster members. Selecting all members performs teardown. Retirement uses a surviving member to remove the target from live metadata:

./kafka-rm.yml -l 10.10.10.13     # retire one member: remove voter entry, unregister broker, clean the host

Retirement first deregisters monitoring targets and stops services. For voters, it runs remove-controller, strictly serializing multiple removals. Next, kafka-cluster.sh unregister removes broker registration. Finally, it cleans local configuration and data according to kafka_rm_data. Broker unregistration tolerates failure so the procedure can resume or handle unreachable nodes. Remove the inventory entry from pigsty.yml only after checking live quorum, broker registrations, replica health, and target local state.

Before retirement, confirm that remaining controllers form a majority and have an odd count. Remaining brokers must support the highest topic RF. If the target still holds replicas, the role warns. Those partitions remain under-replicated until explicit reassignment or replacement with the same kafka_seq. Such a replacement inherits assignments and resynchronizes automatically. For planned shrink, drain replicas through reassignment before retirement.

Replace Failed Node

For permanent node loss, such as destroyed disks or retired hardware, retain the IP and kafka_seq. Replace it in three steps:

./kafka-rm.yml -l 10.10.10.13     # 1. retire the dead member: remove voter entry & broker registration (works while unreachable)
./node.yml     -l 10.10.10.13     # 2. provision the replacement machine (repaired or new, same IP)
./kafka.yml    -l kf-main         # 3. rejoin: format, catch up, admit/promote; replica assignment is re-inherited and resynced

Step 1 delegates metadata operations to a surviving member, so it works for unreachable nodes. It also removes monitoring targets to stop KafkaDown alerts. In step 3, the same kafka_seq makes the broker inherit its previous partition assignment. It resynchronizes from surviving replicas without manual reassignment.

If you skip step 1, kafka.yml stops during configuration on the re-imaged node. It reports the stale voter directory ID and exact kafka-rm.yml command. Run that command, then retry. The join procedure can resume safely: another kafka.yml run continues from live state.

Change Address or Port

inventory_hostname supplies broker advertised addresses and controller bootstrap addresses. Changing it, kafka_port, or kafka_controller_port is a high-risk static change. These changes affect client metadata, broker communication, or quorum. Coordinate DNS, certificate SANs, routing, firewalls, bootstrap addresses, monitoring targets, and every cluster member.

Rotate Credentials and Certificates

Healthy, formatted scram clusters support two mutually exclusive protected actions: internal credential rotation and certificate rotation. Both require exact complete clusters and matching kafka_rotate_confirm. Run --check first where possible. The same Pigsty CA reissues certificates, so old and new certificates trust each other. Strict rolling restarts apply rotation one node at a time.

See Protected Rotation for commands and failure behavior. The security profile remains a bootstrap-only setting. Rotation does not support online migration from plaintext to scram.

Data Protection and Recovery

Kafka protects data through replicas in separate failure domains, correct min-ISR, producer ACKs, and tested recovery procedures. The role provides no data backup, automatic broker drain, or cross-region disaster recovery. Planned shrink requires manual reassignment first.

When a disk or node fails:

1. Inspect Kafka Overview/Node dashboards, quorum, ISR, offline partitions, and under-min-ISR partitions.
2. Preserve journalctl -u kafka, node metrics, the manifest, server.properties, and meta.properties.
3. Confirm the node role, node.id, cluster ID, directory ID, and remaining replica availability.
4. If the node is unrecoverable, follow Replace Failed Node: kafka-rm.yml retirement, node.yml provisioning, then kafka.yml join. If its disk survives and only the service fails, first try ordinary convergence. Do not immediately retire it or delete meta.properties.
5. Use a separately reviewed procedure for data movement, including reassignment and RF changes.

Log Diagnostics

journalctl -u kafka -f
journalctl -u kafka_exporter -f
journalctl SYSLOG_IDENTIFIER=kafka --since today
journalctl SYSLOG_IDENTIFIER=kafka_exporter --since today

VictoriaLogs/Grafana queries:

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter

Check service logs, listener ports, administration-channel health, quorum, broker/partition/ISR state, client addresses and certificates/ACLs, then consumer lag. See Monitoring for dashboards and alerts.

5 - Playbook

kafka.yml deploys Apache Kafka 4.1+ dynamic KRaft and applies security, resources, and monitoring. kafka-rm.yml removes members or tears down clusters.

kafka.yml

./kafka.yml --check -l kf-main   # dry run first
./kafka.yml -l kf-main           # create or converge one cluster
./kafka.yml                      # bare run: create / converge ALL kafka clusters

Every selected cluster must be complete. Select one cluster, several clusters, or the whole inventory without a limit. Execution is serial within each cluster and concurrent across clusters. Partial cluster selection is rejected.

Check mode validates the API, cluster completeness, roles, racks, ports, manifest, and inspectable file changes. It skips formatting, startup, and live acceptance checks. Passing --check does not guarantee runtime success.

Execution Stages

A single kafka.yml play runs node_id, then kafka, like pgsql.yml. The role has six stages. Launch controls cross-node ordering: parallel bootstrap, serial controller join, serial broker admission, and strict rolling restarts.

any_errors_fatal: true stops further dangerous work after a stage failure. Fix the cause, then rerun on the full cluster. The role uses live state and persistent fingerprints to resume without blindly reformatting.

Lifecycle Paths

Configuration uses the role’s administration channel to assess health and select one execution path:

Cold Start, First Deployment, or Repair

Stopped or unhealthy clusters enter convergence:

1. Start controller-capable nodes.
2. Wait for the controller listener and a quorum leader.
3. At first bootstrap, verify that initial controller Directory IDs entered live quorum.
4. Start pure brokers.
5. Wait for broker listeners and full cluster health.
6. Save the static fingerprint only after proving that the configuration runs successfully.

Startup, admission, and rolling decisions use only the role’s Kafka CLI/metadata administration channel. JMX does not participate.

Adding Brokers or Controllers to a Healthy Cluster

Newly formatted kafka_role: broker nodes enter through admit, one at a time. Each must start, register, and become unfenced before the next proceeds.

New combined/controller nodes use serial join. A commissioned cluster formats each with --no-initial-controllers, starts it as an observer, and waits for metadata catch-up. The role then promotes it with add-controller and verifies voter membership and full health. Interrupted runs resume from live state. If node.id retains a dead predecessor’s voter entry, configuration fails with the exact kafka-rm.yml retirement command.

Admission and join prove membership only. They do not move existing partitions to new brokers. Run explicit reassignment separately.

Static Changes on a Healthy Cluster

When the rendered static fingerprint changes, the strict rolling restart handles one node at a time:

• Before restart, check controller majority, zero voter lag, recent catch-up, offline partitions, under-replication, and under-min-ISR. Also check effective partition ISR with the target excluded.
• After restart, require controller voter membership and catch-up, broker registration and unfenced state, and replica reentry into ISR.
• Any failed check immediately stops work on remaining nodes.

When repairs and static changes coincide, convergence starts only stopped members. It does not restart online members in parallel. After quorum recovery and catch-up, strict rolling restarts apply remaining static changes.

An unchanged static fingerprint causes no restart. Dynamic resources still update online during resource convergence.

Task Tags

Run full kafka.yml for ordinary configuration changes. Let the role select the lifecycle path. Tags mainly support development, diagnostics, and controlled repair. Do not bypass the state machine with -t kafka_config or single-node limits.

Identity, Formatting, and Manifest

Before writing configuration, the role verifies:

• Every selected cluster includes all members.
• kafka_seq is unique; roles are all omitted or all explicit.
• At least one controller and one broker exist.
• All broker-capable nodes have racks, or none do.
• Ports are valid and distinct; kafka_parameters does not override role-managed keys.
• The manifest, security profile, meta.properties, and live identity agree.

New clusters generate random Cluster and initial controller Directory IDs, then format nodes explicitly for dynamic quorum. Existing ${kafka_data}/metadata/meta.properties triggers local Cluster/Node ID validation. Only first bootstrap compares initial controller Directory IDs with live quorum. After commissioning, live Raft state controls membership. The role never reformats existing storage automatically.

The authoritative bootstrap manifest lives on every cluster member:

/etc/kafka/manifest.yml

scram members also store /etc/kafka/secrets.yml. The administration node keeps no Kafka state and retrieves both files from member copies each run. Live state controls runtime facts. On conflict, ordinary playbooks do not silently overwrite either side:

• Formatted storage without a member manifest copy blocks execution. Restore the file on any member first.
• A manifest with all data disks empty blocks execution.
• Conflicting Cluster ID, security profile, or controller identity blocks execution.
• A new node with a stale predecessor voter entry for its node.id must first be retired with kafka-rm.yml.

Do not delete meta.properties, the manifest, or the secrets to bypass these protections.

Static Fingerprint and Recoverable Re-runs

The role fingerprints static files that affect Kafka. It writes /etc/kafka/.pigsty-applied-static.sha256 only after one of these conditions:

• Convergence starts the cluster successfully and passes global health checks.
• Strict rolling restarts the node, completes catch-up, and passes post-restart checks.

Interrupted runs do not mark unverified changes as applied. The next full run still detects the pending static restart.

Resource Convergence and Monitoring Registration

After full cluster health, resource and monitoring stages run in order:

1. Apply role-managed dynamic cluster min-ISR.
2. Idempotently apply kafka_users credentials, ACLs, and declared quotas.
3. Idempotently create kafka_topics, increase partitions, and apply declared settings.
4. Check internal-topic RF drift without automatic reassignment.
5. Start protocol exporters on the first two broker-capable nodes by kafka_seq.
6. Refresh discovery targets on all INFRA nodes.

Each instance has one target file. It contains JMX and any selected protocol target under the kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

Full runs refresh targets for current exporter placement. kafka-rm.yml deregistration deletes targets.

Protected Rotation

Rotation variables are one-time extra-vars; do not put them in pigsty.yml. Run only one of the mutually exclusive actions. Every member must be formatted. The cluster must be healthy, use scram, and retain role-managed secrets. kafka_rotate_confirm must exactly match the cluster name.

Internal Credential Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

The role maintains active and standby internal identities. It updates the inactive credential through the live administration channel, then atomically switches the protected local record. A strict rolling restart follows. The old active identity becomes standby for the next rotation. Reruns can therefore recover after interruption.

Certificate Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

The role removes issued node certificates from shared PKI. The same Pigsty CA issues new node keys and certificates. It updates node PEM bundles, then performs strict rolling restarts. Old and new certificates share trust, so staged CA changes are unnecessary. If the health precheck fails, rotation never starts and deployed certificates stay unchanged.

kafka-rm.yml

Removal uses separate kafka-rm.yml, never kafka.yml. It requires non-empty -l/--limit and fails before roles if omitted. Selecting all members with -l tears down a cluster. A strict subset retires members. Both follow this order:

1. Deregister VictoriaMetrics targets (kafka_deregister).
2. Stop and disable kafka and kafka_exporter (kafka).
3. Remove voter entries and broker registrations through survivors (kafka_retire). Survivors exist only for strict-subset selection.
4. Delete exporter configuration, Systemd environments/units, and helpers (kafka_config).
5. Delete data and node-local /etc/kafka recovery state (kafka_data), controlled by kafka_rm_data.
6. Optionally uninstall packages (kafka_pkg), controlled by kafka_rm_pkg.

Before deregistration or shutdown, the role validates a dedicated absolute kafka_data path. It must contain no ./.. segments. Forbidden roots are /, /data, /var, /etc, /opt, /usr, /home, /root, and /pg. kafka_safeguard set to true in inventory or the command line immediately stops removal without deletion. Identity conflicts, exporter errors, and ordinary startup failures do not justify data deletion. First run kafka.yml and read the failure reason.

Cluster Teardown

./kafka-rm.yml -l kf-main                          # Remove the cluster: deregister monitoring and stop services; deletes data and /etc/kafka recovery state by default
./kafka-rm.yml -l kf-main -e kafka_rm_data=false   # Keep data and /etc/kafka recovery state; remove only service integration
./kafka-rm.yml -l kf-main -e kafka_rm_pkg=true     # Also uninstall the kafka-stack packages (the shared Java runtime is not removed)

Member Retirement

./kafka-rm.yml -l 10.10.10.13                      # Retire one member: drop its voter entry and broker registration, then clean the node

Partial retirement needs a broker-capable (combined/broker) survivor and a controller-capable (combined/controller) survivor outside -l. One combined node can satisfy both. Missing either stops execution before deregistration or shutdown. Survivors run remove-controller, strictly serializing multiple targets, then broker unregister, before local cleanup. Delegation also handles dead, unreachable targets: step one of Replace Failed Node.

Broker unregistration tolerates failure for repeatable execution. After a real run, verify live quorum, broker registrations, and replica health. Playbook success alone does not prove retirement.

Plan retirement even with automation. Remaining controllers should have an odd count and live majority. Remaining brokers must support the highest topic RF. The playbook warns if departing brokers still hold replicas. For planned shrink, drain them through reassignment first.

Playbook Boundaries

Neither playbook reassigns partitions, balances data, deletes topics/users, or migrates online from plaintext to scram. They do not upgrade versions, finalize features, back up data, or perform disaster recovery. They do not deploy Connect, Schema Registry, MirrorMaker, or Cruise Control. See Module Boundaries and Administration.

6 - Monitoring

Kafka monitoring combines metrics, logs, dashboards, and alerts. It covers JVM internals and the Kafka protocol view. You can inspect partitions, ISR, and consumer lag together with JVM state, request queues, and KRaft controller health.

Scrape Architecture

The KAFKA module uses two complementary exporters:

On each INFRA node, the role creates one discovery file per instance. JMX and any selected protocol target share that file and kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

Single-broker clusters run one protocol exporter; larger clusters run at most two. Controllers register only JMX. Controllers and unselected brokers have no protocol target by design. Full runs refresh target files for current placement. kafka-rm.yml deregistration removes instance targets.

Label Model

Both targets use job=kafka. JMX has a role label; protocol targets do not.

JMX Target

Protocol Exporter Target

Protocol targets carry cls, ins, ip, and instance (10.10.10.11:9308), without role or node_id. vmagent-side rules distinguish availability by those labels. kafka_up is up{job="kafka",role=~".+"}. kafka_exporter_up is up{job="kafka",role=""}.

Each exporter queries the whole cluster through a broker. Both can return identical topic, partition, and consumer-group state. Recording rules deduplicate exporters before aggregating cluster rates. In scram mode, the role generates exporter TLS/SCRAM settings from its monitoring identity.

Grafana Dashboards

Pigsty ships four complementary dashboards:

Kafka Overview

cls=All shows all Kafka clusters. Selecting one cls changes the same panels to that cluster’s overview.

Main content:

• Inventory of clusters, brokers, topics, partitions, and consumer groups
• Broker availability, exporter health, and cluster workload
• Leaderless, under-replicated, ISR deficit, and non-preferred replica
• Topic offset progress, consumer commit progress, and total lag
• Consumer group members, lag ranking, and topic/group drill-down
• Kafka/exporter log volume, firing alerts, and log detail

Common variables: cls, members, topic, group, topk.

Kafka Instance

Use ins to select a broker or controller JVM, including controller-only nodes. The dashboard also shows related host resources.

Main content:

• Instance identity, role, JMX availability, and scrape quality
• JVM heap, GC, threads, buffer pool, CPU, FD, and uptime
• Broker throughput, replication state, request errors/latency/queues, and handler/network idle
• KRaft member state, metadata log, controller health, and event latency
• Node CPU/memory, disk I/O, network, filesystem, and Kafka logs

Common variables: cls, ins, ip.

Kafka Topic

Use cls and topic to select a logical topic and inspect its topic/partition protocol state.

Main content:

• Topic and partition inventory, leader, replicas, ISR, and preferred leader
• Current offset, retention span, and message append rate
• Leaderless, ISR deficit, and non-preferred replica
• Associated consumer groups, commit progress, and lag

Common variables: cls, topic, topk.

Kafka Consumer

Use cls and group to inspect consumer-group members, committed offsets, progress, and backlog.

Main content:

• Consumer group inventory and member count
• Committed offsets by group/topic/partition
• Commit rate, total lag, maximum partition lag, and backlog trend
• Drill-down from group to topic/partition

Common variables: cls, group, topic, topk.

Choosing a Dashboard

Recording Rules

The Kafka rule file lives at /infra/rules/kafka.yml. The main recorded metrics are:

Offset-change rates measure progress, not client requests. Truncation, offset rollback, and exporter restarts can briefly produce negative changes. Rules use clamp_min(..., 0) to retain forward progress only.

Alert Rules

An unclean leader election can indicate data loss. Immediately preserve controller and broker logs. Identify affected topics and replicas before choosing recovery actions.

Common PromQL

Check scrape targets:

kafka_up
kafka_exporter_up
up{job="kafka"}

Check the replication health of a cluster:

sum by (cls) (kafka_server_replica_manager_under_replicated_partitions{job="kafka"})
sum by (cls) (kafka_server_replica_manager_under_min_isr_partitions{job="kafka"})
max by (cls) (kafka_controller_offline_partition_count{job="kafka"})

Check consumer lag:

topk(20, kafka_consumergroup_lag_sum{cls="kf-main"})

Check request saturation and latency:

kafka_server_request_handler_idle_ratio{job="kafka",cls="kf-main"}
max by (ins,request,quantile) (
  kafka_network_request_total_time_seconds{job="kafka",cls="kf-main",quantile=~"0.95|0.99"}
)

Log Queries

Kafka writes stdout and stderr to Journald. Vector forwards these records to VictoriaLogs under job:syslog.

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter
ip:10.10.10.11 job:syslog (unit:kafka OR app:kafka)

Kafka Instance log panels use similar queries and show time, level, Systemd unit, and message. Compare the same time window across logs, KRaft, ISR, request queues, GC, disk I/O, and network metrics.

Verifying the Monitoring Chain

Verify the raw endpoints on a Kafka node:

curl -fsS http://<kafka-ip>:9404/metrics | grep '^jmx_scrape_error'
curl -fsS http://127.0.0.1:9308/metrics | grep '^kafka_brokers'

Check discovery files on an INFRA node. Each instance has one file; selected nodes include both JMX and protocol targets:

ls -l /infra/targets/kafka/
cat /infra/targets/kafka/kf-main-1.yml

Query up{job="kafka"} in VictoriaMetrics, or recording metrics kafka_up and kafka_exporter_up. Use native Prometheus up for endpoint availability; custom exporter samples can remain briefly after failed scrapes. If raw endpoints work but recording metrics are absent, check discovery files, target state, network, rule loading, and labels in order. If JMX HTTP works but jmx_scrape_error is 1, inspect Kafka logs and MBean matching in /etc/kafka/jmx_exporter.yml.

For complete metric semantics, see Metric Definitions.

7 - Metrics

Both metric sources use job=kafka. JMX targets have a role label and collect each JVM’s internal state. Protocol targets lack role and collect logical cluster, topic, partition, and consumer-group state through Kafka protocol. Protocol exporters run on the one or two broker-capable nodes with the lowest kafka_seq. Single-broker clusters run one.

JMX uses an allowlist for standard JVM and selected broker, replication, request-path, and KRaft metrics. It excludes per-client and per-partition MBeans to limit series counts. Protocol exporters provide partition details.

Common Labels

Both target types use job value kafka. Presence of the role label distinguishes JMX from protocol targets.

Some metrics also carry dimensions such as topic, partition, broker, consumergroup, request, version, error, quantile, state, or operation.

Availability and Scrape Metrics

Protocol Exporter Metrics

Protocol exporters report the same logical cluster state. Before cluster aggregation, deduplicate according to metric meaning. Do not simply sum every ins.

Broker and Topic

current_offset - oldest_offset estimates the retained offset span. It measures neither bytes nor, for compacted topics, an exact message count.

Consumer Group

These series may be absent for consumers that never commit offsets, use external offset storage, or have not consumed the topic.

Exporter Itself

JMX: JVM Baseline

excludeJvmMetrics: false makes the JMX Exporter expose the standard JVM/process metrics. The Kafka Instance dashboard mainly uses:

JMX: Broker Traffic

Broker totals have no topic dimension. Their JMX series count therefore does not grow with topic count. Protocol exporters provide topic offsets and progress.

JMX: Replication and Storage

Under Replicated means some replicas are not synchronized. Under Min ISR means write availability or durability has fallen below the configured minimum. At Min ISR still meets that minimum but has no remaining replica margin.

JMX: Request Path

For high latency, examine request volume, errors, P95/P99, both queues, thread idle ratios, GC, CPU, disk I/O, and network. A low idle ratio alone cannot locate the bottleneck.

JMX: KRaft and Broker Metadata

log_end_offset - high_watermark helps estimate metadata commit lag. Also check member role, leader, epoch, and controller event latency.

JMX: Controller

These MBeans exist only in Kafka processes that carry the controller role:

Healthy clusters should have exactly one active controller. Prioritize increases in offline_partition_count, metadata_errors_total, and unclean_leader_elections_total.

Recording Rule Metrics

Offset Progress

JVM and Broker

Cardinality and Interpretation Notes

• Do not directly sum multiple kafka_exporter results for one cls. They can duplicate cluster state.
• kafka_topic_partition_current_offset measures offsets, not exact bytes, requests, or business events.
• Consumer lag covers only visible groups with committed offsets.
• Controller-only nodes normally lack broker and protocol metrics. Unselected brokers normally lack protocol metrics.
• A JMX series is absent if its MBean does not exist for the Kafka version or role. Interpret this with role.
• The allowlist excludes per-client and per-partition JMX metrics to limit unpredictable series counts.

For how to use the dashboards and alerts, see Monitoring.

8 - FAQ

How mature is the current KAFKA module?

The v1 baseline provides dynamic KRaft, full-cluster checks, cold start and repair, serial broker admission, and dynamic controller join. It supports retirement of live or dead members and three-command failed-node replacement. It also provides strict rolling restarts, TLS/SCRAM/ACL, declared topics and users, protected credential and certificate rotation, and complete monitoring.

Production requires kafka_security: scram, an odd controller count, and sufficient brokers, RF, and minISR. You must plan capacity, reassignment, balancing, upgrades, backup, restore, and failure exercises. This module does not provide a managed Kafka service. Default plaintext suits only development or trusted isolated networks.

Why is there no ZooKeeper and no controller.quorum.voters?

The module targets Kafka 4.1+ with native dynamic KRaft. It installs no ZooKeeper and creates no static quorum. Every member renders controller.quorum.bootstrap.servers. New clusters format explicitly with --initial-controllers/--no-initial-controllers. After startup, the role verifies that initial controller Directory IDs entered live quorum.

The manifest records initial controller identities. After commissioning, live Raft quorum defines membership. kafka.yml adds controllers through observer catch-up and add-controller. Strict-subset kafka-rm.yml runs retire them through remove-controller. Edit the inventory and run the matching playbook.

What is the difference between combined, broker, and controller?

• combined: acts as both broker and controller, listening on 9092 and 9093; this is the default;
• broker: pure data plane, listening only on 9092;
• controller: pure control plane, listening only on 9093.

Omit all roles to use combined, or declare roles on every member. Legacy aliases are unavailable.

Will controller port 9093 collide with Alertmanager?

No. Alertmanager uses alertmanager_port 9059 and cluster port 9094. These do not conflict with KRaft 9093. If custom settings conflict, change kafka_controller_port. The role checks only that Kafka ports 9092, 9093, 9308, and 9404 differ. It does not detect conflicts with other services.

The service is up, but a remote client cannot connect?

advertised.listeners always uses inventory_hostname. After bootstrap, clients must resolve and reach every broker address returned in metadata.

Check in order:

grep '^advertised.listeners' /etc/kafka/server.properties
ss -lntp | grep ':9092'
getent hosts <inventory-hostname>

For scram, also check the CA, SASL mechanism, credentials, and ACLs. Version 1 lacks custom advertised addresses, multiple listeners, and NAT/public mapping. Clients must route directly to inventory_hostname. Raw listener overrides in kafka_parameters cannot bypass this requirement.

Why does it report a Cluster ID, Node ID, or Directory ID mismatch?

The role compares the manifest, ${kafka_data}/metadata/meta.properties, inventory, and live quorum. Common mismatch causes include:

• kafka_cluster or kafka_seq was changed;
• another cluster’s data disk was mounted on the current node;
• an incorrect kafka_cluster_id was given during restore/takeover;
• the controller data directory or directory ID does not match the live voter records;
• the wrong target cluster was selected, or a stale manifest was used.

This failure protects data. Do not delete meta.properties or the manifest, and do not run kafka-rm.yml. First verify data ownership, surviving replicas, actual Cluster/Node/Directory identities, and the recovery target.

What happens if the manifest is lost or only an old one remains?

Each member stores /etc/kafka/manifest.yml; scram members also store /etc/kafka/secrets.yml. Every run retrieves these from any member copy. Administration nodes keep no Kafka state, so replacing one or losing its checkout does not prevent management. Formatted storage without any manifest copy blocks execution until you restore a copy on one member. A formatted scram cluster also blocks execution if no member retains secrets. Certificates cached in files/pki/kafka/ are signed again by the Pigsty CA if missing.

An existing manifest with all Kafka data disks empty blocks execution. This prevents accidental reuse of a lost cluster’s identity. For an intended rebuild, first run kafka-rm.yml, then follow an explicit rebuild procedure.

Why are some keys in kafka_parameters rejected?

The role owns identity, quorum, listeners, storage, replication, rack, and security settings. Overriding any reserved key fails identity checks before writes. See kafka_parameters for the complete list.

Use the corresponding public parameters instead. The role provides no variables for advertised addresses, path subdirectories, the listener map, or exporter options.

How do I enable TLS, SCRAM, and ACLs?

Set on a new cluster:

kafka_security: scram

This enables Pigsty CA certificates, controller mTLS, broker/client SASL_SSL + SCRAM-SHA-512, StandardAuthorizer, and default denial together. Declare application passwords, ACLs, and optional quotas in kafka_users.

Security mode is a bootstrap-only setting. Ordinary playbooks cannot migrate formatted clusters online from plaintext to scram. That requires a separate migration state machine. Healthy scram clusters support protected internal credential or certificate rotation.

Do kafka_topics and kafka_users delete resources?

No. Removing an entry from the inventory never implicitly deletes a topic or a user.

Topics are created idempotently. Partitions only increase, declared settings alone change, and RF changes require explicit reassignment. Declared users receive their password, complete ACL set, and specified quota fields. Topic deletion, user deletion, and full privilege revocation remain separate audited operations.

What is the difference between the JMX Exporter and kafka_exporter?

JMX Exporter runs inside every Kafka JVM. It collects JVM, broker, replication, request-path, and KRaft metrics under job=kafka, with a role label.

kafka_exporter queries cluster, topic, partition, offset, consumer-group, and lag data through Kafka protocol. It uses job=kafka without a role label. It runs on the first two broker-capable nodes by kafka_seq. A single broker runs one; pure controllers run none.

The exporters provide complementary views. Lifecycle health checks use a separate Kafka CLI/metadata channel and depend on neither exporter.

Why does a particular broker or pure controller have no kafka_exporter?

This placement is expected. Protocol exporters report whole-cluster state rather than node metrics. Two copies avoid a single monitoring failure point while limiting duplicate scrape costs.

Check the current targets (one file per instance; the files of selected nodes contain the :9308 protocol exporter target):

ls -l /infra/targets/kafka/
grep 9308 /infra/targets/kafka/*.yml

Full runs refresh target files for current exporter placement. Do not run on one node only to register labels. If placement changes, ordinary playbooks do not stop the old kafka_exporter service. Remove it manually or with kafka-rm.yml.

Why is the JMX endpoint reachable but jmx_scrape_error=1?

HTTP access proves only that the Java agent loaded. jmx_scrape_error=1 means the latest MBean scrape failed:

journalctl -u kafka --since '-30 min' --no-pager
curl -fsS http://<kafka-ip>:9404/metrics | head -n 40

Check /etc/kafka/jmx_exporter.yml against installed Kafka and JMX Exporter versions. Confirm that startDelaySeconds has elapsed. Successful startup requires jmx_scrape_error 0.0, JVM metrics, and at least one role-appropriate kafka_ metric.

Why is there no Consumer Lag data?

Possible causes include consumers without groups, no committed Kafka offsets, external offset storage, or groups that have not consumed the topic. Exporter TLS, SCRAM, ACL, and network problems can also cause missing metrics.

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group <group>

Check kafka_exporter_up, exporter logs, dashboard variables, and raw kafka_consumergroup_* metrics. Use native Prometheus up to assess endpoint availability. Custom metrics can briefly remain after scrape failure.

Why can’t the cluster metrics from two kafka_exporters be added together?

Both exporters can return identical topic, partition, and consumer-group state. Summing them counts data twice. kafka:cls:* rules deduplicate exporter copies before cluster aggregation.

Should applications go through HAProxy, a Keepalived VIP, or an LB?

No. Producers and consumers fetch metadata from a bootstrap.servers seed, then connect directly to partition leaders. A VIP or TCP load balancer cannot identify leaders or rewrite broker addresses. Adding it to the data plane adds long-lived connection state, another failure point, and diagnostic complexity.

DNS or a TCP load balancer can provide a single bootstrap discovery endpoint if required. advertised.listeners must still return reachable addresses for every broker. NAT, public internet, multiple networks, and Kubernetes exposure require dedicated external addresses and extra listeners per broker. The module advertises inventory addresses only and does not support this mapping.

See Quickstart: why applications should connect directly to multiple brokers and Cluster Config: network and listeners.

Can I just add or remove a broker or controller?

Yes. Edit the inventory and use playbooks for each KRaft membership change:

• Add: Declare a broker, combined, or controller member. Run ./kafka.yml -l <cls> on the complete cluster, not just the new node. Pure brokers format, start, and verify registration one at a time. Combined/controller members format with --no-initial-controllers, catch up as observers, and promote through add-controller. Every step checks health.
• Remove: Run ./kafka-rm.yml -l <ip> on a strict subset. A surviving member performs remove-controller and broker unregistration, even if the target is unreachable. Then remove the inventory entry.

Plan each membership change. Retain an odd controller count with a live majority. Change one member at a time. Drain departing broker partitions first, or replace it with the same kafka_seq. New brokers do not receive existing partitions automatically. Run and monitor reassignment separately; registration does not imply balanced capacity.

Which parameter controls the package version?

The role uses package_map['java-runtime'] and package_map['kafka-stack']. It has no kafka_version, scala_version, or exporter version parameters. Target-platform repositories and installed packages determine versions.

The payload verified on 2026-07-16 contained Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0. Before upgrades, separately review compatibility, backup, rollback, rolling order, and feature level. Replacing packages alone is insufficient.

How do I safely wipe Kafka data?

kafka.yml never performs cleanup. kafka-rm.yml owns deletion. Selecting a whole cluster with -l, or all clusters without it, performs teardown. A strict subset performs retirement. Default kafka_rm_data=true permanently deletes data, KRaft metadata, node-local /etc/kafka recovery state, and monitoring targets. kafka_rm_data=false retains data and recovery state; kafka_safeguard=true blocks all deletion.

Removal commands execute immediately without an additional confirmation string. Before running, verify the exact -l target, recoverable backups or explicit rebuild intent, and business decommissioning. Retirement tolerates broker unregistration failure. After execution, inspect quorum, registrations, and replica health. See Playbook: kafka-rm.yml.