Playbook

kafka.yml deploys Apache Kafka 4.1+ dynamic KRaft and applies security, resources, and monitoring. kafka-rm.yml removes members or tears down clusters.

kafka.yml

./kafka.yml --check -l kf-main   # dry run first
./kafka.yml -l kf-main           # create or converge one cluster
./kafka.yml                      # bare run: create / converge ALL kafka clusters

Every selected cluster must be complete. Select one cluster, several clusters, or the whole inventory without a limit. Execution is serial within each cluster and concurrent across clusters. Partial cluster selection is rejected.

Check mode validates the API, cluster completeness, roles, racks, ports, manifest, and inspectable file changes. It skips formatting, startup, and live acceptance checks. Passing --check does not guarantee runtime success.

Execution Stages

A single kafka.yml play runs node_id, then kafka, like pgsql.yml. The role has six stages. Launch controls cross-node ordering: parallel bootstrap, serial controller join, serial broker admission, and strict rolling restarts.

any_errors_fatal: true stops further dangerous work after a stage failure. Fix the cause, then rerun on the full cluster. The role uses live state and persistent fingerprints to resume without blindly reformatting.

Lifecycle Paths

Configuration uses the role’s administration channel to assess health and select one execution path:

Cold Start, First Deployment, or Repair

Stopped or unhealthy clusters enter convergence:

1. Start controller-capable nodes.
2. Wait for the controller listener and a quorum leader.
3. At first bootstrap, verify that initial controller Directory IDs entered live quorum.
4. Start pure brokers.
5. Wait for broker listeners and full cluster health.
6. Save the static fingerprint only after proving that the configuration runs successfully.

Startup, admission, and rolling decisions use only the role’s Kafka CLI/metadata administration channel. JMX does not participate.

Adding Brokers or Controllers to a Healthy Cluster

Newly formatted kafka_role: broker nodes enter through admit, one at a time. Each must start, register, and become unfenced before the next proceeds.

New combined/controller nodes use serial join. A commissioned cluster formats each with --no-initial-controllers, starts it as an observer, and waits for metadata catch-up. The role then promotes it with add-controller and verifies voter membership and full health. Interrupted runs resume from live state. If node.id retains a dead predecessor’s voter entry, configuration fails with the exact kafka-rm.yml retirement command.

Admission and join prove membership only. They do not move existing partitions to new brokers. Run explicit reassignment separately.

Static Changes on a Healthy Cluster

When the rendered static fingerprint changes, the strict rolling restart handles one node at a time:

• Before restart, check controller majority, zero voter lag, recent catch-up, offline partitions, under-replication, and under-min-ISR. Also check effective partition ISR with the target excluded.
• After restart, require controller voter membership and catch-up, broker registration and unfenced state, and replica reentry into ISR.
• Any failed check immediately stops work on remaining nodes.

When repairs and static changes coincide, convergence starts only stopped members. It does not restart online members in parallel. After quorum recovery and catch-up, strict rolling restarts apply remaining static changes.

An unchanged static fingerprint causes no restart. Dynamic resources still update online during resource convergence.

Task Tags

Run full kafka.yml for ordinary configuration changes. Let the role select the lifecycle path. Tags mainly support development, diagnostics, and controlled repair. Do not bypass the state machine with -t kafka_config or single-node limits.

Identity, Formatting, and Manifest

Before writing configuration, the role verifies:

• Every selected cluster includes all members.
• kafka_seq is unique; roles are all omitted or all explicit.
• At least one controller and one broker exist.
• All broker-capable nodes have racks, or none do.
• Ports are valid and distinct; kafka_parameters does not override role-managed keys.
• The manifest, security profile, meta.properties, and live identity agree.

New clusters generate random Cluster and initial controller Directory IDs, then format nodes explicitly for dynamic quorum. Existing ${kafka_data}/metadata/meta.properties triggers local Cluster/Node ID validation. Only first bootstrap compares initial controller Directory IDs with live quorum. After commissioning, live Raft state controls membership. The role never reformats existing storage automatically.

The authoritative bootstrap manifest lives on every cluster member:

/etc/kafka/manifest.yml

scram members also store /etc/kafka/secrets.yml. The administration node keeps no Kafka state and retrieves both files from member copies each run. Live state controls runtime facts. On conflict, ordinary playbooks do not silently overwrite either side:

• Formatted storage without a member manifest copy blocks execution. Restore the file on any member first.
• A manifest with all data disks empty blocks execution.
• Conflicting Cluster ID, security profile, or controller identity blocks execution.
• A new node with a stale predecessor voter entry for its node.id must first be retired with kafka-rm.yml.

Do not delete meta.properties, the manifest, or the secrets to bypass these protections.

Static Fingerprint and Recoverable Re-runs

The role fingerprints static files that affect Kafka. It writes /etc/kafka/.pigsty-applied-static.sha256 only after one of these conditions:

• Convergence starts the cluster successfully and passes global health checks.
• Strict rolling restarts the node, completes catch-up, and passes post-restart checks.

Interrupted runs do not mark unverified changes as applied. The next full run still detects the pending static restart.

Resource Convergence and Monitoring Registration

After full cluster health, resource and monitoring stages run in order:

1. Apply role-managed dynamic cluster min-ISR.
2. Idempotently apply kafka_users credentials, ACLs, and declared quotas.
3. Idempotently create kafka_topics, increase partitions, and apply declared settings.
4. Check internal-topic RF drift without automatic reassignment.
5. Start protocol exporters on the first two broker-capable nodes by kafka_seq.
6. Refresh discovery targets on all INFRA nodes.

Each instance has one target file. It contains JMX and any selected protocol target under the kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

Full runs refresh targets for current exporter placement. kafka-rm.yml deregistration deletes targets.

Protected Rotation

Rotation variables are one-time extra-vars; do not put them in pigsty.yml. Run only one of the mutually exclusive actions. Every member must be formatted. The cluster must be healthy, use scram, and retain role-managed secrets. kafka_rotate_confirm must exactly match the cluster name.

Internal Credential Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

The role maintains active and standby internal identities. It updates the inactive credential through the live administration channel, then atomically switches the protected local record. A strict rolling restart follows. The old active identity becomes standby for the next rotation. Reruns can therefore recover after interruption.

Certificate Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

The role removes issued node certificates from shared PKI. The same Pigsty CA issues new node keys and certificates. It updates node PEM bundles, then performs strict rolling restarts. Old and new certificates share trust, so staged CA changes are unnecessary. If the health precheck fails, rotation never starts and deployed certificates stay unchanged.

kafka-rm.yml

Removal uses separate kafka-rm.yml, never kafka.yml. It requires non-empty -l/--limit and fails before roles if omitted. Selecting all members with -l tears down a cluster. A strict subset retires members. Both follow this order:

1. Deregister VictoriaMetrics targets (kafka_deregister).
2. Stop and disable kafka and kafka_exporter (kafka).
3. Remove voter entries and broker registrations through survivors (kafka_retire). Survivors exist only for strict-subset selection.
4. Delete exporter configuration, Systemd environments/units, and helpers (kafka_config).
5. Delete data and node-local /etc/kafka recovery state (kafka_data), controlled by kafka_rm_data.
6. Optionally uninstall packages (kafka_pkg), controlled by kafka_rm_pkg.

Before deregistration or shutdown, the role validates a dedicated absolute kafka_data path. It must contain no ./.. segments. Forbidden roots are /, /data, /var, /etc, /opt, /usr, /home, /root, and /pg. kafka_safeguard set to true in inventory or the command line immediately stops removal without deletion. Identity conflicts, exporter errors, and ordinary startup failures do not justify data deletion. First run kafka.yml and read the failure reason.

Cluster Teardown

./kafka-rm.yml -l kf-main                          # Remove the cluster: deregister monitoring and stop services; deletes data and /etc/kafka recovery state by default
./kafka-rm.yml -l kf-main -e kafka_rm_data=false   # Keep data and /etc/kafka recovery state; remove only service integration
./kafka-rm.yml -l kf-main -e kafka_rm_pkg=true     # Also uninstall the kafka-stack packages (the shared Java runtime is not removed)

Member Retirement

./kafka-rm.yml -l 10.10.10.13                      # Retire one member: drop its voter entry and broker registration, then clean the node

Partial retirement needs a broker-capable (combined/broker) survivor and a controller-capable (combined/controller) survivor outside -l. One combined node can satisfy both. Missing either stops execution before deregistration or shutdown. Survivors run remove-controller, strictly serializing multiple targets, then broker unregister, before local cleanup. Delegation also handles dead, unreachable targets: step one of Replace Failed Node.

Broker unregistration tolerates failure for repeatable execution. After a real run, verify live quorum, broker registrations, and replica health. Playbook success alone does not prove retirement.

Plan retirement even with automation. Remaining controllers should have an odd count and live majority. Remaining brokers must support the highest topic RF. The playbook warns if departing brokers still hold replicas. For planned shrink, drain them through reassignment first.

Playbook Boundaries

Neither playbook reassigns partitions, balances data, deletes topics/users, or migrates online from plaintext to scram. They do not upgrade versions, finalize features, back up data, or perform disaster recovery. They do not deploy Connect, Schema Registry, MirrorMaker, or Cruise Control. See Module Boundaries and Administration.