Parameters

The KAFKA role deliberately exposes only 15 persistent parameters. Details such as topology, listeners, security implementation, storage subdirectories, replication safety, and Exporter placement are derived by the role in a single, consistent way, and cannot be overridden as additional persistent variables.

Parameter Overview

kafka_cluster and kafka_seq must be defined; kafka_role has a real default. The cluster’s roles are either all omitted or all declared explicitly.

Identity and Topology

kafka_cluster

The required cluster identity. It must start with a letter or digit and contain only letters, digits, underscores, and hyphens:

kafka_cluster: kf-main

It is used to discover the complete cluster membership, generate instance names, and locate the bootstrap manifest. Every kafka.yml lifecycle operation must use a precise -l to select all members of this cluster.

kafka_seq

A required non-negative integer, unique within the same kafka_cluster, which becomes the KRaft node.id directly:

10.10.10.11: { kafka_seq: 1 }

The instance name is derived as ${kafka_cluster}-${kafka_seq}. Once a node has been formatted, do not change or reuse a sequence number that still has associated data.

kafka_role

Defaults to combined, and accepts only:

When all cluster members omit it, they consistently use combined; as soon as any member sets it explicitly, all members must set it explicitly. No legacy role aliases are provided.

kafka_cluster_id

Unset by default, used only to assert the identity of an existing cluster during takeover or recovery. It must be a 22-character Kafka UUID:

kafka_cluster_id: MkU3OEVBNTcwNTJENDM2Qk

Do not set it for an ordinary new cluster. The role generates the Cluster ID randomly and writes it into files/kafka/<kafka_cluster>/manifest.yml. This parameter does not relabel existing data; it fails closed when it conflicts with the manifest or meta.properties.

kafka_rack

An optional broker fault-domain label, rendered as broker.rack:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }

All broker-capable nodes must either all declare it or all omit it. Controller-only nodes do not use this value. Changing the rack is a static change that goes through a strict rolling restart, but does not reassign existing replicas.

Storage, JVM, and Network

kafka_data

The data root directory, defaulting to /data/kafka:

kafka_data: /data/kafka

The role derives ${kafka_data}/data and ${kafka_data}/metadata in a fixed way. This path must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg. kafka-rm.yml deletes the entire root directory by default, so do not mix other services or business files into it.

kafka_heap_opts

The Kafka JVM heap, defaulting to:

kafka_heap_opts: '-Xms1G -Xmx1G'

In production, set it according to load and memory load testing. Typically keep Xms and Xmx equal, and leave enough memory for the operating-system page cache and other processes.

kafka_port

The broker/client listener port, defaulting to 9092, and listening only on broker-capable nodes. plaintext mode uses PLAINTEXT; scram mode uses SASL_SSL + SCRAM-SHA-512.

kafka_controller_port

The KRaft controller listener port, defaulting to 9093 (the conventional Kafka KRaft port), and listening only on controller-capable nodes. When sharing a node with other services, verify yourself that the ports do not collide; the role does not automatically detect cross-service port usage.

The four public ports must all differ from one another. The broker listener binds 0.0.0.0, while the controller listener, the broker advertised address, and the controller bootstrap address all use inventory_hostname in a fixed way, with no separate address parameters.

kafka_parameters

Defaults to {}. It is the only Kafka broker-parameter escape hatch, rendered only onto broker-capable nodes:

kafka_parameters:
  num.partitions: 12
  num.network.threads: 6
  num.io.threads: 16
  log.retention.hours: 168
  log.segment.bytes: 1073741824

The following keys or patterns are owned by the role and cannot be overridden through this mapping:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

Identity, listeners, security, storage, and replication policy must remain single-authority; when a reserved key is included, the preflight fails outright.

Observability

kafka_jmx_exporter_port

The JMX Exporter HTTP port, defaulting to 9404. The role injects the JMX Exporter Java agent unconditionally into every Kafka JVM and registers it as job=kafka; there is no separate toggle parameter. The lifecycle health gate uses the role-owned Kafka CLI/metadata channel and does not depend on JMX. Infra monitoring nodes must be able to reach this port; the endpoint does not automatically enable HTTPS because of kafka_security: scram, so it should be protected by the monitoring network and firewall.

kafka_exporter_port

The HTTP port of the protocol-level kafka_exporter, defaulting to 9308. The role configures, starts, and registers it only on the first two broker-capable nodes after sorting by kafka_seq; a single-broker cluster runs only one. The monitoring target file is refreshed on every full run according to the current placement, but a stale Exporter service on a node that was previously selected is not stopped automatically by an ordinary playbook.

The Kafka protocol version, TLS/SCRAM parameters, and replica placement used by the Exporter are all internal role conventions, with no additional public toggles or options parameters.

Security and Resources

kafka_security

Defaults to plaintext, and accepts only:

scram simultaneously configures the Pigsty CA-issued node certificates, the role-owned admin/monitoring/internal identities, and the ordering in which TLS/SCRAM and ACLs are enabled. The security profile is written into the bootstrap manifest; once the cluster is formatted, an ordinary rerun can neither switch plaintext to scram nor switch back.

kafka_users

Defaults to [], and may only be declared in scram mode. Each object accepts only name, password, acls, and quota:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
      - resource: transactional_id
        name: order.
        pattern: prefixed
        operations: [Write, Describe]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

Constraints:

• name is unique within the list; password is required, must be at least 12 characters, and should reference a secret management system;
• ACL resource is one of topic, group, transactional_id, cluster;
• pattern is literal (default) or prefixed;
• Operations are Read, Write, Create, Delete, Alter, Describe, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite;
• Quota keys are producer_byte_rate, consumer_byte_rate, request_percentage, controller_mutation_rate.

The role converges the SCRAM password, the complete ACL set, and the explicitly provided quota fields for a declared user. Removing a user entry does not implicitly delete the principal or credentials; deletion/revocation requires a separate, audited operation.

kafka_topics

Defaults to []. Each object accepts only name, partitions, replication_factor, and config:

kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 604800000

name is unique within the list; both partitions and RF must be at least 1, and RF cannot exceed the current broker count. The convergence semantics are:

• The topic is created idempotently when it does not exist;
• Partitions may only increase; decreasing fails;
• When RF differs from the live state, ordinary convergence is refused and an explicit reassignment is required;
• Only the keys declared in config are updated;
• Removing a topic from the list never deletes the topic.

Transient Protected Operational Variables

The following variables are used only via the command-line -e for one-off operational actions. They are not part of the 15 persistent API parameters and should not be written into pigsty.yml:

The two rotation actions are mutually exclusive, and must target a precise, complete cluster. kafka-rm.yml deletes the data directory and the Manifest/Secret/PKI cache on the admin node by default, so before running it you must explicitly confirm the target cluster and your backup/rebuild intent. For the commands and full semantics, see Playbooks.