Quick Start

This tutorial starts from a minimal single-node cluster, walking through topic creation and message read/write; it then deploys a separate, secured three-node cluster with application users, ACLs, quotas, and production topics; finally, it demonstrates core parameter changes, client access, monitoring verification, and pre-launch checks.

Learning Path

Before You Begin

Unless noted otherwise, the following commands are run from the project directory on the Pigsty admin node:

cd ~/pigsty

Before you start, confirm that:

• pigsty.yml is the source of configuration for the current environment — back it up and review its existing contents first;
• each Kafka node’s inventory_hostname can be resolved and routed directly by every Kafka member and client;
• the admin node and the Kafka nodes have synchronized clocks;
• ports 9092, 9093, 9308, and 9404 are free of conflicts;
• /data/kafka maps to a dedicated data disk or directory that holds nothing else;
• every kafka.yml run uses -l to select exactly all members of one Kafka cluster;
• before any real change, you run --check first, review the output, and obtain approval for the change.

The inventory must keep the all.children hierarchy. Merge the groups below into your existing pigsty.yml; do not let the examples overwrite your existing all.vars, infra, etcd, pgsql, or other configuration.

Part 1: Deploy a Single-Node Kafka

1. Define the Cluster

Add the following kf-dev group to all.children. This node omits kafka_role, so it uses the default combined and serves as both broker and controller:

all:
  children:
    # keep your existing infra, etcd, pgsql, and other groups

    kf-dev:
      hosts:
        10.10.10.10: { kafka_seq: 1 }
      vars:
        kafka_cluster: kf-dev
        kafka_data: /data/kafka
        kafka_security: plaintext
        kafka_topics:
          - name: quickstart.events
            partitions: 1
            replication_factor: 1
            config:
              retention.ms: 86400000   # 1 day, tutorial only

This configuration yields:

• a random cluster ID;
• a single dynamic KRaft combined node;
• default RF=1 and minISR=1;
• a single-partition topic named quickstart.events;
• a JMX exporter on :9404 and a protocol exporter on :9308.

plaintext has no transport encryption, authentication, or ACLs, so use it only for local development or a trusted, isolated network.

2. Bring the Node Under Management

If this host has not yet been initialized by NODE, run check mode first:

./node.yml --check -l kf-dev

After reviewing the results and getting approval, bring the node under management:

./node.yml -l kf-dev

You can skip this step for a node that Pigsty already manages and whose package repository and time synchronization are healthy. For full NODE preparation and day-to-day management, see Node Administration.

3. Deploy Kafka

Run a check against the full cluster first:

./kafka.yml --check -l kf-dev

Confirm the target is exactly the full membership of kf-dev, review the data path, packages, ports, and configuration changes, then run:

./kafka.yml -l kf-dev

The role installs Java and kafka-stack, generates a random identity and bootstrap manifest, formats the KRaft storage, starts the service, creates the topics, and registers the monitoring targets.

4. Verify the Service and Quorum

Log in to the Kafka node and check the services:

systemctl is-active kafka kafka_exporter
journalctl -u kafka --since '-10 min' --no-pager

Run the role’s own health check:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties

The returned JSON should contain "healthy": true. Continue by checking the dynamic quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

You should see a valid LeaderId, a CurrentVoters list that includes this node, and quickstart.events with RF=1 and ISR=1.

5. Produce and Consume Messages

Start a console producer:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events

Type a few lines of messages, then press Ctrl-D to finish. Consume them from another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

At this point, the single-node deployment, topic convergence, and message read/write are complete. For further status checks, see Day-to-Day Administration.

Part 2: Deploy a Three-Node Production Baseline

The three-node example is a brand-new kf-main cluster built from three combined nodes. It can tolerate the loss of one controller; business topics use RF=3/minISR=2, and the scram production security profile is enabled.

1. Define the Secured Cluster and Its Resources

Add the following group to your existing all.children:

all:
  children:
    # keep your existing groups

    kf-main:
      hosts:
        10.10.10.11: { kafka_seq: 1 }
        10.10.10.12: { kafka_seq: 2 }
        10.10.10.13: { kafka_seq: 3 }
      vars:
        kafka_cluster: kf-main
        kafka_data: /data/kafka
        kafka_heap_opts: '-Xms4G -Xmx4G'
        kafka_security: scram

        kafka_parameters:
          num.partitions: 12
          num.network.threads: 6
          num.io.threads: 16
          log.retention.hours: 168
          log.segment.bytes: 1073741824

        kafka_users:
          - name: quickstart-app
            password: "{{ vault_kafka_quickstart_password }}"
            acls:
              - resource: topic
                name: quickstart.
                pattern: prefixed
                operations: [Read, Write, Describe]
              - resource: group
                name: quickstart.
                pattern: prefixed
                operations: [Read]
              - resource: cluster
                name: kafka-cluster
                operations: [Describe, IdempotentWrite]
            quota:
              producer_byte_rate: 10485760
              consumer_byte_rate: 20971520

        kafka_topics:
          - name: quickstart.events
            partitions: 12
            replication_factor: 3
            config:
              min.insync.replicas: 2
              cleanup.policy: delete
              retention.ms: 604800000

vault_kafka_quickstart_password must be supplied by your existing Ansible Vault, KMS, or another secret-injection mechanism, and must be at least 12 characters. Never commit a real password directly to Git, logs, or tickets.

The key semantics of this configuration:

• all three nodes omit kafka_role, so they consistently use combined;
• the new cluster is bootstrapped directly as dynamic KRaft;
• scram enables node TLS, controller mTLS, SCRAM-SHA-512, ACLs, and deny-by-default all at once;
• with three brokers, the initial replication policy is automatically derived as RF=3 and minISR=2;
• quickstart.events is created explicitly with 12 partitions and 3 replicas;
• quickstart-app can read and write quickstart.* topics, read quickstart.* groups, and use the idempotent producer;
• at most two brokers run kafka_exporter, while all three Kafka JVMs run the JMX exporter.

If the three brokers truly reside in different failure domains, you may add kafka_rack: az-a/az-b/az-c to all nodes respectively. Do not use fictitious rack labels to manufacture a disaster-recovery guarantee that does not exist; for the detailed rules, see Cluster Configuration: Rack.

2. Bring Under Management and Deploy

If the nodes are not yet under management:

./node.yml --check -l kf-main
./node.yml -l kf-main

When deploying Kafka, you must select all three members:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

You cannot use -l 10.10.10.11 alone, nor select kf-dev,kf-main at once. The role rejects a missing, partial, or cross-cluster limit.

3. Verify the Health of All Three Nodes

From the admin node, check the three Kafka services:

ansible kf-main -b -m command -a 'systemctl is-active kafka'

Run the full health check on any broker:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties

Query the quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

Before launch, you should see one active controller, three current voters, and three available brokers; every quickstart.events partition has three replicas with ISR=3, and there are no offline, under-replicated, or under-min-ISR partitions.

Part 3: Connect Application Clients

1. Distribute the CA Public Certificate

Securely copy the public CA certificate from the admin node to the application host:

files/pki/ca/ca.crt  ->  /etc/kafka-client/pigsty-ca.crt

ca.crt is a public certificate that is safe to distribute. Never copy, expose, or distribute files/pki/ca/ca.key. On the application host, the CA file should be owned by root and set read-only. An application host already managed by Pigsty needs no copy: the NODE module has installed the same CA at /etc/pki/ca.crt, which the client can reference directly.

2. Create the Client Configuration

On the application host, create /etc/kafka-client/client.properties:

bootstrap.servers=10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="quickstart-app" password="<secret-from-vault>";

ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka-client/pigsty-ca.crt
ssl.endpoint.identification.algorithm=https

The Kafka Java client supports SASL_SSL + SCRAM and a PEM truststore. A real application should inject the password from a secret manager at runtime rather than committing a file containing the password to the repository. For the complete set of fields, see the Kafka 4.3 SASL/SCRAM and producer configuration references.

3. Why Applications Should Connect Directly to Multiple Brokers

The Kafka client is itself a cluster-aware, intelligent client. bootstrap.servers is used only to obtain the initial metadata; once connected, the client uses that metadata to connect directly to the leader broker of each partition and refreshes its routing whenever a leader changes. The standard practice in production is therefore to:

• configure at least two, and usually three, broker addresses in bootstrap.servers, located in different failure domains;
• allow the application to reach 9092 on all brokers, and ensure that each broker’s advertised inventory_hostname is resolvable and routable;
• let producers and consumers rely on the Kafka client’s own retry, metadata-refresh, idempotence, and consumer-group protocols;
• not place HAProxy, a Keepalived VIP, a layer-4 load balancer, or a layer-7 reverse proxy in front of the Kafka data plane.

A single VIP or LB cannot substitute for the broker addresses in Kafka’s metadata, nor can it transparently forward a connection to the correct partition leader; it also adds complexity around long-lived connection state, fault localization, and capacity planning. If your platform must provide a unified discovery entry point, you can use a DNS name or a TCP load balancer as a bootstrap-only entry point, but each broker’s advertised.listeners must still return an address that the client can reach directly, and the application must not be permitted to reach only the LB.

Scenarios that span NAT, the public internet, Kubernetes, or multiple networks usually require a dedicated externally reachable address/port and an additional listener for each broker. The current module fixes the inventory address as advertised.listeners and does not support arbitrary listener mappings of this kind; do not use a single HAProxy VIP to paper over an address model that does not hold.

4. Authenticate and Read/Write with the Application Identity

On an application host with the Kafka 4.3 CLI installed, run:

kafka-console-producer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events

When consuming, use a group prefix permitted by the ACL:

kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

A production application should also explicitly review its client semantics:

Client-side acks, retries, idempotence, batching, compression, and offset strategy are application configuration and should not be written into the broker’s kafka_parameters.

Part 4: Change Core Parameters

Always express persistent intent for Kafka by editing pigsty.yml, never by editing /etc/kafka/server.properties directly. Common intents map as follows:

Example: Adjust the Heap and Broker Default Parameters

Suppose that after load testing you decide to raise the heap to 6G, increase the thread counts, and change the default retention for new topics to 72 hours:

kf-main:
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms6G -Xmx6G'
    kafka_parameters:
      num.partitions: 12
      num.network.threads: 8
      num.io.threads: 24
      log.retention.hours: 72
      log.segment.bytes: 1073741824

Do not copy 6G/8/24 verbatim; these values must be determined by load-testing against your CPU, memory, connection count, message size, partition count, disk, and page cache.

Example: Add Partitions and Shorten Topic Retention

Increase quickstart.events from 12 partitions to 24 and change the retention to three days:

kafka_topics:
  - name: quickstart.events
    partitions: 24
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 259200000

Partitions cannot be reduced. When replication_factor differs from what is live, the role refuses ordinary convergence and requires an explicit partition reassignment; it never relocates existing replicas automatically.

Apply the Change

Whether you change static parameters or dynamic resources, run the full state machine:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Do not run -t kafka_config alone. The role decides automatically: a static change triggers a strict node-by-node rolling restart; when only dynamic resources such as topics or users change, Kafka is not restarted.

The following keys belong to the role itself and must not be placed in kafka_parameters:

kafka_parameters:
  min.insync.replicas: 2          # wrong: owned by the role
  default.replication.factor: 3   # wrong: owned by the role
  listeners: ...                  # wrong: owned by the role

For all 15 public parameters, their defaults, and the reserved keys, see the Parameter Reference.

Part 5: Key Pre-Launch Checks

Topology and Data Safety

• Production uses at least three brokers and an odd number of controllers; for critical or large clusters, consider a separated 3-controller + N-broker topology;
• topic RF, minISR, and the producer’s acks form a consistent failure model;
• kafka_rack expresses only real failure domains, and replica placement has been verified;
• data-disk capacity, throughput, latency, retention, peak write rate, and recovery time have been load-tested;
• there is an explicit reassignment plan for after a new broker joins, since existing topics’ RF is not raised automatically;
• the procedures for Kafka data backup/rebuild, broker replacement, controller membership, and disaster recovery are all defined.

Security and Network

• a new production cluster uses kafka_security: scram from bootstrap onward;
• application passwords are injected by Vault/KMS/a secret manager and never reach Git or logs;
• only the CA public certificate is distributed to clients, never the CA private key;
• clients can resolve and reach the inventory_hostname of every broker directly;
• 9092/9093 are open only to the principals that need them, and 9308/9404 only to the monitoring network;
• a review checklist for application principals, topic/group/cluster ACLs, and quotas is in place;
• a protected rotation of internal credentials and certificates is scheduled.

Operations and Monitoring

• /usr/local/bin/pigsty-kafka-health cluster reports healthy;
• the dynamic quorum has exactly one leader, and every expected controller is in the current voters;
• there are no offline, under-replicated, or under-min-ISR partitions;
• produce/consume verification is done over the real application network with a real principal;
• the Kafka Overview, Kafka Instance, and Kafka Node dashboards show healthy data;
• alert routing, log search, capacity thresholds, on-call responsibilities, and rollback conditions are confirmed;
• upgrades, feature-level changes, topic deletion, user deletion, and cluster teardown each have a separate approval process.

For detailed alerts and PromQL, see Monitoring and Alerting; for metric semantics, see Metric Definitions.

Documentation Index and Next Steps

We recommend continuing along the following path:

One recommended reading path is: Quick Start → Cluster Configuration → Parameter Reference → Playbook → Day-to-Day Administration → Monitoring and Alerting → FAQ.