Monitoring

Kafka monitoring combines metrics, logs, dashboards, and alerts. It covers JVM internals and the Kafka protocol view. You can inspect partitions, ISR, and consumer lag together with JVM state, request queues, and KRaft controller health.

Scrape Architecture

The KAFKA module uses two complementary exporters:

On each INFRA node, the role creates one discovery file per instance. JMX and any selected protocol target share that file and kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

Single-broker clusters run one protocol exporter; larger clusters run at most two. Controllers register only JMX. Controllers and unselected brokers have no protocol target by design. Full runs refresh target files for current placement. kafka-rm.yml deregistration removes instance targets.

Label Model

Both targets use job=kafka. JMX has a role label; protocol targets do not.

JMX Target

Protocol Exporter Target

Protocol targets carry cls, ins, ip, and instance (10.10.10.11:9308), without role or node_id. vmagent-side rules distinguish availability by those labels. kafka_up is up{job="kafka",role=~".+"}. kafka_exporter_up is up{job="kafka",role=""}.

Each exporter queries the whole cluster through a broker. Both can return identical topic, partition, and consumer-group state. Recording rules deduplicate exporters before aggregating cluster rates. In scram mode, the role generates exporter TLS/SCRAM settings from its monitoring identity.

Grafana Dashboards

Pigsty ships four complementary dashboards:

Kafka Overview

cls=All shows all Kafka clusters. Selecting one cls changes the same panels to that cluster’s overview.

Main content:

• Inventory of clusters, brokers, topics, partitions, and consumer groups
• Broker availability, exporter health, and cluster workload
• Leaderless, under-replicated, ISR deficit, and non-preferred replica
• Topic offset progress, consumer commit progress, and total lag
• Consumer group members, lag ranking, and topic/group drill-down
• Kafka/exporter log volume, firing alerts, and log detail

Common variables: cls, members, topic, group, topk.

Kafka Instance

Use ins to select a broker or controller JVM, including controller-only nodes. The dashboard also shows related host resources.

Main content:

• Instance identity, role, JMX availability, and scrape quality
• JVM heap, GC, threads, buffer pool, CPU, FD, and uptime
• Broker throughput, replication state, request errors/latency/queues, and handler/network idle
• KRaft member state, metadata log, controller health, and event latency
• Node CPU/memory, disk I/O, network, filesystem, and Kafka logs

Common variables: cls, ins, ip.

Kafka Topic

Use cls and topic to select a logical topic and inspect its topic/partition protocol state.

Main content:

• Topic and partition inventory, leader, replicas, ISR, and preferred leader
• Current offset, retention span, and message append rate
• Leaderless, ISR deficit, and non-preferred replica
• Associated consumer groups, commit progress, and lag

Common variables: cls, topic, topk.

Kafka Consumer

Use cls and group to inspect consumer-group members, committed offsets, progress, and backlog.

Main content:

• Consumer group inventory and member count
• Committed offsets by group/topic/partition
• Commit rate, total lag, maximum partition lag, and backlog trend
• Drill-down from group to topic/partition

Common variables: cls, group, topic, topk.

Choosing a Dashboard

Recording Rules

The Kafka rule file lives at /infra/rules/kafka.yml. The main recorded metrics are:

Offset-change rates measure progress, not client requests. Truncation, offset rollback, and exporter restarts can briefly produce negative changes. Rules use clamp_min(..., 0) to retain forward progress only.

Alert Rules

An unclean leader election can indicate data loss. Immediately preserve controller and broker logs. Identify affected topics and replicas before choosing recovery actions.

Common PromQL

Check scrape targets:

kafka_up
kafka_exporter_up
up{job="kafka"}

Check the replication health of a cluster:

sum by (cls) (kafka_server_replica_manager_under_replicated_partitions{job="kafka"})
sum by (cls) (kafka_server_replica_manager_under_min_isr_partitions{job="kafka"})
max by (cls) (kafka_controller_offline_partition_count{job="kafka"})

Check consumer lag:

topk(20, kafka_consumergroup_lag_sum{cls="kf-main"})

Check request saturation and latency:

kafka_server_request_handler_idle_ratio{job="kafka",cls="kf-main"}
max by (ins,request,quantile) (
  kafka_network_request_total_time_seconds{job="kafka",cls="kf-main",quantile=~"0.95|0.99"}
)

Log Queries

Kafka writes stdout and stderr to Journald. Vector forwards these records to VictoriaLogs under job:syslog.

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter
ip:10.10.10.11 job:syslog (unit:kafka OR app:kafka)

Kafka Instance log panels use similar queries and show time, level, Systemd unit, and message. Compare the same time window across logs, KRaft, ISR, request queues, GC, disk I/O, and network metrics.

Verifying the Monitoring Chain

Verify the raw endpoints on a Kafka node:

curl -fsS http://<kafka-ip>:9404/metrics | grep '^jmx_scrape_error'
curl -fsS http://127.0.0.1:9308/metrics | grep '^kafka_brokers'

Check discovery files on an INFRA node. Each instance has one file; selected nodes include both JMX and protocol targets:

ls -l /infra/targets/kafka/
cat /infra/targets/kafka/kf-main-1.yml

Query up{job="kafka"} in VictoriaMetrics, or recording metrics kafka_up and kafka_exporter_up. Use native Prometheus up for endpoint availability; custom exporter samples can remain briefly after failed scrapes. If raw endpoints work but recording metrics are absent, check discovery files, target state, network, rule loading, and labels in order. If JMX HTTP works but jmx_scrape_error is 1, inspect Kafka logs and MBean matching in /etc/kafka/jmx_exporter.yml.

For complete metric semantics, see Metric Definitions.