Apache Kafka

​

Overview

• Run it directly to get a managed Kafka cluster with ZooKeeper coordination
• Inherit it in your own runnable to seamlessly add event streaming capabilities to your stack

​

What this template manages

• Kafka broker containers (confluentinc/cp-kafka image, configurable tag)
• ZooKeeper ensemble for cluster coordination (confluentinc/cp-zookeeper image, configurable tag)
• Network services on ports 9092 (Kafka) and 2181 (ZooKeeper)
• Persistent volumes for Kafka data and ZooKeeper state
• Cluster configuration and topic management

​

Quick start (run directly)

1. Load templates

monk load MANIFEST

2. Run Kafka stack with defaults

monk run kafka/stack

3. Customize configuration (recommended via inheritance)

• Preferred: inherit and set variables as shown below
• Alternative: fork/clone and edit the variables in kafka.yaml and stack.yaml, then monk load MANIFEST and run

​

Configuration

variables:
  kafka-image-tag: "7.2.1"                # Kafka container image tag
  zookeeper-image-tag: "7.2.1"            # ZooKeeper container image tag
  kafka-port: 9092                        # Kafka broker port
  zookeeper-port: 2181                    # ZooKeeper client port
  kafka-advertised-listeners: "..."       # KAFKA_ADVERTISED_LISTENERS
  kafka-broker-id: 1                      # KAFKA_BROKER_ID
  kafka-log-retention-hours: 168          # Log retention (hours)
  kafka-num-partitions: 1                 # Default number of partitions

​

Use by inheritance (recommended for apps)

namespace: myapp
messaging:
  defines: runnable
  inherits: kafka/stack
api:
  defines: runnable
  containers:
    api:
      image: myorg/api
  connections:
    kafka:
      runnable: messaging
      service: kafka
  variables:
    kafka-host:
      value: <- connection-hostname("kafka")
    kafka-port:
      value: <- connection-port("kafka")

monk run myapp/api

​

Ports and connectivity

• Service: kafka on TCP port 9092 (broker)
• Service: zookeeper on TCP port 2181 (coordination)
• From other runnables in the same process group, use connection-hostname("\<connection-name>") to resolve the Kafka broker host
• External clients can connect via the advertised listeners configuration

​

Persistence and configuration

• Kafka data path: ${monk-volume-path}/kafka:/var/lib/kafka/data
• ZooKeeper data path: ${monk-volume-path}/zookeeper:/var/lib/zookeeper
• Ensure the host volumes are writable by the container user

​

Features

• High-throughput messaging (millions of messages per second)
• Fault-tolerant storage with replication
• Stream processing (Kafka Streams)
• Scalable architecture (horizontal scaling)
• Exactly-once semantics
• Multiple consumer groups
• Built-in partitioning for parallelism

​

Related templates

• Use with microservices for async communication
• Combine with stream processing frameworks (Kafka Streams, Flink, Spark Streaming)
• Integrate with monitoring (prometheus-grafana/) for metrics and alerting
• Pair with schema registry for data governance

​

Troubleshooting

• Ensure ZooKeeper is running before Kafka starts. The stack handles this dependency automatically.
• If connection fails, verify KAFKA_ADVERTISED_LISTENERS matches your network configuration
• For external access, ensure firewall rules allow traffic on port 9092
• Check logs:

monk logs -l 500 -f kafka/stack

• To check ZooKeeper separately:

monk logs -l 500 -f kafka/zookeeper