Apache Druid

​

Overview

• Run it directly to get a managed Druid deployment with all necessary components
• Inherit it in your own stack to seamlessly add real-time analytics capabilities

​

What this template manages

• Druid coordinator (cluster management)
• Druid broker (query routing)
• Druid router (HTTP routing)
• Druid historical (segment serving)
• Druid middlemanager (task execution)
• PostgreSQL database (metadata storage)
• ZooKeeper (coordination)

​

Quick start (run directly)

1. Load templates

monk load MANIFEST

2. Run Druid stack with defaults

monk run druid/stack

3. Customize configuration (recommended via inheritance)

• Preferred: inherit and replace variables with secret("...") as shown below.
• Alternative: fork/clone and edit the variables in stack.yml, then monk load MANIFEST and run.

​

Configuration

variables:
  image_tag: "v2.00.9"                                # Druid container image tag
  java_xmx: "1024m"                                   # Maximum heap size
  java_xms: "1024m"                                   # Initial heap size
  java_max_new_size: "256m"                           # Java max new size
  java_max_direct_memory: "512m"                      # Max direct memory size
  single_node: "micro-quickstart"                     # Single node configuration
  log_level: "debug"                                  # Log level (debug, info, warn, error)
  metadata_storage_type: "postgresql"                 # Metadata storage backend
  metadata_storage_connector_user: "monk"             # Metadata DB user
  metadata_storage_connector_password: "monk"         # Metadata DB password
  droid_storage_type: "local"                         # Storage type (local, s3, etc.)
  droid_storage_directory: "/opt/shared"              # Local storage directory
  druid_processing_num: "2"                           # Number of processing threads
  druid_processing_num_merge: "2"                     # Number of merge buffers
  druid_processing_buffer: "56m"                      # Processing buffer size
  coordinator_balance: "cachingCost"                  # Coordinator balance strategy

​

Stack components

• druid/coordinator - Cluster management and segment assignment
• druid/broker - Query routing and result merging
• druid/router - HTTP request routing
• druid/historical - Segment storage and querying
• druid/middlemanager - Task execution and ingestion
• druid/db - PostgreSQL metadata storage
• druid/dzookeeper - ZooKeeper coordination

​

Use by inheritance (recommended for apps)

namespace: myapp
analytics:
  defines: runnable
  inherits: druid/stack
  variables:
    metadata_storage_connector_password:
      value: <- secret("druid-db-password")
api:
  defines: runnable
  containers:
    api:
      image: myorg/api
  connections:
    druid:
      runnable: analytics
      service: broker
  variables:
    druid-broker-url:
      value: <- connection-hostname("druid") concat-all ":" connection-port("druid")

monk secrets add -g druid-db-password="STRONG_DB_PASSWORD"
monk run myapp/api

​

Features

• Real-time Ingestion: Ingest streaming data with exactly-once semantics
• Fast Queries: Sub-second queries on large datasets
• Scalable: Horizontally scalable architecture
• Column-oriented: Efficient storage and query execution
• Multi-tenant: Supports multiple tenants and workloads
• SQL Support: Query using SQL or native queries

​

Ports and connectivity

• Service: router on TCP port 8888 (web console)
• Service: broker on TCP port 8082 (query API)
• Service: coordinator on TCP port 8081
• Service: historical on TCP port 8083
• Service: middlemanager on TCP port 8091
• From other runnables in the same process group, use connection-hostname("\<connection-name>") to resolve the service host.

​

Persistence and configuration

• Deep storage path: ${monk-volume-path}/druid:/opt/shared
• PostgreSQL data: ${monk-volume-path}/postgresql:/var/lib/postgresql/data
• ZooKeeper data: ${monk-volume-path}/zookeeper:/data
• You can adjust JVM settings and Druid configuration through the template variables.

​

Related templates

• See other templates in this repository for complementary services
• Combine with monitoring tools (prometheus-grafana/) for observability
• Integrate with your application stack as needed

​

Troubleshooting

• Ensure all required ports are available (8081-8083, 8088, 8091, 8888)
• Druid requires ZooKeeper and metadata storage (PostgreSQL) to be running first
• If you changed metadata_storage_connector_password but the database has existing data, authentication may fail. Either reset the data volume or update the password in PostgreSQL.
• Ensure the host volumes are writable by the container user
• Check logs for any component:

monk logs -l 500 -f local/druid/stack
monk logs -l 500 -f local/druid/broker

• Verify JVM settings are appropriate for your workload
• Ensure sufficient memory is available for the configured heap sizes
• For single-node deployments, use single_node: "micro-quickstart" or "small"