Apache Kafka Customization Properties

The kafka_values.yaml file contains many settings that you can customise.

For test installations, these values were customized, as shown below:

  1. Disable persistence.
  2. Reduce the Kafka cluster size to a single replica.
  3. Reduce the zookeeper cluster size to a single replica.
  4. Enable service monitoring using a Prometheus operator.
  5. Affinity rules to restrict where Kafka can be deployed, preferring the same node as the zookeeper Pod.

# ------------------------------------------------------------------------------
# Kafka:
# SOURCE: https://github.com/helm/charts/blob/master/incubator/kafka/values.yaml
# NOTES:
# 1. some values have been modified for TESTING, such as replicas and persistence
# 2. the affinity rules have been applied to deploy kafka to same node as zookeeper
# ------------------------------------------------------------------------------
 
## The StatefulSet installs 3 pods by default
## TESTING: single replica
replicas: 1
 
## Pod scheduling preferences (by default keep pods within a release on separate nodes).
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
## By default we don't set affinity
# affinity: {}
## TESTING: affinity to zookeeper, anti-affinity to other kafka replicas
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - kafka
        topologyKey: "kubernetes.io/hostname"
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - zookeeper
          topologyKey: "kubernetes.io/hostname"
 
## Persistence configuration. Specify if and how to persist data to a persistent volume.
##
persistence:
  ## TESTING: disable persistence
  enabled: false
 
# ------------------------------------------------------------------------------
# Zookeeper:
# ------------------------------------------------------------------------------
zookeeper:
  ## If true, install the Zookeeper chart alongside Kafka
  ## ref: https://github.com/kubernetes/charts/tree/master/incubator/zookeeper
  enabled: true
 
  ## Desired quantity of ZooKeeper pods. This should always be (1, 3, 5 or 7)
  ## TESTING: single replica
  replicaCount: 1