Kafka Deployment

This guide assumes you have a clean Kubernetes cluster, and teaches you how to deploy Kafka, Schema Registry and Kafka Connect using the Confluent Helm charts.

Note that it's also possible to use a different implementation such as Strimzi (see below).

Prepare a values yaml file for the Helm chart:

cp-kafka:
  persistence:
    enabled: true
    size: 256Gi
    disksPerBroker: 1

cp-kafka-connect:
  image: aporiadocker/kafka-connect-s3
  imageTag: latest

  # NOTE: In production you should create a ServiceAccount with limited IAM access to the relevant S3 buckets
  customEnv:
    AWS_ACCESS_KEY_ID: <AWS_ACCESS_KEY_ID>
    AWS_SECRET_ACCESS_KEY: <AWS_SECRET_ACCESS_KEY>
    AWS_DEFAULT_REGION: <REGION>

Install everything:

helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/
helm repo update

helm install kafka confluentinc/cp-helm-charts -f cp-values.yaml

Note the aporiadocker/kafka-connect-s3 Docker image - it's a simple Docker image that contains Kafka Connect with the Confluent S3 Sink connector.

You can build your own with the following Dockerfile:

FROM confluentinc/cp-kafka-connect-base:7.0.1
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:latest

Alternative - Strimzi

If you'd like to use Strimzi Kafka instead of the Confluent Kafka, run the following commands instead. First, install the Strimzi operator:

kubectl create namespace kafka
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

Create a Kafka instance:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.1.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.1"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

You'll still need to use the Confluent Schema Registry and S3 Connector, which are licensed under the Confluent Community license:

git clone https://github.com/confluentinc/cp-helm-charts.git
helm install schema-registry cp-helm-charts/charts/cp-schema-registry --set kafka.bootstrapServers="my-cluster-kafka-bootstrap:9092" 
helm install kafka-connect cp-helm-charts/charts/cp-kafka-connect --set kafka.bootstrapServers="my-cluster-kafka-bootstrap:9092",cp-schema-registry.url="schema-registry-cp-schema-registry:8081" --set image=aporiadocker/kafka-connect-s3 --set imageTag=latest

Kafka UI (Optional)

This is very useful for development.

helm repo add kafka-ui https://provectus.github.io/kafka-ui
helm repo update

helm install kafka-ui kafka-ui/kafka-ui --set envs.config.KAFKA_CLUSTERS_0_NAME=kafka --set envs.config.KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka-cp-kafka:9092 

To connect to Kafka-UI web application, run:

kubectl port-forward svc/kafka-ui 8080:80

Next Steps

Now that you have a prepared environment, see our Quickstart guide.