Running a Single Node Apache Kafka with Vagrant

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Vagrant allows you to create and configure lightweight, reproducible, and portable development environments. This post is a walkthrough in setting up a single node Kafka cluster based on Vagrant with a Virtualbox provider (Make the necessary adjustments for your provider).

This post (where I borrowed the original setup) shows you how to extend to more than one node for your development cluster.

I will assume that you have Vagrant from vagrantup.com installed.

Follow the following steps:

  1. Navigate to the directory where you would like to setup your Vagrant node.
    mkdir ubuntu-cluster-node-1
    cd ubuntu-cluster-node-1
    vagrant init ubuntu/vivid64
  2. Create the following shell script and save it as bootstrap.sh
    #!/usr/bin/env bash
    
    apt-get update
    apt-get install -y openjdk-7-jdk build-essential git
    wget -nc -nv 
    
    http://www.webhostingjams.com/mirror/apache/kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz
    mkdir /usr/local/kafka
    tar -zxvf kafka_2.10-0.9.0.1.tgz
    mv kafka_2.10-0.9.0.1 /usr/local/kafka
    
    wget -nc -nv http://apache.mirrors.hoobly.com/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
    mkdir /usr/local/zookeeper
    tar -zxvf zookeeper-3.4.8.tar.gz --directory /usr/local/zookeeper
    
    export ZK_HOME=/usr/local/zookeeper/zookeeper-3.4.8
    export KAFKA_HOME=/usr/local/kafka/kafka_2.10-0.9.0.1
    export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
    
    cp $ZK_HOME/conf/zoo_sample.cfg $ZK_HOME/conf/zoo.cfg
    sed -i 's/\/tmp\/zookeeper/\/var\/zookeeper\/data/g' $ZK_HOME/conf/zoo.cfg
    
    echo 'server.1=192.168.33.10:2888:3888' >> /$ZK_HOME/conf/zoo.cfg
    mkdir -p /var/zookeeper/data
    #Zookeeper uses a file named “myid” to identify itself within the cluster. It holds a single character, 1-255. Let’s set it to 1.
    echo "1" > /var/zookeeper/data/myid
    
    #configure kafka
    sed -i 's/broker\.id\=0/broker\.id\=1/g' $KAFKA_HOME/config/server.properties
    
    #uncomment host.name=localhost
    sed -i 's/\#host.name\=localhost/host.name\=192.168.33.10/g' $KAFKA_HOME/config/server.properties
    sed -i 's/zookeeper.connect=localhost:2181/zookeeper.connect=192.168.33.10:2181/g' $KAFKA_HOME/config/server.properties
    sed -i 's/listeners\=PLAINTEXT\:\/\/\:9092/listeners\=PLAINTEXT\:\/\/192.168.33.10\:9092/g' $KAFKA_HOME/config/server.properties
    
  3. Create the following shell script and save it as startup.sh. You could also use configure upstart
    #!/usr/bin/env bash
    
    export ZK_HOME=/usr/local/zookeeper/zookeeper-3.4.8
    export KAFKA_HOME=/usr/local/kafka/kafka_2.10-0.9.0.1
    export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
    
    #Start Zookeeper
    
    $ZK_HOME/bin/zkServer.sh start
    #Start Kafka
    
    $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
  4. Update your Vagrantfile to include  bootstrap.sh and startup.sh
    config.vm.provision :shell, path: "bootstrap.sh"
    config.vm.provision :shell, path: "startup.sh", run: "always"
  5. Update your Vagrant memory configuration
       config.vm.provider "virtualbox" do |vb|
         # Display the VirtualBox GUI when booting the machine
         vb.gui = true
    
         # Customize the amount of memory on the VM:
         vb.memory = "2048"
       end
  6. Configure a static IP for your Node
    config.vm.network "private_network", ip: "192.168.33.10"
  7. Startup Vagrant
    vagrant up

		
Advertisements