Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Vagrant allows you to create and configure lightweight, reproducible, and portable development environments. This post is a walkthrough in setting up a single node Kafka cluster based on Vagrant with a Virtualbox provider (Make the necessary adjustments for your provider).

This post (where I borrowed the original setup) shows you how to extend to more than one node for your development cluster.

I will assume that you have Vagrant from vagrantup.com installed.

Follow the following steps:

1. Navigate to the directory where you would like to setup your Vagrant node.
   mkdir ubuntu-cluster-node-1
cd ubuntu-cluster-node-1
vagrant init ubuntu/vivid64

2. Create the following shell script and save it as bootstrap.sh

   #!/usr/bin/env bash
   
   apt-get update
   apt-get install -y openjdk-7-jdk build-essential git
   wget -nc -nv 
   
   http://www.webhostingjams.com/mirror/apache/kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz
   mkdir /usr/local/kafka
   tar -zxvf kafka_2.10-0.9.0.1.tgz
   mv kafka_2.10-0.9.0.1 /usr/local/kafka
   
   wget -nc -nv http://apache.mirrors.hoobly.com/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
   mkdir /usr/local/zookeeper
   tar -zxvf zookeeper-3.4.8.tar.gz --directory /usr/local/zookeeper
   
   export ZK_HOME=/usr/local/zookeeper/zookeeper-3.4.8
   export KAFKA_HOME=/usr/local/kafka/kafka_2.10-0.9.0.1
   export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
   
   cp $ZK_HOME/conf/zoo_sample.cfg $ZK_HOME/conf/zoo.cfg
   sed -i 's/\/tmp\/zookeeper/\/var\/zookeeper\/data/g' $ZK_HOME/conf/zoo.cfg
   
   echo 'server.1=192.168.33.10:2888:3888' >> /$ZK_HOME/conf/zoo.cfg
   mkdir -p /var/zookeeper/data
   #Zookeeper uses a file named “myid” to identify itself within the cluster. It holds a single character, 1-255. Let’s set it to 1.
   echo "1" > /var/zookeeper/data/myid
   
   #configure kafka
   sed -i 's/broker\.id\=0/broker\.id\=1/g' $KAFKA_HOME/config/server.properties
   
   #uncomment host.name=localhost
   sed -i 's/\#host.name\=localhost/host.name\=192.168.33.10/g' $KAFKA_HOME/config/server.properties
   sed -i 's/zookeeper.connect=localhost:2181/zookeeper.connect=192.168.33.10:2181/g' $KAFKA_HOME/config/server.properties
   sed -i 's/listeners\=PLAINTEXT\:\/\/\:9092/listeners\=PLAINTEXT\:\/\/192.168.33.10\:9092/g' $KAFKA_HOME/config/server.properties
   

3. Create the following shell script and save it as startup.sh. You could also use configure upstart

   #!/usr/bin/env bash
   
   export ZK_HOME=/usr/local/zookeeper/zookeeper-3.4.8
   export KAFKA_HOME=/usr/local/kafka/kafka_2.10-0.9.0.1
   export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
   
   #Start Zookeeper
   
   $ZK_HOME/bin/zkServer.sh start
   #Start Kafka
   
   $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

4. Update your Vagrantfile to include  bootstrap.sh and startup.sh

   config.vm.provision :shell, path: "bootstrap.sh"
   config.vm.provision :shell, path: "startup.sh", run: "always"

5. Update your Vagrant memory configuration

      config.vm.provider "virtualbox" do |vb|
        # Display the VirtualBox GUI when booting the machine
        vb.gui = true
   
        # Customize the amount of memory on the VM:
        vb.memory = "2048"
      end

6. Configure a static IP for your Node

   config.vm.network "private_network", ip: "192.168.33.10"

7. Startup Vagrant

   vagrant up