Running a Single Node Apache Kafka with Vagrant

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Vagrant allows you to create and configure lightweight, reproducible, and portable development environments. This post is a walkthrough in setting up a single node Kafka cluster based on Vagrant with a Virtualbox provider (Make the necessary adjustments for your provider).

This post (where I borrowed the original setup) shows you how to extend to more than one node for your development cluster.

I will assume that you have Vagrant from vagrantup.com installed.

Follow the following steps:

  1. Navigate to the directory where you would like to setup your Vagrant node.
    mkdir ubuntu-cluster-node-1
    cd ubuntu-cluster-node-1
    vagrant init ubuntu/vivid64
  2. Create the following shell script and save it as bootstrap.sh
    #!/usr/bin/env bash
    
    apt-get update
    apt-get install -y openjdk-7-jdk build-essential git
    wget -nc -nv 
    
    http://www.webhostingjams.com/mirror/apache/kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz
    mkdir /usr/local/kafka
    tar -zxvf kafka_2.10-0.9.0.1.tgz
    mv kafka_2.10-0.9.0.1 /usr/local/kafka
    
    wget -nc -nv http://apache.mirrors.hoobly.com/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
    mkdir /usr/local/zookeeper
    tar -zxvf zookeeper-3.4.8.tar.gz --directory /usr/local/zookeeper
    
    export ZK_HOME=/usr/local/zookeeper/zookeeper-3.4.8
    export KAFKA_HOME=/usr/local/kafka/kafka_2.10-0.9.0.1
    export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
    
    cp $ZK_HOME/conf/zoo_sample.cfg $ZK_HOME/conf/zoo.cfg
    sed -i 's/\/tmp\/zookeeper/\/var\/zookeeper\/data/g' $ZK_HOME/conf/zoo.cfg
    
    echo 'server.1=192.168.33.10:2888:3888' >> /$ZK_HOME/conf/zoo.cfg
    mkdir -p /var/zookeeper/data
    #Zookeeper uses a file named “myid” to identify itself within the cluster. It holds a single character, 1-255. Let’s set it to 1.
    echo "1" > /var/zookeeper/data/myid
    
    #configure kafka
    sed -i 's/broker\.id\=0/broker\.id\=1/g' $KAFKA_HOME/config/server.properties
    
    #uncomment host.name=localhost
    sed -i 's/\#host.name\=localhost/host.name\=192.168.33.10/g' $KAFKA_HOME/config/server.properties
    sed -i 's/zookeeper.connect=localhost:2181/zookeeper.connect=192.168.33.10:2181/g' $KAFKA_HOME/config/server.properties
    sed -i 's/listeners\=PLAINTEXT\:\/\/\:9092/listeners\=PLAINTEXT\:\/\/192.168.33.10\:9092/g' $KAFKA_HOME/config/server.properties
    
  3. Create the following shell script and save it as startup.sh. You could also use configure upstart
    #!/usr/bin/env bash
    
    export ZK_HOME=/usr/local/zookeeper/zookeeper-3.4.8
    export KAFKA_HOME=/usr/local/kafka/kafka_2.10-0.9.0.1
    export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
    
    #Start Zookeeper
    
    $ZK_HOME/bin/zkServer.sh start
    #Start Kafka
    
    $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
  4. Update your Vagrantfile to include  bootstrap.sh and startup.sh
    config.vm.provision :shell, path: "bootstrap.sh"
    config.vm.provision :shell, path: "startup.sh", run: "always"
  5. Update your Vagrant memory configuration
       config.vm.provider "virtualbox" do |vb|
         # Display the VirtualBox GUI when booting the machine
         vb.gui = true
    
         # Customize the amount of memory on the VM:
         vb.memory = "2048"
       end
  6. Configure a static IP for your Node
    config.vm.network "private_network", ip: "192.168.33.10"
  7. Startup Vagrant
    vagrant up

		
Advertisements

OSX Microsoft SQL Server Client

I work on a Mac and in a bandwidth rich environment, such as my work network, I usually connect to SQL Server via a MSSQL client running on a terminal server which I access over RDP. I like the GUI with the added intelliSense features from RedGate tools, plus the OSX RDP Client by Microsoft works great.

While traveling, I prefer to have access to a low bandwidth solution. I started investigating and playing with a few OSX MSSQL clients available.

I tried all the tools listed here including a DB Visualizer plugin for eclipse. In my analysis I factored in cost and feature set as well as Mean time to annoyance (MTTA).

Tool Cost MTTA
SQLPro for MSSQL >$0.00 Unknown
SQuirrel SQL (free, open source) Free Open Source Immediate (Not so intuitive)
DB Visualizer Freemium Tolerable up to a day
SQL Workbench/J (free, open source) Free Open Source Immediate (Not so intuitive)
Oracle SQL Developer (free) Free Not Open Source Still Using
TOAD For MAC Free Immediate (No MSSQL Support on OSX)

My recommendation is to use the free Oracle SQL Developer with this instructions for setting up the MSSQL JTDS driver. This tool gets the job done!