Vagrant provisioning using shell - how to setup storm project

Provisioning in Vagrant

There is a lot of choices when you want to provision your virtual machine with vagrant:
1. Run scripts manually - completely insane idea
2. Run shell scripts inline or not
3. Use chef or puppet

The first one is good only for testing, trying, ..., but it is insane if you want to provision all machines like this.

The second one is looks quite optimistic, especially if you don't know what are tools from third point.

Today I will try to present how to setup storm project on vagrant machine using only shell provisioning. To simplify the operation I will use tutorial from Hortonworks:
1000 words which describe my work looks like:
setup storm with vagrant in bash

Step .0. Choosing and preparing machine

First we need to create directory and init vagrant so let's do this:

mkdir storm-vagrant  
cd  storm-vagrant  
vagrant init  

I will use centos box which I downloaded before (you can use URL instead of file). Why Centos you asked? Because in many companies the only allowed Linux is Red Hat. Centos is closed enough. So let's add the box:

 vagrant box add "centos-6.5" file:///c:/vagrantfiles/

In vagrant file we need to change our box name so = "base" becomes = "centos-6.5"

Step .1. Configure network

We will need network access to guest machine. It can be done in two ways:
1. Using port forwarding
2. Setting IP adress for guest machine.

I will add code to enable private network in machine so vagrantfile will look like below.:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|  
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at

  # Every Vagrant virtual environment requires a box to build off of. = "centos-6.5" "private_network", ip: ""

Caution: In next code listings I will only append above lines.

Now we are ready to run:

vagrant up  

After the above command is complete (don't worry about read color on console), let's explore our box. Run vagrant ssh and you are inside. Check what you want and logout from the machine.

Step .2. Installing storm rpms.

According to " Chapter 1. Getting Ready to Install" we need to configure remote repositories. For Centos 6 the line is:

wget -nv -O /etc/yum.repos.d/hdp.repo  

So let's change our vagrant file to (first two lines already exist in our file): = "centos-6.5" "private_network", ip: ""
  config.vm.provision :shell, :inline  => "wget -nv -O /etc/yum.repos.d/hdp.repo"

According to Chapter 17.1 Install the Storm RPMs. we can install storm rpm with

yum install storm  

but above command will prompt as if we are sure to download storm + zookeeper (yes storm needs zookeeper to run). To avoid prompt just add -y to the command. So vagrant file will evolve to: = "centos-6.5"  
  config.vm.provision :shell, :inline  => "wget -nv -O /etc/yum.repos.d/hdp.repo"
  config.vm.provision :shell, :inline  => "yum -y install storm"

To call above provisioning we have to choices:
1. Run vagrant destroy and again vagrant up - if we do too much manual changes in previous steps
2. Or just vagrant provision to run only provisioning on running box

Step .3. Running zookeeper

Zookeeper? Why? -you think now. Storm need zookeeper to comunicate between its nodes. The second line of last chapter: 5. Validate the Installation is:

You must start ZooKeeper before starting Storm.

First we need Java (for storm and zookeeper). I prefer Java 7 form Sun, so I just downloaded rpm form official page to vagrant folder. In recipe please add:

 config.vm.provision :shell, :inline  => '
    javarpm="$(ls /vagrant/ | grep "^jdk.*.rpm" | tail -1)"
    echo $javarpm
    rpm -Uvh /vagrant/$javarpm
    java -version'

Please notice that I have to change " into '. The second option will be escape " with \"

Then we need to create dirs as it is explain in Chapter 5.2: Set Directories and Permissions. I created a file with below content. It should be placed side-by-side to Vagrantfile:


# Directory where ZooKeeper will store data. For example, /grid1/hadoop/zookeeper/data
export ZOOKEEPER_DATA_DIR="/grid1/hadoop/zookeeper/data";  
# Directory to store the ZooKeeper configuration files.
export ZOOKEEPER_CONF_DIR="/etc/zookeeper/conf";  
# Directory to store the ZooKeeper logs.
export ZOOKEEPER_LOG_DIR="/var/log/zookeeper";  
# Directory to store the ZooKeeper process ID.
export ZOOKEEPER_PID_DIR="/var/run/zookeeper";  

Caution: If you are on Windows host, remember to change line ending to Unix style to avoid : command not found printing on console during login to guest machine

And do provisioning below with all steps mention in zookeeper chapter (creating dirs, set zookeeper node id and start it).

config.vm.provision :shell, :inline   => "  
    yes | cp /vagrant/ /etc/profile.d/
    chmod 755 /etc/profile.d/"

config.vm.provision :shell, :inline  => "  
    mkdir -p $ZOOKEEPER_LOG_DIR;
    chmod -R 755 $ZOOKEEPER_LOG_DIR;

    mkdir -p $ZOOKEEPER_PID_DIR;
    chmod -R 755 $ZOOKEEPER_PID_DIR;

    mkdir -p $ZOOKEEPER_DATA_DIR;
    chmod -R 755 $ZOOKEEPER_DATA_DIR;

    echo '1' >> $ZOOKEEPER_DATA_DIR/myid

    su - zookeeper -c 'source /etc/zookeeper/conf/ ; export ZOOCFGDIR=/etc/zookeeper/conf;/usr/lib/zookeeper/bin/ start >> /var/log/zookeeper/zoo.out 2>&1'"

To check if ZooKeeper is running we need to login to our box and run:

ps aux | grep zookeeper  

Is process does not exist just check:

cat /var/log/zookeeper/zoo.out  

Step .4. Configure Storm

After reading 17.2. Configure Storm a new file is needed: storm.yaml. Because everything will be on one machine I will set ZOOKEEPER_SERVERS and NIMBUS_HOSTNAME to localhost. Create a storm.yaml again side-by-side to vagrantfile with following content:

    - localhost localhost

    - "localhost"

storm.local.dir: /tmp/storm/local

logviewer.port: 8081  

and copy it into /etc/storm/conf/storm.yaml. The above file is located in shared /vagrant directory.

Both above modification in vagrantfile:

  config.vm.provision :shell, :inline  => "yum -y install storm"
  config.vm.provision :shell, :inline  => "
    mkdir -p  $STORM_LOCAL_DIR
    chown -R storm:storm $STORM_LOCAL_DIR
   chmod -R 755 $STORM_LOCAL_DIR

   cp -f /vagrant/storm.yaml /etc/storm/conf/storm.yaml"

Attention: I added mkdir -p $STORM_LOCAL_DIR because this folder wasn't create on destination machine.

The last step is to add proper exports in file:

#storm local dir
export STORM_LOCAL_DIR="/tmp/storm/local";  

To validate install we should login to guest and run:

sudo su - storm  
storm nimbus  

The expected output is something similar to:

/usr/bin/storm: line 2: /etc/default/hadoop: No such file or directory

Running: java -server -Dstorm.options= -Dstorm.home=/usr/lib/storm   [...snip...] backtype.storm.daemon.nimbus  

Don't be afraid about first line. We won't need Hadoop at all, but Hortonworks installation files assume that we install it everywhere.

Step .5. Configure Process Controller

The optional 3 chapter I omit because we don't need it now - we didn't secure zookeeper. What is interesting we didn't do anything about zookeeper - let's back to this later. Just believe me.

In chapter 4. Configure Process Controller there is a mention about tool called supervisord. If we check it using yum search supervisord we won't find it in enabled repositories. To install it we need EPEL repo with

cd /tmp  
sudo rpm -Uhv epel-release-6-8.noarch.rpm  

and then run: yum install -y supervisor. Because supervisor.conf can contain password it is suggested to mark it with 600

So in vagrant file we have to add above (remember with -y in yum command):

config.vm.provision :shell, :inline  => "  
    cd /tmp  
    sudo rpm -Uhv epel-release-6-8.noarch.rpm  
    yum install -y supervisor
     sudo chmod 600 /etc/supervisord.conf"

Let's run the machine (with vagrant provision), login to it and copy default conf file: sudo cp /etc/supervisord.conf /vagrant. In copied file we can append lines from 4. Configure Process Controller.

After this step we need to add copy back this file in provisioning:

config.vm.provision :shell, :inline  => "  
    yes | cp /vagrant/supervisord.conf /etc/supervisord.conf
    /etc/init.d/supervisord restart"

Remember: supervisord.conf, storm.yaml and vagrantfile are integral parts of our recipe

Step.6. Opening ports

The last part will be open ports for storm-ui, which is by default 8080. We should add line in /etc/sysconfig/iptables file and restart iptables. The file will look like (remember about UNIX line endings):

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

And provisioning is:

  config.vm.provision :shell, :inline  => "
    yes | cp /vagrant/iptables /etc/sysconfig/iptables
    /etc/init.d/iptables restart"

Then run vagrant provision for the last time.

Step.7. Check installation

At least we check if everything is working. Just open your favorite browser on and you should see Storm UI web page. If there is a connection error you can:
1. Wait a while - it need some time to start :)
2. Check logs in /var/log/storm and /var/log/zookeeper
3. Recreate your machine with running vagrant destroy and then vagrant up

That all. Next time I will show how to use better provisiong than shell, because as you probably notice shell provisionig works, but is quite ugly and has a lot of harcoded stuff.