Saturday, 20 December 2014

Introduction and getting started with Apache Mesos

Introduction to Apache Mesos

In this era of distributed computing, where we spin up clusters for Hadoop, Storm, Jenkins, Cassandra, etc separately, we are not making effective use of the resources. There would be long pauses in the cluster after entering a burst of information, thus making it very in-efficient. 
Now, what if all these frameworks shared the same set of machines and resources, then small slices of time spent waiting for some resources could be granted to other frameworks. This is the concept of Time Sharing. 

Apache Mesos is a datacenter operating system and it shares the same philosophy of time-sharing. Mesos is called a datacenter because it hosts different frameworks under a single roof. It is called an operating system because shares many concepts of Linux.
1. Isolation : Linux creates isolation through processes where, each of these processes has its own file descriptors and its own address space. This is achieved by Linux Containers (wiki:LXC) in Mesos
2. Process Scheduler : The processes have accesses to the system resources by balancing the work loads across multiple computing resources, thereby optimizing resources, maximize the throughput, minimizing the response time and avoiding the overhead by any one resource. There are various scheduling algorithms to execute more than one process at a time (wiki:Multitasking) and also transmit multiple data streams simultaneously across a single physical channel(wiki:Multiplexing). Mesos uses such scheduling algorithms.
3. Common Infrastructure : Linux has a set of calls irrespective of filesystems, drivers etc. Similarly, Mesos has a common set of calls which helps in the execution of tasks.
4. Package Manager : Linux has apt-get, aptitude, synaptic, yum etc that helps in the automation of the process of installing, upgrading, configuring, and removing software. Similarly, Mesos has a recent support for Docker(wiki:Docker)

Traditionally, distributed systems has 2 components in a non-peer to peer systems.
1. Coordinator : Generate tasks, send the tasks to worker and receive results from Worker.
2. Worker : Execute the tasks and send the status and results back to Worker.

With Mesos, there are three levels Coordinator, Mesos master and Mesos slaves where coordinator negotiates with mesos master and then master decides on partitioning the cluster to distribute the tasks. Thus, we can schedule jobs across the machines, thereby running hadoop, cassandra, spark etc.
All the distributed systems that run on Mesos are called applications or frameworks and the coordinator is called as scheduler in Mesos vocabulary.

How does Mesos work?

In summary, Mesos works on a request/offer based model. Whenever, you want to run a job, you send a request. These requests are simplified subset of specification like number of GPUs, RAM etc, at that point of time. Mesos, checks for the request specification and it will reply back with the resource offers of what resources are available on a set of machines. This is non-blocking and has two level of scheduling : Offering and Scheduling.
Mesos master: Control the resource allocation to the schedulers
Scheduler: Uses the resource offers to decide which tasks to run and which one to run next.
More information on Mesos architecture is here : Mesos Architecture

Getting started with mesos

1. Download the tarball from the Mirror Apache Mesos v0.21.0 and untar it.

tar -zvxf  mesos-0.21.0.tar.gz 
cd mesos-0.21.0/

2. Install the dependencies

sudo apt-get update
sudo apt-get install build-essential openjdk-6-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev maven  libapr1-dev libsvn-dev

3. Building Mesos
Please make sure it has appropriate permissions while building.

mkdir build
cd build
make check 
make install

Start Mesos Master

./bin/ --ip= --work_dir=/var/lib/mesos

Start Mesos Slave

./bin/ --master=

Web UI

Running a test framework in Java


Mesos Home
Mesos Frameworks

Mesos  executor tasks
Mesos Slave Nodes

In the next post, let us see how to build our own distributed framework on Apache Mesos.
Happy Learning! :)

Tuesday, 5 August 2014

Raspberry Pi B+ : Connect to your Pi with no display monitor, keyboard or mouse

        The new Raspberry Pi B+ was released on July 14th 2014. The following link does all the justice to know about the improved features of B+.  Please refer it here : Raspberry Pi B+

        The moment I had this credit card sized thing on my palm, I was so excited to get started with it. I downloaded Raspbian from Raspberry Pi Downloads Page and unzipped it. This is the image file 2014-06-20-wheezy-raspbian.img you get after the unzip operation. I used the SanDisk 8GB Class 4 Micro SD card and dd tool to write to it. Refer Installing images on Linux here ( Please note, I did bs=1M with dd tool while writing). This was one of the guide I was following Quick Start Guide

But then, wait! Do you need a HDMI TV or display monitor, USB keyboard and a USB mouse to connect to it? Oh really!? I was not interested to invest any further. 

Here is what I did:

  1. I inserted the Micro SD card into the slot behind. 
  2. I had a micro USB cable my for Nexus 5. I used the same for the 5V power supply
  3. I had an Ethernet crossover cable already with me for some VPN stuffs. I took one of them and connected  one end to the router and the other end to the Pi Ethernet Port (next to 4 USB Ports) 
  4. Next, I used my laptop to connect to my Pi in the same network.
  5. On my laptop, 
    1. Opened Terminal. Typed in $ ifconfig  to get to know the IP address. 
      • For ex: if  inet addr: then our Pi should have  inet addr:192.168.0.*
  6. Next, for my rescue came the nmap tool. If you do not have it already installed, $ sudo apt-get install nmap 
    • Execute $ nmap -T4 -F 192.168.0.* on the terminal ( as per the example considered ) This will scan all the hosts which are up and then lists out the open ports. You will find Pi as one among them.
  7. If you like GUI Based, you may like Zenmap and can be installed by $ sudo apt-get install zenmap.
    • After installing, type $ sudo zenmap
    • Type Target as 192.168.0.*
    • Choose Profile as Quick Scan
    • Click on Scan
    • This will list the Raspberry Pi MAC Address and the IP Address
  8. Now, you know the IP Address of Pi. The default login for Raspbian is username "pi" with the password "raspberry"
  9. Time to ssh!
  10. Need to change your configuration settings after login? Type $ sudo raspi-config and change it accordingly!
Happy hacking! :)