Introduction and getting started with Apache Mesos

Introduction to Apache Mesos

In this era of distributed computing, where we spin up clusters for Hadoop, Storm, Jenkins, Cassandra, etc separately, we are not making effective use of the resources. There would be long pauses in the cluster after entering a burst of information, thus making it very in-efficient. 
Now, what if all these frameworks shared the same set of machines and resources, then small slices of time spent waiting for some resources could be granted to other frameworks. This is the concept of Time Sharing. 

Apache Mesos is a datacenter operating system and it shares the same philosophy of time-sharing. Mesos is called a datacenter because it hosts different frameworks under a single roof. It is called an operating system because shares many concepts of Linux.
1. Isolation : Linux creates isolation through processes where, each of these processes has its own file descriptors and its own address space. This is achieved by Linux Containers (wiki:LXC) in Mesos
2. Process Scheduler : The processes have accesses to the system resources by balancing the work loads across multiple computing resources, thereby optimizing resources, maximize the throughput, minimizing the response time and avoiding the overhead by any one resource. There are various scheduling algorithms to execute more than one process at a time (wiki:Multitasking) and also transmit multiple data streams simultaneously across a single physical channel(wiki:Multiplexing). Mesos uses such scheduling algorithms.
3. Common Infrastructure : Linux has a set of calls irrespective of filesystems, drivers etc. Similarly, Mesos has a common set of calls which helps in the execution of tasks.
4. Package Manager : Linux has apt-get, aptitude, synaptic, yum etc that helps in the automation of the process of installing, upgrading, configuring, and removing software. Similarly, Mesos has a recent support for Docker(wiki:Docker)

Traditionally, distributed systems has 2 components in a non-peer to peer systems.
1. Coordinator : Generate tasks, send the tasks to worker and receive results from Worker.
2. Worker : Execute the tasks and send the status and results back to Worker.

With Mesos, there are three levels Coordinator, Mesos master and Mesos slaves where coordinator negotiates with mesos master and then master decides on partitioning the cluster to distribute the tasks. Thus, we can schedule jobs across the machines, thereby running hadoop, cassandra, spark etc.
All the distributed systems that run on Mesos are called applications or frameworks and the coordinator is called as scheduler in Mesos vocabulary.

How does Mesos work?

In summary, Mesos works on a request/offer based model. Whenever, you want to run a job, you send a request. These requests are simplified subset of specification like number of GPUs, RAM etc, at that point of time. Mesos, checks for the request specification and it will reply back with the resource offers of what resources are available on a set of machines. This is non-blocking and has two level of scheduling : Offering and Scheduling.
Mesos master: Control the resource allocation to the schedulers
Scheduler: Uses the resource offers to decide which tasks to run and which one to run next.
More information on Mesos architecture is here : Mesos Architecture

Getting started with mesos

1. Download the tarball from the Mirror Apache Mesos v0.21.0 and untar it.

tar -zvxf  mesos-0.21.0.tar.gz 
cd mesos-0.21.0/

2. Install the dependencies

sudo apt-get update
sudo apt-get install build-essential openjdk-6-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev maven  libapr1-dev libsvn-dev

3. Building Mesos
Please make sure it has appropriate permissions while building.

mkdir build
cd build
../configure
make
make check 
make install

Start Mesos Master

./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos

Start Mesos Slave

./bin/mesos-slave.sh --master=127.0.0.1:5050

Web UI

http://127.0.0.1:5050

Running a test framework in Java

./src/examples/java/test-framework 127.0.0.1:5050

Mesos Home

Mesos Frameworks

Mesos  executor tasks

Mesos Slave Nodes

In the next post, let us see how to build our own distributed framework on Apache Mesos.
Happy Learning! :)