tag:blogger.com,1999:blog-61878790327857915982024-02-07T11:43:02.208+05:30* Techie(S)pArK *Learn... Unlearn... Relearn...Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-6187879032785791598.post-10081119867736808252015-09-02T14:42:00.002+05:302015-09-02T14:42:45.223+05:30Adieu!First of all - I would like to thank you all for following my posts, for all those lovely interactions on and off the blog and encouraging me to do better, all this time!<br />
<br />
Its time to say goodbye to the Blogger, as the very experience of writing posts here has become such a pain!! Adding to it, there has been no update to the blogspot, since a long time! I understand that they are busy changing logos! Sure! so be it!!<br />
So, for the obvious reasons - its time to move on!<br />
<br />
I've moved all the posts on this site to my new site : <a href="http://femgeekz.github.io/">http://femgeekz.github.io/</a>, based on a simple <a href="http://jekyllrb.com/">jekyll</a> theme.<br />
The site is still a work in progress, but that shouldn't stop me from posting new content, with all new variety!<br />
<br />
This site is no longer being updated. Please visit : <a href="http://femgeekz.github.io/">http://femgeekz.github.io/</a><br />
Thanks for stopping by! :)<br />
Adieu!<br />
<br />
- SwatzAnonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Los Angeles, CA, USA34.0522342 -118.243684933.2099567 -119.5345784 34.8945117 -116.95279140000001tag:blogger.com,1999:blog-6187879032785791598.post-81199593110948656002015-01-03T17:00:00.000+05:302015-01-03T21:29:43.094+05:30Apache Mesos : Writing your own distributed framework<span style="font-family: Trebuchet MS, sans-serif;">In the previous <a href="http://femgeekz.blogspot.in/2014/12/introduction-and-getting-started-with.html">post</a>, we saw what mesos is, how is it useful and getting started with it. In this post, we shall see how to write your own framework on mesos.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">(In mesos, a framework is any application running on it.) </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">This post explains about a framework called "mesos-pinspider" which fetches the user profile information and user board information of a pinterest page of a user.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Mesos Framework</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">In general, a Mesos framework has three basic components. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- <b>Driver</b> which submits the tasks to the framework</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- <b>Scheduler</b> which registers with the master to be offered resources, takes the tasks and runs them on executor</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- <b>Executor</b> process that is launched on slave nodes to run the framework’s tasks</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Pinspider Framework Example</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">You may check the code here on <a href="https://github.com/SwathiMystery/mesos-pinspider">github</a>. Let's break it down to PinDriver, PinScheduler and Pin</span><span style="font-family: 'Trebuchet MS', sans-serif;">UserProfileExecutor</span><span style="font-family: Trebuchet MS, sans-serif;">.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Driver</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">The driver component of the framework is PinDriver. </span><br />
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create Executor Info</span></li>
</ul>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;">Describe the information about the executor using the Builder pattern and mesos use Google Protocol Buffers for the data interchange. Here, we need to set the executorID, command which is basically a shell command, executed via:</span><span style="font-family: Courier New, Courier, monospace;"> '/bin/sh -c value'</span><span style="font-family: Trebuchet MS, sans-serif;">. Any URIs specified are fetched before executing the command. The name is set by </span><span style="font-family: Courier New, Courier, monospace;">setName()</span><span style="font-family: Trebuchet MS, sans-serif;">. The source is set by </span><span style="font-family: Courier New, Courier, monospace;">setSource()</span><span style="font-family: Trebuchet MS, sans-serif;">, an identifier style string used by frameworks to track the source of an executor. This is useful when it's possible for different executor ids to be related semantically.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.ExecutorInfo userProfileExecutorInfo = Protos.ExecutorInfo.newBuilder().setExecutorId(Protos.ExecutorID.newBuilder().setValue("PinUserProfileExecutor")).setCommand(commandInfoUserProfile).setName("PinUserProfileExecutor Java").setSource("java").build();</span></blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create Framework Info</span></li>
</ul>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;">Describe the framework information. The user field is used to determine the Unix user that an executor/task should be launched as. If the user field is set to an empty string Mesos will auto-magically set it to the current user. The amount of time that the master will wait for the scheduler to fail-over before removing the framework is specified by </span><span style="font-family: Courier New, Courier, monospace;">setFailoverTimeout()</span><span style="font-family: Trebuchet MS, sans-serif;">. The name of the framework is set by </span><span style="font-family: Courier New, Courier, monospace;">setName()</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.FrameworkInfo.Builder frameworkBuilder = Protos.FrameworkInfo.newBuilder().setFailoverTimeout(120000)</span><span style="font-family: 'Courier New', Courier, monospace;">.setUser("").setName("Pinspider Framework"); </span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Instantiate Scheduler</span></li>
</ul>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">You need to instantiate the Scheduler with the number of tasks that needs to be submitted for the executor to run.</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Scheduler scheduler = args.length == 1 ?</span><span style="font-family: 'Courier New', Courier, monospace;">new PinScheduler(userProfileExecutorInfo,userBoardExecutorInfo) : </span><span style="font-family: 'Courier New', Courier, monospace;">new PinScheduler(userProfileExecutorInfo, userBoardExecutorInfo, Integer.parseInt(args[1]), args[2]);</span></blockquote>
</blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Note: Please note that two ExecutorInfo are used ie. one for fetching user profile information and the other one for user board information for demonstration. This explanation involves only one executorinfo - userProfileExecutorInfo</span><br />
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Starting the mesos scheduler driver.</span></li>
</ul>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">MesosSchedulerDriver is an implementation of SchedulerDriver which is an abstract interface to connect scheduler to mesos. This is done by managing the life-cycle of the scheduler ( start, stop and wait for tasks to finish) and also to interact with Mesos (launch tasks, kill tasks etc). </span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">MesosSchedulerDriver schedulerDriver = new MesosSchedulerDriver(scheduler,frameworkBuilder.build(), args[0]);</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">int status = schedulerDriver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1;</span><span style="font-family: 'Courier New', Courier, monospace;">schedulerDriver.stop();</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">System.exit(status);</span></blockquote>
</blockquote>
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Executor Implementation</span></h3>
<span style="font-family: 'Trebuchet MS', sans-serif;">The </span><span style="font-family: 'Trebuchet MS', sans-serif;">Executor</span><span style="font-family: 'Trebuchet MS', sans-serif;"> component of the framework is PinUserProfileExecutor.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Executor is a callback interface which is implemented by frameworks' executors. In our implementation, let us concentrate on launchTask()</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">@Override public void launchTask(final ExecutorDriver executorDriver, final Protos.TaskInfo taskInfo) { </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Set the task status by setting the ID and the state with a builder pattern.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.TaskStatus taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId())</span><span style="font-family: 'Courier New', Courier, monospace;">.setState(Protos.TaskState.TASK_RUNNING).build();</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Send the status update to the framework scheduler retrying as necessary until an acknowledgement has been received or the executor is terminated, in which case, a TASK_LOST status update will be sent.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">executorDriver.sendStatusUpdate(taskStatus);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Get the data from the tasks and run your logic.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">try {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>message = ("userprofile :" + getUserProfileInfo(url)).getBytes();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>} catch (IOException e) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.error("Error parsing the Pinterest URL :" + e.getMessage());</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Send the framework the message.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">executorDriver.sendFrameworkMessage(message);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Mark the state of the task as finished and send the status update to the framework scheduler.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId())</span><span style="font-family: Courier New, Courier, monospace;"> .setState(Protos.TaskState.TASK_FINISHED).build();</span><span style="font-family: Courier New, Courier, monospace;">executorDriver.sendStatusUpdate(taskStatus);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> main() method to create an instance of MesosExecutorDriver and run </span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">mesosExecutorDriver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1</span></blockquote>
</blockquote>
<h3>
<span style="font-family: 'Trebuchet MS', sans-serif;">Scheduler Implementation</span></h3>
<span style="font-family: 'Trebuchet MS', sans-serif;">The Scheduler</span><span style="font-family: 'Trebuchet MS', sans-serif;"> component of the framework is Pin</span><span style="font-family: 'Trebuchet MS', sans-serif;">Scheduler</span><span style="font-family: 'Trebuchet MS', sans-serif;">.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Scheduler is a callback interface to be implemented by frameworks' schedulers. In our implemenation, let us concentrate on resourceOffers(), statusUpdate() and frameworkMessage()</span><br />
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Constructor : construct with the executor information and the number of launch tasks.</span></li>
</ul>
<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="font-family: 'Trebuchet MS', sans-serif; white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;">public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor , Protos.ExecutorInfo pinUserBoardExecutor ) {<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>this(pinUserProfileExecutor,pinUserBoardExecutor, 5, "http://www.pinterest.com/techcrunch");<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor,Protos.ExecutorInfo pinUserBoardExecutor, int totalTasks, String url) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.pinUserProfileExecutor = pinUserProfileExecutor;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.pinUserBoardExecutor = pinUserBoardExecutor;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.totalTasks = totalTasks;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.crawlQueue = Collections.synchronizedList(new ArrayList<String>());</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.crawlQueue.add(url);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Resource Offers</span></li>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> A resource offer can be resources like CPU, memory etc. From the offers list, get the scalar value of the resources. We need to give our requirements of resources for the tasks while setting the task info.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">for (Protos.Offer offer : list) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>List<Protos.TaskInfo> taskInfoList = new ArrayList<Protos.TaskInfo>();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>double offerCpus = 0;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>double offerMem = 0;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for (Protos.Resource resource : offer.getResourcesList()) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if (resource.getName().equals("cpus")) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>offerCpus += resource.getScalar().getValue();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>} else if (resource.getName().equals("mem")) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>offerMem += resource.getScalar().getValue();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.info("Received Offer : " + offer.getId().getValue() + " with cpus = " + offerCpus + " and mem ="</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>+ offerMem);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create task ID.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.TaskID taskID = Protos.TaskID.newBuilder().setValue(Integer.toString(launchedTasks++)).build();</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create task info by setting task ID, adding resources, setting data and setting executor.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.TaskInfo pinUserProfileTaskInfo = Protos.TaskInfo.newBuilder().setName("task " + taskID.getValue())</span><span style="font-family: Courier New, Courier, monospace;">.setTaskId(taskID).setSlaveId(offer.getSlaveId())</span><span style="font-family: Courier New, Courier, monospace;">.addResources(Protos.Resource.newBuilder().setName("cpus")</span><span style="font-family: Courier New, Courier, monospace;">.setType(</span><span style="font-family: Courier New, Courier, monospace;">Protos.Value.Type.SCALAR)</span><span style="font-family: Courier New, Courier, monospace;">.setScalar(Protos.Value.Scalar</span><span style="font-family: Courier New, Courier, monospace;">.newBuilder().setValue(</span><span style="font-family: Courier New, Courier, monospace;">CPUS_PER_TASK)))</span><span style="font-family: Courier New, Courier, monospace;">.addResources(Protos.Resource.newBuilder().setName("mem")</span><span style="font-family: Courier New, Courier, monospace;">.setType(</span><span style="font-family: Courier New, Courier, monospace;">Protos.Value.Type.SCALAR)</span><span style="font-family: Courier New, Courier, monospace;">.setScalar(Protos.Value.Scalar</span><span style="font-family: Courier New, Courier, monospace;">.newBuilder()</span><span style="font-family: Courier New, Courier, monospace;">.setValue(MEM_PER_TASK)))</span><span style="font-family: Courier New, Courier, monospace;">.setData(ByteString.copyFromUtf8(crawlQueue.get(0)))</span><span style="font-family: Courier New, Courier, monospace;">.setExecutor(Protos.ExecutorInfo.newBuilder(</span><span style="font-family: Courier New, Courier, monospace;">pinUserProfileExecutor))</span><span style="font-family: 'Courier New', Courier, monospace;">.build();</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Launch the tasks through the SchedulerDriver.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">...<span class="Apple-tab-span" style="white-space: pre;"> </span></span> <span style="font-family: Courier New, Courier, monospace;">taskInfoList.add(pinUserProfileTaskInfo);</span><span style="font-family: Courier New, Courier, monospace;">taskInfoList.add(pinUserBoardTaskInfo);</span><span style="font-family: Courier New, Courier, monospace;">}</span><span style="font-family: Courier New, Courier, monospace;">schedulerDriver.launchTasks(offer.getId(), taskInfoList);</span></blockquote>
</blockquote>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Status update</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">This is invoked when the status of a task has changed ie., a slave is lost and so the task is lost, a task finishes and an executor sends a status update saying so.</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">@Override public void statusUpdate(SchedulerDriver schedulerDriver, Protos.TaskStatus taskStatus) {</span><span style="font-family: Courier New, Courier, monospace;">...</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Stop the SchedulerDriver if tasks are finished</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="font-family: 'Trebuchet MS', sans-serif; white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;">if (taskStatus.getState() == Protos.TaskState.TASK_FINISHED) {</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">finishedTasks++;</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">LOGGER.info("Finished tasks : " + finishedTasks);</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">if (finishedTasks == totalTasks) {</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">schedulerDriver.stop();</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">}</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Abort the SchedulerDriver if the tasks are killed, lost or failed</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="font-family: 'Trebuchet MS', sans-serif; white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;">if (taskStatus.getState() == Protos.TaskState.TASK_FAILED<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>|| taskStatus.getState() == Protos.TaskState.TASK_KILLED<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>|| taskStatus.getState() == Protos.TaskState.TASK_LOST) {<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.error("Aborting because the task " + taskStatus.getTaskId().getValue() + " is in unexpected state : "<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>+ taskStatus.getState().getValueDescriptor().getName() + "with reason : " + taskStatus.getReason()<br /><span class="Apple-tab-span" style="white-space: pre;"> </span> .getValueDescriptor()<br /><span class="Apple-tab-span" style="white-space: pre;"> </span> .getName()<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>+ " from source : " + taskStatus.getSource().getValueDescriptor().getName() + " with message : "<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>+ taskStatus.getMessage());<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>schedulerDriver.abort();<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Framework Message</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">This is invoked when an executor sends a message.</span><br />
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Handle your message</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>@Override public void frameworkMessage(SchedulerDriver schedulerDriver, Protos.ExecutorID executorID,</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Protos.SlaveID slaveID, byte[] bytes) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>String data = new String(bytes);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>System.out.println(data);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.info("User Profile Information : " + data);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<span style="font-family: Trebuchet MS, sans-serif;">Complete code is available <a href="https://github.com/SwathiMystery/mesos-pinspider">here</a> with the instructions to run and sample output.</span><br />
Happy Learning! :)Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Los Angeles, CA, USA34.0522342 -118.243684933.2099567 -119.5345784 34.8945117 -116.95279140000001tag:blogger.com,1999:blog-6187879032785791598.post-46985957125851608882014-12-20T17:38:00.002+05:302014-12-20T17:38:44.049+05:30Introduction and getting started with Apache Mesos<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Introduction to Apache Mesos</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">In this era of distributed computing, where we spin up clusters for Hadoop, Storm, Jenkins, Cassandra, etc separately, we are not making effective use of the resources. There would be long pauses in the cluster after entering a burst of information, thus making it very in-efficient. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Now, what if all these frameworks shared the same set of machines and resources, then small slices of time spent waiting for some resources could be granted to other frameworks. This is the concept of Time Sharing. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Apache Mesos is a datacenter operating system and it shares the same philosophy of time-sharing. Mesos is called a datacenter because it hosts different frameworks under a single roof. It is called an operating system because shares many concepts of Linux.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">1. <b>Isolation</b> : Linux creates isolation through processes where, each of these processes has its own file descriptors and its own address space. This is achieved by Linux Containers (wiki:<a href="https://en.wikipedia.org/wiki/LXC">LXC</a>) in Mesos</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">2. <b>Process Scheduler</b> : The processes have accesses to the system resources by balancing the work loads across multiple computing resources, thereby optimizing resources, maximize the throughput, minimizing the response time and avoiding the overhead by any one resource. There are various scheduling algorithms to execute more than one process at a time (wiki:<a href="https://en.wikipedia.org/wiki/Computer_multitasking">Multitasking</a>) and also transmit multiple data streams simultaneously across a single physical channel(wiki:<a href="https://en.wikipedia.org/wiki/Multiplexing">Multiplexing</a>). Mesos uses such scheduling algorithms.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">3. <b>Common Infrastructure</b> : Linux has a set of calls irrespective of filesystems, drivers etc. Similarly, Mesos has a common set of calls which helps in the execution of tasks.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">4.<b> Package Manager</b> : Linux has apt-get, aptitude, synaptic, yum etc that helps in the automation of the process of installing, upgrading, configuring, and removing software. Similarly, Mesos has a recent support for Docker(wiki:<a href="https://en.wikipedia.org/wiki/Docker_(software)">Docker</a>)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Traditionally, distributed systems has 2 components in a non-peer to peer systems.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">1. <b>Coordinator</b> : Generate tasks, send the tasks to worker and receive results from Worker.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">2.<b> Worker</b> : Execute the tasks and send the status and results back to Worker.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">With Mesos, there are three levels <b>Coordinator</b>,<b> Mesos master </b>and<b> Mesos slaves</b> where coordinator negotiates with mesos master and then master decides on partitioning the cluster to distribute the tasks. Thus, we can schedule jobs across the machines, thereby running hadoop, cassandra, spark etc.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">All the distributed systems that run on Mesos are called applications or frameworks and the coordinator is called as scheduler in Mesos vocabulary.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<h3>
<span style="font-family: Trebuchet MS, sans-serif;"><b>How does Mesos work?</b></span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">In summary, Mesos works on a request/offer based model. Whenever, you want to run a job, you send a request. These requests are simplified subset of specification like number of GPUs, RAM etc, at that point of time. Mesos, checks for the request specification and it will reply back with the resource offers of what resources are available on a set of machines. This is non-blocking and has two level of scheduling : Offering and Scheduling.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b>Mesos master</b>: Control the resource allocation to the schedulers</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b>Scheduler</b>: Uses the resource offers to decide which tasks to run and which one to run next.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">More information on Mesos architecture is here : <a href="http://mesos.apache.org/documentation/latest/mesos-architecture/">Mesos Architecture</a></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<h3>
<span style="font-family: Trebuchet MS, sans-serif;"><b>Getting started with mesos</b></span></h3>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">1. Download the tarball from the </span><span style="font-family: Courier New, Courier, monospace;"><a href="http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.21.0/">Mirror</a></span><span style="font-family: Trebuchet MS, sans-serif;"> Apache Mesos v0.21.0 and untar it.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">tar -zvxf mesos-0.21.0.tar.gz </span><br />
<span style="font-family: Courier New, Courier, monospace;">cd mesos-0.21.0/</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">2. Install the dependencies</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">sudo apt-get update</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo apt-get install build-essential openjdk-6-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">3. Building Mesos</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Please make sure it has appropriate permissions while building.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">mkdir build</span><br />
<span style="font-family: Courier New, Courier, monospace;">cd build</span><br />
<span style="font-family: Courier New, Courier, monospace;">../configure</span><br />
<span style="font-family: Courier New, Courier, monospace;">make</span><br />
<span style="font-family: Courier New, Courier, monospace;">make check </span><br />
<span style="font-family: Courier New, Courier, monospace;">make install</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Start Mesos Master</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Start Mesos Slave</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">./bin/mesos-slave.sh --master=127.0.0.1:5050</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Web UI</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">http://127.0.0.1:5050</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Running a test framework in Java</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">./src/examples/java/test-framework 127.0.0.1:5050</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" height="350" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos Home</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuKZ9hqj5is9eHGHq-9bZ9QCkn43eRsos1JooGuSyQsBlFEgZc6EzR2e-vdpEhyphenhyphenZAC374WNxFePRXFAgClXN7JBewD13FKyv79W9DXW7j35mNq26g_mjy3TOyBEpVWEOhSVeN9rH8hJGy5/s1600/mesos_frameworks.png" imageanchor="1" style="margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuKZ9hqj5is9eHGHq-9bZ9QCkn43eRsos1JooGuSyQsBlFEgZc6EzR2e-vdpEhyphenhyphenZAC374WNxFePRXFAgClXN7JBewD13FKyv79W9DXW7j35mNq26g_mjy3TOyBEpVWEOhSVeN9rH8hJGy5/s1600/mesos_frameworks.png" height="306" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos Frameworks</td></tr>
</tbody></table>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKGzMr0GyMQMeVdGpAd-jDmQoRXMMjHO0K0Os2_PPmIA6hU0UIQOZY-9VE4S5Y45HkcVa0ndwl0p8on_TfnuaoWx7WvGNP4QvufKU2uWR9AScYPp7w0g4UEx84cmtbINkqrpZUOhvzR-FW/s1600/mesos_task_execution.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKGzMr0GyMQMeVdGpAd-jDmQoRXMMjHO0K0Os2_PPmIA6hU0UIQOZY-9VE4S5Y45HkcVa0ndwl0p8on_TfnuaoWx7WvGNP4QvufKU2uWR9AScYPp7w0g4UEx84cmtbINkqrpZUOhvzR-FW/s1600/mesos_task_execution.png" height="310" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos executor tasks</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc1vjJ1b33-JvkZiHDEh3kIBK254r5Gb_kPruzqxOyYHr5yItuFdh2bieC5CHSKU25RQR75KESWBwq1qyT6bwlSQvmOKrv1_hCKlFvVnt9qWdTbCzEGI34QAZ09HrSgpHzKYJlC5avH5ep/s1600/mesos_slaves.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc1vjJ1b33-JvkZiHDEh3kIBK254r5Gb_kPruzqxOyYHr5yItuFdh2bieC5CHSKU25RQR75KESWBwq1qyT6bwlSQvmOKrv1_hCKlFvVnt9qWdTbCzEGI34QAZ09HrSgpHzKYJlC5avH5ep/s1600/mesos_slaves.png" height="336" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos Slave Nodes</td></tr>
</tbody></table>
<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: 'Trebuchet MS', sans-serif;">In the next post, let us see how to build our own distributed framework on Apache Mesos.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Happy Learning! :)</span><br />
<br />Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Los Angeles, CA, USA34.0522342 -118.243684933.2099567 -119.5345784 34.8945117 -116.95279140000001tag:blogger.com,1999:blog-6187879032785791598.post-50996102488650622892014-08-05T04:16:00.000+05:302014-08-05T04:16:59.824+05:30Raspberry Pi B+ : Connect to your Pi with no display monitor, keyboard or mouse<span style="font-family: 'Trebuchet MS', sans-serif;"> </span><span style="font-family: 'Trebuchet MS', sans-serif;"> </span><span style="font-family: Trebuchet MS, sans-serif;">The new Raspberry Pi B+ was released on July 14th 2014. The following link does all the justice to know about the improved features of B+. Please r</span><span style="font-family: 'Trebuchet MS', sans-serif;">efer it here : </span><a href="http://makezine.com/2014/07/14/first-look-at-the-new-raspberry-pi-b/" style="font-family: 'Trebuchet MS', sans-serif;">Raspberry Pi B+</a><br />
<br />
<span style="font-family: 'Trebuchet MS', sans-serif;"> </span><span style="font-family: 'Trebuchet MS', sans-serif;"> </span><span style="font-family: Trebuchet MS, sans-serif;">The moment I had this credit card sized thing on my palm, I was so excited to get started with it. I downloaded Raspbian from <a href="http://www.raspberrypi.org/downloads/">Raspberry Pi Downloads Page</a> and unzipped it. </span><span style="font-family: 'Trebuchet MS', sans-serif;">This is the image file 2014-06-20-wheezy-raspbian.img you get after the unzip operation. </span><span style="font-family: 'Trebuchet MS', sans-serif;">I used the SanDisk 8GB Class 4 Micro SD card and dd tool to write to it. </span><span style="font-family: Trebuchet MS, sans-serif;">Refer Installing images on Linux <a href="http://www.raspberrypi.org/documentation/installation/installing-images/linux.md">here</a> </span><span style="font-family: 'Trebuchet MS', sans-serif;">( Please note, I did bs=1M with dd tool while writing). </span><span style="font-family: Trebuchet MS, sans-serif;">This was one of the guide I was following <a href="http://www.raspberrypi.org/help/quick-start-guide/">Quick Start Guide</a></span><br />
<br />
<span style="font-family: 'Trebuchet MS', sans-serif;">But then, wait! Do you need a HDMI TV or display monitor, USB keyboard and a USB mouse to connect to it? Oh really!? </span><span style="font-family: 'Trebuchet MS', sans-serif;">I was not interested to invest any further. </span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span>
<span style="font-family: 'Trebuchet MS', sans-serif;">Here is what I did:</span><br />
<br />
<ol>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">I inserted the Micro SD card into the slot behind. </span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">I had a micro USB cable my for Nexus 5. I used the same for the 5V power supply</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">I had an Ethernet crossover cable already with me for some VPN stuffs. I took one of them and connected one end to the router and the other end to the Pi Ethernet Port (next to 4 USB Ports) </span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Next, I used my laptop to connect to my Pi in the same network.</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">On my laptop, </span></li>
<ol>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Opened Terminal. Typed in </span><span style="font-family: Courier New, Courier, monospace;">$ ifconfig</span><span style="font-family: 'Trebuchet MS', sans-serif;"> to get to know the IP address. </span></li>
</ol>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">For ex: if </span><span style="font-family: Trebuchet MS, sans-serif;"> inet addr:192.168.0.101 then our Pi should have inet addr:192.168.0.*</span></li>
</ul>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Next, for my rescue came the </span><span style="font-family: Courier New, Courier, monospace;">nmap</span><span style="font-family: Trebuchet MS, sans-serif;"> tool. If you do not have it already installed, </span><span style="font-family: Courier New, Courier, monospace;">$ sudo apt-get install nmap </span></li>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Execute </span><span style="font-family: Courier New, Courier, monospace;">$ nmap -T4 -F 192.168.0.*</span><span style="font-family: 'Trebuchet MS', sans-serif;"> on the terminal ( as per the example considered ) This will scan all the hosts which are up and then lists out the open ports. You will find Pi as one among them.</span></li>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">If you like GUI Based, you may like Zenmap and can be installed by </span><span style="font-family: Courier New, Courier, monospace;">$ sudo apt-get install zenmap</span><span style="font-family: 'Trebuchet MS', sans-serif;">.</span></li>
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">After installing, type </span><span style="font-family: Courier New, Courier, monospace;">$ sudo zenmap</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Type Target as </span><span style="font-family: 'Trebuchet MS', sans-serif;">192.168.0.*</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Choose Profile as Quick Scan</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Click on Scan</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">This will list the Raspberry Pi MAC Address and the IP Address</span></li>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Now, you know the IP Address of Pi. The default login for Raspbian is username "<i>pi</i>" with the password "<i>raspberry</i>"</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Time to ssh!</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Need to change your configuration settings after login? Type </span><span style="font-family: Courier New, Courier, monospace;">$ sudo raspi-config</span><span style="font-family: Trebuchet MS, sans-serif;"> and change it accordingly!</span></li>
</ol>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Happy hacking! :)</span></div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Los Angeles, CA, USA34.0522342 -118.243684933.2099567 -119.5345784 34.8945117 -116.95279140000001tag:blogger.com,1999:blog-6187879032785791598.post-38546453695125116362013-06-18T15:28:00.000+05:302013-06-18T15:28:09.289+05:30One Cap to rule 'em all ...<div dir="ltr" style="text-align: left;" trbidi="on">
Wondering which cap it could be?<br />
....Well its Capistrano :D<br />
<br />
I am a fan of Capistrano from way back and we use it for almost all kind of deployments - Hadoop, MongoDB clusters and so on.<br />
If you have not tried Capistrano, you must try it and figure out how you can use for deployments in your environment.<br />
<br />
Its highly configurable - so capify your stuffs!<br />
<br />
Checkout the capified scripts to <b>Deploy a replicated sharded MongoDB cluster on AWS EC2 instances</b> in the following link<br />
<a href="https://github.com/SwathiMystery/deploy_shard_mongodb">https://github.com/SwathiMystery/deploy_shard_mongodb</a><br />
Feel free to experiment, report bugs/ issues and contribute back.<br />
<br />
For more details, follow the link below :<br />
<a href="https://github.com/SwathiMystery/deploy_shard_mongodb/blob/master/README.md#deploy-replicated-sharded-mongodb-cluster">https://github.com/SwathiMystery/deploy_shard_mongodb/blob/master/README.md#deploy-replicated-sharded-mongodb-cluster</a></div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0United States38.272688535980969 -104.414062512.750654035980968 -145.7226565 63.794723035980965 -63.1054685tag:blogger.com,1999:blog-6187879032785791598.post-57956970845589149062013-04-15T22:22:00.000+05:302013-04-18T03:05:15.724+05:30Monitoring S3 uploads for a real time data<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Trebuchet MS, sans-serif;"> If you are working on Big Data and its bleeding edge technologies like Hadoop etc., the primary thing you need is a "dataset" to work on. So, this data can be reviews, blogs, news, social media data (Twitter, Facebook etc), domain specific data, research data, forums, groups, feeds, fire hose data etc. Generally, companies reach the data vendors to fetch such kind of data.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"> Normally, these data vendors dump the data into a shared server kind of environment. For us to use this data for processing with MapReduce and so forth, we move them to S3 for storage first and processing next. Assume, the data belong to social media such as Twitter or Facebook, then the data can be dumped according to the date format directory. Majority of the cases, its the practice.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Also assuming 140-150GB /day being dumped in a hierarchy like 2013/04/15 ie. yyyy/mm/dd format, stream of data, how do you </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- upload them to s3 in the same hierarchy to a given bucket?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- monitor the new incoming files and upload them?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- save the space effectively on the disk?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- ensure the reliability of uploads to s3?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- clean if the logging is enabled to track?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- re-try the failed uploads?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">These were some of the questions, running at the back of my mind, when I wanted to automate the uploads to S3. Also, I wanted 0 human intervention or at-least the least!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">So, I came up with </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- s3sync / s3cmd.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- the python Watcher script by </span><span style="font-family: 'Trebuchet MS', sans-serif;">Greggory Hernandez, here </span><a href="https://github.com/greggoryhz/Watcher" style="font-family: 'Trebuchet MS', sans-serif;">https://github.com/greggoryhz/Watcher</a><span style="font-family: 'Trebuchet MS', sans-serif;"> </span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">A big thanks! This helped me with monitoring part and it works so great!</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">- few of my own scripts.</span><br />
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">What are the ingredients?</span><br />
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Installation of s3sync. I have just used one script of s3cmd here and not s3sync in real. May be in future -- so I have this.</span></li>
<script src="https://gist.github.com/SwathiMystery/5388392.js"></script>
<li><span style="font-family: Trebuchet MS, sans-serif;">Installation of Watcher.</span></li>
<script src="https://gist.github.com/SwathiMystery/5388464.js"></script>
<li><span style="font-family: Trebuchet MS, sans-serif;">My own wrapper scripts.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">cron</span></li>
</ul>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Next, having set up of the environment ready, lets make some common "assumptions".</span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: Trebuchet MS, sans-serif;">Data being dumped will be at /home/ubuntu/data/ -- from there it could be 2013/04/15 for ex.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">s3sync is located at /home/ubuntu</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Watcher repository is at /home/ubuntu</span></li>
</ul>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Getting our hands dirty...</span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Goto Watcher and set the directory to be watched for and corresponding action to be undertaken.</span></li>
<script src="https://gist.github.com/SwathiMystery/5388835.js"></script>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create a script called monitor.sh to upload to s3 in s3sync directory as below.</span></li>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">The variables you may like to change is s3bucket path in "s3path" in monitor.sh</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">This script will upload the new incoming file detected by the watcher script in the reduced redundancy storage format. (you can remove the header -- provided you are not interested to store in RRS format)</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">The script will call s3cmd ruby script to upload recursively and thus maintains the hierarchy ie. yyyy/mm/dd format with files *.*</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">It will delete the file successfully uploaded to s3 from the local path -- to save the disk space.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">The script would not delete the directory, as it will be taken care by yet another script re-upload.sh, which acts as a backup for the failed uploads to be uploaded again to s3.</span></li>
</ul>
<script src="https://gist.github.com/SwathiMystery/5389095.js"></script>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create a script called re-upload.sh which will upload the failed file uploads.</span></li>
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">This script ensures that the files that are left over from monitor.sh (failed uploads -- this chance is very less. May be 2-4 files/day. -- due to various reasons.), will be uploaded to s3 again with the same hierarchy in RRS format.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;"> Post successful upload, deletes the file and hence the directory if empty.</span></li>
</ul>
<script src="https://gist.github.com/SwathiMystery/5389191.js"></script>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Now, more dirtiest work -- Logging and cleaning logs.</span></li>
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">All the "echo" created in monitor.sh can be found in ~/.watcher/watcher.log when the watcher.py is running.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">This log helps us initially and may be later too, to backtrack errors or so.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Call of duty - Janitor for cleaning logs. To do this, we can use cron to run a script at sometime. I was interested to run - Every Saturday at 8.00 AM</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Create a script to clean log as "clean_log.sh" in /home/ubuntu/s3sync</span></li>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Time for cron</span></li>
<script src="https://gist.github.com/SwathiMystery/5389305.js"></script>
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">All set! logging clean happens every Saturday 8.00 AM and re-upload script runs for the previous day, to check if files exist and does the cleaning accordingly.</span></li>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Let's start the script</span></li>
<script src="https://gist.github.com/SwathiMystery/5389359.js"></script>
</ul>
<div>
So, this assures successful uploads to S3. </div>
</div>
</div>
<div>
My bash-fu with truth! ;)</div>
<div>
Happy Learning! :)</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Bangalore, Karnataka, India12.9715987 77.59456269999998312.4764197 76.949115699999979 13.4667777 78.240009699999987tag:blogger.com,1999:blog-6187879032785791598.post-40704086330126898592013-03-19T04:41:00.000+05:302013-03-19T04:45:36.361+05:30Phabricator - pretty()<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Trebuchet MS, sans-serif;">Ah pretty() -- liking it too much! These days I'm into MongoDB so.. :P</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Its like 3am ish now and I know, I'm blabbering ... Gotta sleep!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Ah! please bear with rest of my ramblings!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Last month, I was looking into some of the code review tools!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">You know - reviewing the review tools! ;)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Well, I experimented with reviewboard, gerrit, barkeep.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Gerrit does not support post-commit review. I was looking for, a way to conduct code review, after the commit is pushed to git.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Reviewboard is cool. Its existing from a longer time. I experimented a lot on the cloud. Well, the post-review process was little painful with command line.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Barkeep installation is kind of different. I did not invest much time. But it looks interesting!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">I went with Phabricator, released by Facebook and Open source too.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Wanna contribute ? Fork here <a href="https://github.com/facebook/phabricator/">https://github.com/facebook/phabricator/</a> : else go <a href="http://phabricator.org/">http://phabricator.org/</a></span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Okay I don't wanna geeko-Fi more with operators ;)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Well, Phabricator is used by many companies like Quora (unsure if they are presently using it or not, have a doubt , as I read in some quora thread), DropBox, Path (Android app), Disqus (online commenting system), deviantArt( Painting app -- famous in Chrome web store) .... Ah well, lots more!!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<ul style="text-align: left;">
<li><span style="font-family: Trebuchet MS, sans-serif;">What's cool about Phabricator?</span></li>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">It has complete software tools. Ok!! You want ...</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<ul style="text-align: left;"><ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Code Review ? Differential</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Post Commit Review ? Audits</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Code Browser ? Diffusion</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Bug Tracking ? Maniphest</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Wiki ? Phriction</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Want more ? It's under development... So you gonna get more too!</span></li>
</ul>
</ul>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Well, personally, I prefer using Audits, Diffusion, Maniphest. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Wiki - is used in Confluence/BitBucket -- so for now not essential.</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Code Review -- Pre-commit reveiw process stalls the development in agile kinda environment. So not going with it.</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: Trebuchet MS, sans-serif;">How's the installation?</span></li>
</ul>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">I read a question in Quora that why is it difficult to install Phabricator. I believe its not so difficult too. </span></div>
</div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">May be I will post -- how to install/configure everything with Phabricator. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Please bear with the mentions about Quora, I'm a quor(a)ddict! :D</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: Trebuchet MS, sans-serif;">How to start with Audits?</span></li>
</ul>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">To get started with the workflow, it does take time, probably because its still under development and documentation demands update. But the greatest thing, I love about Phabricator is, its still under the development and yet, functions -- as it purports to be!! That's Amazing! The UI is lovely too!</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">I faced this issue -- If you have huge repository, where codereview has never happened before, how do you start the process with previous commits? </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Unfortunately, you cannot do with existing commits! The commits that occur after installation/configuration of Phabricator with that repository, audits are possible.</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">You will have to create some rules that trigger post commit audits. Its called Herald in Phabricator.</span></div>
</div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">I found a workaround for the old commits -- if you wanna review -- You can browse the repository with Diffusion, goto the module / code you are interested in and click on the commit number. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Upon clicking on commit number, you have the freedom to post in-line comments/review comments, where even the diff is visible. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">After finishing your commenting process, you can cook it with Raise Concern. I also prefer mapping this with a task tracker -- I mean bug tracker ( Maniphest ) -- add the assignee, link to the comments you gave in description and assign. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">This will send a mail to the team.</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Email configuration is one big task. </span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">I used Postfix with SMTP as configuring outbound mail and have updated in github Phabricator.</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Mapping Your Projects/Audits/Team</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">This takes a lot of time and effort too. </span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: Trebuchet MS, sans-serif;">First, you need to create accounts for all. </span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">You will have to configure your remote repository (using git). </span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Map the modules with the module owners and team lead.</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Create herald rule which triggers audits.</span></li>
</ul>
</div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">So, you can set up completely in 2days. But its fun!! Audits can be real fun..!! :)</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">I will really recommend Phabricator! :D</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Happy Tweaking! :)</span></div>
</div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Bangalore, Karnataka, India12.9715987 77.59456269999998312.4764147 76.949115699999979 13.4667827 78.240009699999987tag:blogger.com,1999:blog-6187879032785791598.post-85357152307189137542013-02-20T13:55:00.001+05:302013-02-20T14:01:31.629+05:30Books that every developer must read!<span style="font-family: Trebuchet MS, sans-serif;"> I have a rack-space of books! I just wanted to make a wishlist of all the books which I've read and yet to read! :) </span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">1.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Design Patterns: Elements of Reusable Object-Oriented Software</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Erich Gamma Richard Helm Ralph Johnson John Vlissides</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51kuc0iWoKL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA278_PIkin4,BottomRight,-66,22_AA300_SH20_OU01_.jpg" height="320" width="320" /></span><br />
<span style="font-family: Trebuchet MS, sans-serif;">2.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Structure And Interpretation Of Computer Programs, Second Edition : By Harold Abelson and Gerald Jay Sussman</span><br />
<a href="http://mitpress.mit.edu/sites/default/files/imagecache/booklist_node/9780262510875.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://mitpress.mit.edu/sites/default/files/imagecache/booklist_node/9780262510875.jpg" height="320" width="219" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">3.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Refactoring to Patterns</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Joshua Kerievsky</span><br />
<a href="http://ecx.images-amazon.com/images/I/516pPX8YmvL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/516pPX8YmvL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">4.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Types and Programming Languages</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Benjamin C. Pierce</span><br />
<a href="http://ecx.images-amazon.com/images/I/51aw+2+090L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51aw+2+090L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">5.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Code: The Hidden Language of Computer Hardware and Software</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Charles Petzold</span><br />
<a href="http://ecx.images-amazon.com/images/I/31DR-p2+1fL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/31DR-p2+1fL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">6.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Object-Oriented Analysis and Design with Applications (2nd Edition)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Grady Booch</span><br />
<a href="http://ecx.images-amazon.com/images/I/51l8wcgd-RL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51l8wcgd-RL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">7.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Code Complete: A Practical Handbook of Software Construction, Second Edition</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Steve McConnell</span><br />
<a href="http://ecx.images-amazon.com/images/I/51nWkLCu1SL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51nWkLCu1SL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">8.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">The Design of the UNIX Operating System [Prentice-Hall Software Series]</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Maurice J. Bach</span><br />
<a href="http://ecx.images-amazon.com/images/I/71K140K5BFL._SL500_AA300_.gif" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/71K140K5BFL._SL500_AA300_.gif" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">9.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">The Pragmatic Programmer: From Journeyman to Master</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Andrew Hunt David Thomas</span><br />
<a href="http://ecx.images-amazon.com/images/I/41wy+xyro-L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/41wy+xyro-L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">10.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Practical API Design: Confessions of a Java Framework Architect</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Jaroslav Tulach</span><br />
<a href="http://ecx.images-amazon.com/images/I/51-JO7VmQ4L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="clear: left; font-family: Trebuchet MS, sans-serif; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://ecx.images-amazon.com/images/I/51-JO7VmQ4L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">11.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">The Practice of Programming (Addison-Wesley Professional Computing Series)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Brian W. Kernighan Rob Pike</span><br />
<a href="http://ecx.images-amazon.com/images/I/21WkP5B9mNL._SL500_AA300_.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/21WkP5B9mNL._SL500_AA300_.jpg" height="320" width="320" /></span></a><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">12.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Programming Pearls (2nd Edition)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Jon Bentley</span><br />
<a href="http://ecx.images-amazon.com/images/I/41WonSY9PbL._SL500_AA300_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/41WonSY9PbL._SL500_AA300_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">13.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Writing Secure Code, Second Edition</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Michael Howard David LeBlanc</span><br />
<a href="http://ecx.images-amazon.com/images/I/51EnG5Gj6yL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51EnG5Gj6yL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">14.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Frederick P. Brooks Jr.</span><br />
<a href="http://ecx.images-amazon.com/images/I/51WIpM70FEL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51WIpM70FEL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">15.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Patterns of Enterprise Application Architecture</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Martin Fowler</span><br />
<a href="http://ecx.images-amazon.com/images/I/511D6FdsbXL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/511D6FdsbXL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">16.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Introduction to Functional Programming (Prentice Hall International Series in Computing Science)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Richard Bird</span><br />
<a href="http://ecx.images-amazon.com/images/I/41ESYc6S6GL._SL500_AA300_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/41ESYc6S6GL._SL500_AA300_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">17.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">The Art of Computer Programming</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Donald E. Knuth</span><br />
<a href="http://ecx.images-amazon.com/images/I/41gCSRxxVeL._SL500_AA300_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/41gCSRxxVeL._SL500_AA300_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">18.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Effective Java (2nd Edition)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Joshua Bloch</span><br />
<a href="http://ecx.images-amazon.com/images/I/51EHjWvpClL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51EHjWvpClL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">19.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Thinking in Java (4th Edition) </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Bruce Eckel</span><br />
<a href="http://ecx.images-amazon.com/images/I/31bmX1EXVZL._SL500_AA300_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/31bmX1EXVZL._SL500_AA300_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">20.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Programmers at Work: Interviews With 19 Programmers Who Shaped the Computer Industry (Tempus)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Susan Lammers</span><br />
<a href="http://ecx.images-amazon.com/images/I/51-ZmoesdsL._SL500_AA300_.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/51-ZmoesdsL._SL500_AA300_.jpg" height="320" width="320" /></span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span>
<span style="font-family: 'Trebuchet MS', sans-serif;">21.</span><br />
<div class="buying" style="padding: 0.25em 0em;">
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;">Coders at Work: Reflections on the Craft of Programming Peter Seibel</span></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://ecx.images-amazon.com/images/I/41JzhuA--iL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://ecx.images-amazon.com/images/I/41JzhuA--iL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" height="320" width="320" /></span></a></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;">Well, I compiled all the aforementioned from Amazon. I will keep appending to list, if I remember :)</span></div>
<div class="buying" style="padding: 0.25em 0em;">
<span style="font-family: Trebuchet MS, sans-serif;">Happy Learning! :)</span></div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-14555738039831673512013-02-11T17:23:00.000+05:302013-02-14T16:45:20.281+05:30Hadoop Hangover: How-to launch a hadoop cluster CDH4 [MRv1 / YARN + Ganglia] using Apache Whirr<br />
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;"><span style="line-height: 25px;"> This post is about how-to launch a CDH4 MRv1 or CDH4 Yarn cluster on EC2 instances. It's said that you can launch a cluster with the help of Whirr and in a matter of 5 minutes! This is very true if and only if everything works out well! ;) </span></span><br />
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;"><span style="line-height: 25px;"></span></span><br />
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;">Hopefully, this article helps you in that regard.</span><br />
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;">So, let's row the boat...</span><br />
<ul>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">Download the stable version of Apache Whirr ie. whirr-0.8.1.tar.gz from the following link <a href="http://apache.techartifact.com/mirror/whirr/stable/whirr-0.8.1.tar.gz">whirr-0.8.1.tar.gz</a></span></span></li>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">Extract from the tarball and generate the key </span></span></li>
<script src="https://gist.github.com/SwathiMystery/4753960.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">Generate the key</span></li>
<script src="https://gist.github.com/SwathiMystery/4753989.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">Make a properties file to launch the cluster with that configuration.</span></li>
<script src="https://gist.github.com/SwathiMystery/4753999.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">Now let me tell you how to avoid getting headaches!</span></li>
</ul>
<div>
<ul><ul>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">cluster name: Keep your cluster name simple. Avoid testCluster, testCluster1 etc. ie. No Caps, numerics..</span></span></li>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">Decide on the number of datanodes you want judiciously.</span></span></li>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">Your launch may not be successful, if java is not installed. Make sure the image has Java. However, this properties file takes care of that.</span></span></li>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">It will be good to go ahead with MRv1 for now and later switch to MRv2, when we get a production stable release.</span></span></li>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">This is the minimal set of configurations for launching a Hadoop cluster. But, you can do a lot performance tuning upon this.</span></span></li>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">I had launched this cluster from an ec2 instance, Initially i faced errors, regarding user. Setting the configuration below, solved the problem.</span></span></li>
<script src="https://gist.github.com/SwathiMystery/4754004.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">Set proper permissions for ~/.ssh and whirr-0.8.1 folder before launching.</span></li>
</ul>
</ul>
<ul>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;"> Well, we are ready to launch the cluster. Name the properties file as "whirr_cdh.properties".</span></span></li>
<script src="https://gist.github.com/SwathiMystery/4754006.js"></script>
</ul>
</div>
<div>
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;">In the console you can see, links to Namenode and JobTracker Web UI. It also prints how to ssh to the instances in the end.</span><br />
<br />
<ul>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;">Now, you should be having the files generated. </span></span><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">You will be able to see these files: </span><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">instances, hadoop-proxy.sh and hadoop-site.xml</span></li>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">Starting the proxy</span></li>
<script src="https://gist.github.com/SwathiMystery/4754019.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">Open another terminal, and type</span></li>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">You should be able to access the HDFS.</span></li>
<script src="https://gist.github.com/SwathiMystery/4754021.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;">You can alternatively download hadoop tarball and launch with </span></li>
<script src="https://gist.github.com/SwathiMystery/4754031.js"></script>
<li><span style="color: #444444; font-family: 'Trebuchet MS', sans-serif;"> Okay! So I know that you will not be satisfied unless you a web UI</span></li>
<script src="https://gist.github.com/SwathiMystery/4952132.js"></script>
</ul>
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;">So, we are good to go! </span><br />
<ul>
<li><span style="color: #444444;"><span style="font-family: Trebuchet MS, sans-serif;"> If you want to launch MRv2, use this.</span></span></li>
<script src="https://gist.github.com/SwathiMystery/4754048.js"></script>
</ul>
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;">and the same process! </span><br />
<span style="color: #444444; font-family: Trebuchet MS, sans-serif;">Happy Learning! :)</span><br />
<div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-40912081168361967512012-12-18T17:36:00.001+05:302012-12-18T22:53:37.439+05:30FUSE on Amazon S3<span style="font-family: 'Trebuchet MS', sans-serif;"><b>FUSE</b>: File System In User Space, hosted on sourceforge, a well known open source project <a href="http://fuse.sourceforge.net/">http://fuse.sourceforge.net/</a></span><br />
<span style="font-family: Trebuchet MS, sans-serif;">You either put the files in S3 bucket directly or in the mount point, both will always be in the same hierarchy and in Sync. The best thing is that any arbitrary program can just point to this mount point and perform simple/ normal commands, rather than file system specific commands.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Here is a small documentation about how we can achieve this.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">1. Check out the code from google code.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">2. Switch to the working directory</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ cd s3fs</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ ls</span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">AUTHORS autogen.sh ChangeLog configure.ac COPYING doc INSTALL Makefile.am NEWS README src test</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">3. Now same old ritual of configure , make and install.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">To run the subsequent command you need autoconf. So make sure you have it by running the following command.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo apt-get install autoconf</span><span style="font-family: Courier New, Courier, monospace;"><br /></span><span style="font-family: Courier New, Courier, monospace;">$ autoreconf --install </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Trebuchet MS', sans-serif;">It is silently notifying you that you lack the libraries. Time to get them installed...</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Getting back...</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ ./configure --prefix=/usr</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">$ make</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo make install</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">4. Done with the Installation process.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Cross-check:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ /usr/bin/s3fs </span> </blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">s3fs: missing BUCKET argument</span><span style="font-family: Courier New, Courier, monospace;">Usage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">5. Add the following line to your ~/.bashrc file and source it.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">export s3fs=/usr/bin/s3fs</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$source ~/.bashrc</span><span style="font-family: Courier New, Courier, monospace;">$ s3fs </span><span style="font-family: Courier New, Courier, monospace;">s3fs: missing BUCKET argument</span><span style="font-family: Courier New, Courier, monospace;">Usage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">6. Install s3cmd. Many of you must be using this tool to interact with s3.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo apt-get install s3cmd</span><span style="font-family: Courier New, Courier, monospace;">$ s3cmd --configure </span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;">This will configure with the S3 account using Access and Secret Key.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b>Configuring FUSE</b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;">1. First set use_allow_other for others to use. Uncomment in fuse.conf</span><br />
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">$ vi /etc/fuse.conf</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">2. Set the AcessKey:SecretKey in the format in passwd-s3fs file</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo vi /etc/passwd-s3fs</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo chmod 640 /etc/passwd-s3fs</span></blockquote>
<span style="font-family: 'Trebuchet MS', sans-serif;">3. Created a bucket called "s3dir-sync" for this experiment.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ s3cmd ls</span><span style="font-family: Courier New, Courier, monospace;">2012-12-18 09:23 s3://s3dir-sync</span></blockquote>
<span style="font-family: 'Trebuchet MS', sans-serif;">4. Creating a mount point where you want to dump/place the files and keep them in sync with the S3 bucket. Create as root user.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo mkdir -p /mnt/s3Sync</span><span style="font-family: Courier New, Courier, monospace;">$ sudo chmod 777 /mnt/s3Sync</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">5. With s3fs, as a root user.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo s3fs s3dir-sync -o default_acl=public-read -o allow_other /mnt/</span><span style="font-family: 'Courier New', Courier, monospace;">s3Sync</span><span style="font-family: Courier New, Courier, monospace;">/</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;">Cross-check:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ mount -l</span><span style="font-family: Courier New, Courier, monospace;">s3fs on /mnt/s3Sync type fuse.s3fs (rw,nosuid,nodev,allow_other)</span></blockquote>
<span style="font-family: 'Trebuchet MS', sans-serif;">If you try mounting again, you will get the following Warning</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">mount: according to mtab, s3fs is already mounted on /mnt/s3Sync</span></blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">6. I created a directory structure of </span><br />
<span style="font-family: Courier New, Courier, monospace;">/mnt/s3Sync/</span><br />
<span style="font-family: Courier New, Courier, monospace;">-> 2012/12/18$ more test.txt</span><br />
<span style="font-family: Courier New, Courier, monospace;">This is a check file to sync with the s3dir-sync.</span><br />
<span style="font-family: Courier New, Courier, monospace;">Blah..!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The same is synced in the bucket "s3dir-sync"</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Cross-Check: </span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ s3cmd ls s3://s3dir-sync</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">DIR s3://s3dir-sync/2012/</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">2012-12-18 09:57 0 s3://s3dir-sync/2012</span></blockquote>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<span style="font-family: Trebuchet MS, sans-serif;">Happy Learning! :)</span>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-73846761635761566232012-12-10T14:59:00.000+05:302012-12-26T17:36:23.349+05:30Get your wireless working on DELL Inspiron 5220<br />
<span style="font-family: "Trebuchet MS",sans-serif;">I brought a new DELL Inspiron 5220. It's amazing! <br />Configuration : </span><br />
<ul>
<li><span style="font-family: "Trebuchet MS",sans-serif;">3rd Generation i5 Processor</span></li>
<li><span style="font-family: "Trebuchet MS",sans-serif;">4GB RAM</span></li>
<li><span style="font-family: "Trebuchet MS",sans-serif;">1TB Hard Disk</span></li>
<li><span style="font-family: "Trebuchet MS",sans-serif;">15" Screen</span></li>
<li><span style="font-family: "Trebuchet MS",sans-serif;">1GB Graphics </span></li>
</ul>
<span style="font-family: "Trebuchet MS",sans-serif;"><br />It ships with Windows 8! ;)<br />However, made a dual boot upon it. Although BIOS looked different this time!!<br /><br />Well,<br />I'm working on Ubuntu :)<br />Release : 11.10 (Oneiric)<br />Kernel Linux : 3.0.0-28-generic<br />GNOME 3.2.1<br /><br />But Wi-Fi was not getting detected. This was not unusual, as I had set this up in earlier Dell models.<br />Well, the remedy is easy.<br /><b>Step 1</b>: Make sure that you can witness the Device Card and the ID. Especially, the Network !!<br />Type in the following command.</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ lspci -nnk | grep Network<br />08:00.0 Network controller [0280]: Intel Corporation Device [8086:0887] (rev c4)</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;"><br /><b>Step 2</b>: Figure out which kernel version. This is because the driver which we will be installing works on 2.6.37 or higher</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ uname -a<br />Linux Swathi 3.0.0-28-generic #45-Ubuntu SMP Wed Nov 14 21:57:26 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;"><br /><b>Step 3</b>: Install the network manager</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Trebuchet MS",sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get install network-manager*</span></span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;"><br /><b>Step 4</b>: Install few packages</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get install build-essential linux-headers</span></blockquote>
<b style="font-family: 'Trebuchet MS', sans-serif;">Step 5: </b><span style="font-family: 'Trebuchet MS', sans-serif;">Check the output of</span><br />
<span style="font-family: Courier New, Courier, monospace;"> $ dmesg</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">If it outputs the failure of firmware file, then its time to download the .ucode and place it in /lib/firmware.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b>Reboot</b>. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">It should be working. If not, try <b>Step 6</b>.</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"><br /><b>Step 6</b>: Download the compat wireless tarball from this location<br /><a href="http://linuxwireless.org/download/compat-wireless-2.6/compat-wireless-2012-05-10-p.tar.bz2">http://linuxwireless.org/download/compat-wireless-2.6/compat-wireless-2012-05-10-p.tar.bz2</a><br /><br />Extract from the tarball</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ tar -xvf <path_to_compat_wireless_bz2><br />$ cd <extracted_path_compat_wireless></span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;"><br />Installing the packages</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Trebuchet MS",sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">$ make</span></span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">$ sudo make install</span></span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;"><br />After this command it will show on the console the command to disable bluetooth, ethernet and Wi-Fi. Type in the 3 commands.<br />Place this module into the kernel.</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Trebuchet MS",sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">$ sudo modprobe alx</span></span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;">The Ethernet LAN should be detected.<br />Add the driver module into this file : <span style="font-family: "Courier New",Courier,monospace;">/etc/modules</span><br /><i>Append</i> the following lines. Don't touch the rest. This will enable this module while restarting the system as it loads the module.</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Trebuchet MS",sans-serif;">$ sudo vi /etc/modules</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;">#E2200 support</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;">alx</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"></span></blockquote>
<span style="font-family: "Trebuchet MS",sans-serif;"><br /><b>Reboot</b> your machine.<br />You must witness "Wi-Fi Networks Available!" notification on you desktop :)<br />Happy Learning! :)</span>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-37405182491519444122012-11-21T22:27:00.001+05:302013-02-02T23:06:22.206+05:30Starfish : Hadoop Performance Tuning Tool<b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"> Its been a long time blogging... lapse of 3-4months or so... :(</span><span style="vertical-align: baseline;"> </span><span style="vertical-align: baseline; white-space: pre-wrap;">Well, I thought of writing about an awesome tool, I was playing with 4 months ago, for performance tuning in Hadoop, called “</span><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Starfish</span><span style="vertical-align: baseline; white-space: pre-wrap;">”.</span><br /><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">What is Starfish? </span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">Starfish is a Self-tuning System For Big Data Analytics. Its an open source project hosted at GitHub. </span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">Github Link: </span></span><span style="background-color: white; color: #222222; font-family: arial, sans-serif;"> </span><a href="https://github.com/jwlent55/Starfish" style="background-color: white; color: #1155cc; font-family: arial, sans-serif;" target="_blank">https://github.com/jwlent55/<wbr></wbr>Starfish</a><span style="font-family: Trebuchet MS, sans-serif;"><br /><span style="vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">What is the need for Starfish?</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">Need for Performance!!</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">What it does and what are its components?</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">It enables Hadoop users and applications to get good performance automatically.</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">It has three main components. </span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">1. Profiler</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">2. What-if Engine</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;">3. Optimizer</span><br /><span style="vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">1. Job Profile / Profiler :</span></span></b><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Profile is a concise statistical summary of MR Job execution.</span></span></b></li>
<li dir="ltr" style="list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">This profiling is based on the dataflow and cost estimation of MR Job.</span></span></b></li>
<li dir="ltr" style="list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Data Flow estimation would be considered with the number of bytes of <K,V> pairs processed during a job’s execution.</span></span></b></li>
<li dir="ltr" style="list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Cost estimation would be considered with execution time at the level of tasks and phases within the tasks for a MR job execution. (Basically, the resource usage and execution time)</span></span></b></li>
<li dir="ltr" style="list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">The performance models consider the above two and the configuration parameters associated with the MR Job.</span></span></b></li>
<li dir="ltr" style="list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Space of configuration choices:</span></span></b></li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Number of map tasks</span></span></b></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Number of reduce tasks</span></span></b></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Partitioning of map outputs to reduce tasks</span></span></b></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Memory allocation to task-level buffers</span></span></b></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Multiphase external sorting in the tasks</span></span></b></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Whether output data from tasks should be compressed</span></span></b></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">Whether combine function should be used ...</span></span></b></li>
</ul>
</ol>
<b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><span style="background-color: white; font-family: 'Trebuchet MS', sans-serif; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">job j = < program p, data d, resources r, configuration c ></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Thus, we can tell performance is a function of a job j.</span><br /><span style="background-color: white; font-family: 'Trebuchet MS', sans-serif; font-style: italic; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">perf = F(p,d,r,c)</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Job profile is generated by Profiler through measurement or by the What-if Engine through estimation.</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">2. What-if Engine:</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">The What-if Engine uses a mix of simulation and model-based estimation at the phase level of MapReduce job execution, in order to predict the performance of a MapReduce job before executed on a Hadoop cluster.</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">It estimates the perf using properties of p, d, r, and c.</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">ie. Given profile for job j = <p, d1, r1, c1></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;"> Estimate profile for job j' = <p, d2, r2, c2></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">It has white box models consisting detailed set of equations for Hadoop.</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Example:</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Input data properties</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Dataflow statistics</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Configuration parameters</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">⇒ Calculate dataflow in each task phase in a map task</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">3. Optimizer:</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">It finds the optimal configuration settings to use for executing a MapReduce job. It recommends and can also run with the recommended job configuration settings.</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Benchmark:</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Normal Execution:</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Program : WordCount</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Data Size : 4.45GB</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Time taken to complete the job : 8m 5s</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Starfish Profiling and Optimized Execution:</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Program : WordCount</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Data Size: 4.45GB</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Time taken to complete the job : 4m 59s</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Executed with cluster of 1 Master, 3 Slave nodes</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">What’s achieved?</span></b><br />
<ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Perform in-depth job analysis with profiles</span></li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Predict the behavior of hypothetical job executions</span></li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Optimize arbitrary MapReduce programs</span></li>
</b></ul>
<b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;">
<span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Installation ??</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">It’s pretty easy to install. </span></b><br />
<ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Prerequisites : </span></li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Hadoop Cluster of 0.20.2 or 0.20.203.0 should be up and running. Tested for Cloudera Distributions.</span></li>
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Java JDK should be installed.</span></li>
</ul>
</b></ul>
<b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;">
<br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Download from the repository</span></li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><ul style="display: inline !important; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="display: inline !important; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">git clone from the repository </span><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;"><a href="https://github.com/hherodotou/Starfish">https://github.com/hherodotou/Starfish</a></span></li>
</ul>
</b></li>
</ul>
</ul>
<ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;"><ul>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">or download the tarball from here </span><a href="http://www.cs.duke.edu/starfish/files/starfish-0.3.0.tar.gz"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">http://www.cs.duke.edu/starfish/files/starfish-0.3.0.tar.gz</span></a></li>
</ul>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Compile the source code</span><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Compile the entire source code and create the jar files:</span></li>
</ul>
<div dir="ltr" style="margin: 0pt 4pt 0pt 72pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">ant</span></span></div>
<br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Execute all available JUnit tests and verify the code was compiled successfully:</span></li>
</ul>
<div dir="ltr" style="margin: 0pt 4pt 0pt 36pt; text-indent: 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">ant test</span></span></div>
<br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><div dir="ltr" style="margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">Generate the javadoc documentation in </span><span style="vertical-align: baseline; white-space: pre-wrap;">docs/api</span><span style="vertical-align: baseline; white-space: pre-wrap;">:</span></div>
</li>
</ul>
<div dir="ltr" style="margin: 0pt 4pt 0pt 36pt; text-indent: 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">ant javadoc</span></span><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span></div>
</b></li>
</ul>
<div dir="ltr" style="font-family: 'Trebuchet MS', sans-serif; margin: 0pt 4pt 0pt 36pt; text-indent: 36pt;">
<span style="background-color: #f0f0f0; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt;">
<span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Ensure that in </span><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">~/.bashrc</span><span style="font-family: Trebuchet MS, sans-serif;">,</span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">JAVA_HOME </span><span style="font-family: Trebuchet MS, sans-serif;">and </span><span style="font-family: Courier New, Courier, monospace;">HADOOP_HOME</span><span style="font-family: Trebuchet MS, sans-serif;"> </span></span><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">environment variables are set.</span></div>
<span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">BTrace Installation in the Slave Nodes</span></div>
</li>
</ul>
<div dir="ltr" style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt; text-indent: 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>After the compilation, btrace directory created will contain all the classes and the jars. These must be shipped to the slave nodes.</span></div>
<br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><div dir="ltr" style="margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">Create a file (in Master node) “</span><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">slaves_list.txt”</span></div>
</li>
</ul>
<div dir="ltr" style="font-family: 'Trebuchet MS', sans-serif; margin: 0pt 4pt 0pt 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">This file should contain the slave node IP addresses or the hostnames. Make sure the hostnames are updated in the Master node ie. /etc/hosts (IP address and their respective slave hostname).</span></div>
<div dir="ltr" style="font-family: 'Trebuchet MS', sans-serif; margin: 0pt 4pt 0pt 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">Example :</span></div>
<div dir="ltr" style="margin: 0pt 4pt 0pt 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">$vi slaves_list.txt</span></span></div>
<div dir="ltr" style="margin: 0pt 4pt 0pt 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">slave1</span></span></div>
<div dir="ltr" style="margin: 0pt 4pt 0pt 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">slave2 </span></span></div>
<div dir="ltr" style="margin: 0pt 4pt 0pt 36pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">slave3</span></span></div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"><br /></span><b id="internal-source-marker_0.7587950562592596" style="font-weight: normal;"><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><div dir="ltr" style="margin-bottom: 0pt; margin-right: 4pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">Set the global profile parameter in </span><span style="vertical-align: baseline; white-space: pre-wrap;">bin/config.sh</span><span style="vertical-align: baseline; white-space: pre-wrap;"> </span></div>
</li>
</ul>
<br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; margin-left: -24px; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">SLAVES_BTRACE_DIR</span><span style="vertical-align: baseline; white-space: pre-wrap;">: BTrace installation directory at the slave nodes. Please specify the </span><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">full path</span><span style="vertical-align: baseline; white-space: pre-wrap;"> and ensure you have the appropriate write permissions. The path will be created if it doesn't exist.</span></li>
<li dir="ltr" style="list-style-type: disc; margin-left: -24px; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">CLUSTER_NAME</span><span style="vertical-align: baseline; white-space: pre-wrap;">: A descriptive name for the cluster. Do not include spaces or special characters in the name.</span></li>
<li dir="ltr" style="list-style-type: disc; margin-left: -24px; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">PROFILER_OUTPUT_DIR</span><span style="vertical-align: baseline; white-space: pre-wrap;">: The local directory to place the collected logs and profile files. Please specify the </span><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">full path</span><span style="vertical-align: baseline; white-space: pre-wrap;"> and ensure you have the appropriate write permissions. The path will be created if it doesn't exist.</span></li>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Run the script</span></li>
</ul>
<div dir="ltr" style="margin-bottom: 0pt; margin-left: 72pt; margin-top: 0pt;">
<span style="font-family: Courier New, Courier, monospace;"><span style="vertical-align: baseline; white-space: pre-wrap;"> </span><span style="vertical-align: baseline; white-space: pre-wrap;">bin/install_btrace.sh <absolute_path_slaves_list.txt></span></span></div>
<br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: square; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">This will copy the btrace jars in the </span><span style="vertical-align: baseline; white-space: pre-wrap;">SLAVES_BTRACE_DIR </span><span style="vertical-align: baseline; white-space: pre-wrap;">of the slave nodes.</span></li>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">This is all with the installation.</span><br /><span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Execution is followed by</span><br /><ul style="font-family: 'Trebuchet MS', sans-serif; margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Profiling</span></li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Job Analysis</span></li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">What-if analysis</span></li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Optimisation </span></li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Follow for execution from this link </span><a href="http://www.cs.duke.edu/starfish/tutorial/profile.html"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">http://www.cs.duke.edu/starfish/tutorial/profile.html</span></a><span style="vertical-align: baseline; white-space: pre-wrap;"></span></li>
</ul>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span></span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;"> The link </span><a href="http://www.cs.duke.edu/starfish/tutorial/" style="font-family: 'Trebuchet MS', sans-serif;"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">http://www.cs.duke.edu/starfish/tutorial/</span></a><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;"> is a great source to get started with both installation and execution. The documentation is equally great!</span><br /><span style="font-family: 'Trebuchet MS', sans-serif; vertical-align: baseline; white-space: pre-wrap;">Happy Learning! :)</span></b></b><br />
<div>
</div>
Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-34349088100852876942012-08-08T18:22:00.000+05:302012-08-12T15:47:05.606+05:30Tweak Ganglia With Hadoop<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: "Trebuchet MS",sans-serif;">
<span style="font-size: small;"> Its very important to monitor all the machines in the cluster in terms of OS health, bottlenecks, performance hits and so on. There are numerous tools available which spits out huge number of graphs and statistics. But to the administrator of the cluster, only the prominent stats which seriously affects the performance or the health of the cluster, should be portrayed.<br /> Ganglia, fits the bill.<br />Introduction to Ganglia:<br /> Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It relies on a multicast-based listen/announce protocol to monitor state within clusters and uses a tree of point-to-point connections amongst representative cluster nodes to federate clusters and aggregate their state. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency.<br /> The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on over 500 clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.<br /><br />Having known bits of the fact what Ganglia is, there are some pure terminologies to be known before the kick start.<br />1. Node : Generally, it's a machine targeted to perform a single task with its (1-4)core processor.<br />2. Cluster : Cluster consists of a group of nodes.<br />3. Grid : Grid consists of group of clusters.<br /><br /><b>Heading towards Ganglia...</b><br /> The ganglia system is comprised of two unique daemons, a PHP-based web frontend and a few other small utility programs.</span></div>
<div style="font-family: "Trebuchet MS",sans-serif;">
<span style="font-size: small;"><br />- <b>Ganglia Monitoring Daemon(gmond)</b><br /> Gmond is a multi-threaded daemon which runs on each cluster node you want to monitor.Its responsible for monitoring changes in host state, announcing relevant changes, listening to the state of all other ganglia nodes via a unicast or multicast channel and answer requests for an XML description of the cluster state. Each gmond transmits an information in two different ways unicasting/multicasting host state in external data representation (XDR) format using UDP messages or sending XML over a TCP connection.<br /> </span></div>
<div style="font-family: "Trebuchet MS",sans-serif;">
<span style="font-size: small;">- <b>Ganglia Meta Daemon(gmetad)</b><br /> Ganglia Meta Daemon ("gmetad") periodically polls a collection of child data sources, parses the collected XML, saves all numeric, volatile metrics to round-robin databases and exports the aggregated XML over TCP sockets to clients. Data sources may be either "gmond" daemons, representing specific clusters, or other "gmetad" daemons, representing sets of clusters. Data sources use source IP addresses for access control and can be specified using multiple IP addresses for failover. The latter capability is natural for aggregating data from clusters since each "gmond" daemon contains the entire state of its cluster.<br /> </span></div>
<div style="font-family: "Trebuchet MS",sans-serif;">
<span style="font-size: small;">- <b>Ganglia PHP Web Frontend</b><br /> The Ganglia web frontend provides a view of the gathered information via real-time dynamic web pages to the system administrators.<br /><br /><b>Setting up Ganglia:</b><br /> Assume the following:<br />- Cluster : "HadoopCluster"<br />- Nodes : "Master", "Slave1", "Slave2". {Considered only 3 nodes for examples. Similarly many nodes/slaves can be configured.}<br />- Grid : "Grid" consists of "HadoopCluster" for now.<br /><br /> gmond is supposed to be installed on all the nodes ie. "Master", "Slave1", "Slave2". gmetad and web-frontend will be on Master. On, the Master node, we can see all the statistics in the web-frontend. However, we can have a dedicated server for web-frontend too.<br /><br /><b>Step 1</b>: Install gmond on "Master", "Slave1" and "Slave2"<br /> Installing Ganglia can be done by downloading the respective tar.gz, extracting, configure, make and install. But, why to reinvent the wheel. Let's go by installing the same with repository.<br /><br />OS: Ubuntu 11.04<br />Ganglia version 3.1.7<br />Hadoop version CDH3 hadoop-0.20.2<br /><br />Update your repository packages.<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get update</span><br /><br />Installing depending packages.<br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev</span><br /><br />Installing gmond.<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get install ganglia-monitor</span><br /><br />Making changes in /etc/ganglia/gmond.conf<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo vi /etc/ganglia/gmond.conf</span></span><br />
<blockquote>
<span style="font-size: small;"><span style="font-size: x-small;">/* This configuration is as close to 2.5.x default behavior as possible </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> The values closely match ./gmond/metric.h definitions in 2.5.x */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">globals { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> daemonize = yes </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> setuid = no</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> user = ganglia </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> debug_level = 0 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> max_udp_msg_len = 1472 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mute = no </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> deaf = no </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> host_dmax = 0 /*secs */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> cleanup_threshold = 300 /*secs */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> gexec = no </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> send_metadata_interval = 0 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* If a cluster attribute is specified, then all gmond hosts are wrapped inside </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> * of a <CLUSTER> tag. If you do not specify a cluster tag, then all <HOSTS> will </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> * NOT be wrapped inside of a <CLUSTER> tag. */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">cluster { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> name = "HadoopCluster" </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> owner = "Master" </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> latlong = "unspecified" </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> url = "unspecified" </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* The host section describes attributes of the host, like the location */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">host { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> location = "unspecified" </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* Feel free to specify as many udp_send_channels as you like. Gmond </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> used to only support having a single channel */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">udp_send_channel { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> host = Master</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8650</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> ttl = 1 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* You can specify as many udp_recv_channels as you like as well. */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">udp_recv_channel { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"># mcast_join = 239.2.11.71</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8650 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"># bind = 239.2.11.71 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* You can specify as many tcp_accept_channels as you like to share </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> an xml description of the state of the cluster */ </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">tcp_accept_channel { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8649</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span>...</span></blockquote>
<span style="font-size: small;">About the configuration changes:<br style="font-family: "Trebuchet MS",sans-serif;" /> Check the globals once. In the cluster {}, change the name of the cluster as assumed to "HadoopCluster" from unspecified, owner to "Master"( can be your organisation/admin name), latlong,url can be specified according to your location. No harm in keeping them unspecified. <br /> As said, gmond communicates using UDP messages or sending XML over a TCP connection. So, lets get this clear!</span><br />
<blockquote class="tr_bq">
<span style="font-size: small;"><span style="font-size: x-small;">udp_send_channel { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> host = Master</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8650</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> ttl = 1 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span>...</span></blockquote>
<span style="font-size: small;">means that host recieving at the end point will be "Master" (where Master is the hostname with associated IP address. Add all the hostnames in /etc/hosts with the respective IPs). The port at which it accepts is 8650.<br />Since, gmond is configured now at "Master", the UDP recieve channel is 8650.</span><br />
<blockquote class="tr_bq">
<span style="font-size: small;"><span style="font-size: x-small;">udp_recv_channel { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"># mcast_join = 239.2.11.71</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8650 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"># bind = 239.2.11.71 </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span></span></blockquote>
<span style="font-size: small;">All the XML description which could be hadoop metrics, system metrics etc. is accepted at port:8649</span><br />
<blockquote class="tr_bq">
<span style="font-size: small;"><span style="font-size: x-small;">tcp_accept_channel { </span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8649</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">} </span>..</span></blockquote>
<span style="font-size: small;"><br />Starting Ganglia:<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/ganglia-monitor start</span><br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">$ telnet Master 8649</span><br />Output should contain XML format.<br /><br />This means gmond is up.<br /><span style="font-family: "Courier New",Courier,monospace;">$ ps aux | grep gmond</span><br />shows gmond <br /><br />Installing gmond on Slave machines is same with gmond.conf being,</span><br />
<blockquote>
<span style="font-size: small;"><span style="font-size: x-small;">/* This configuration is as close to 2.5.x default behavior as possible</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> The values closely match ./gmond/metric.h definitions in 2.5.x */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">globals {</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> daemonize = yes</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> setuid = no</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> user = ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> debug_level = 0</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> max_udp_msg_len = 1472</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mute = no</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> deaf = no</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> host_dmax = 0 /*secs */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> cleanup_threshold = 300 /*secs */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> gexec = no</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> send_metadata_interval = 0</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">}</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* If a cluster attribute is specified, then all gmond hosts are wrapped inside</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> * of a <CLUSTER> tag. If you do not specify a cluster tag, then all <HOSTS> will</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> * NOT be wrapped inside of a <CLUSTER> tag. */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">cluster {</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> name = "HadoopCluster"</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> owner = "Master"</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> latlong = "unspecified"</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> url = "unspecified"</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">}</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* The host section describes attributes of the host, like the location */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">host {</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> location = "unspecified"</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">}</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* Feel free to specify as many udp_send_channels as you like. Gmond</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> used to only support having a single channel */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">udp_send_channel {</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> host = Master</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8650</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> ttl = 1</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">}</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* You can specify as many udp_recv_channels as you like as well. */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">udp_recv_channel {</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># mcast_join = 239.2.11.71</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8650</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> # bind = 239.2.11.71</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">}</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;">/* You can specify as many tcp_accept_channels as you like to share</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> an xml description of the state of the cluster */</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">tcp_accept_channel {</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> port = 8649</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">}</span>....</span></blockquote>
<span style="font-size: small;"><br /><br />Starting Ganglia:<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/ganglia-monitor start</span><br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">$ telnet Slave1 8649</span><br style="font-family: "Courier New",Courier,monospace;" />Output should contain XML format<br /><br /><br />This means gmond is up.<br /><span style="font-family: "Courier New",Courier,monospace;">$ ps aux | grep gmond</span><br />shows gmond <br /><br /><b>Step 2</b> : Installing gmetad on Master.<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get install ganglia-webfrontend</span><br /><br />Installing the dependencies<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev</span><br /><br />Making the required changes in gmetad.conf<br />data_source "HadoopCluster" Master <br />gridname "Grid"<br />setuid_username "ganglia"<br /><br />datasource specifies the cluster name as "HadoopCluster" and Master as the sole point of consolidating all the metrics and statistics.<br />gridname is assumed as "Grid" initially.<br />username is ganglia.<br /><br />Check for /var/lib/ganglia directory. If not existing then,<br />mkdir /var/lib/ganglia<br />mkdir /var/lib/ganglia/rrds/<br />and then<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo chown -R ganglia:ganglia /var/lib/ganglia/</span><br /><br />Running the gmetad<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/gmetad start</span><br /><br />You can stop with <br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/gmetad stop </span><br />and run in debugging mode once.<br /><span style="font-family: "Courier New",Courier,monospace;">$ gmetad -d 1</span><br /><br />Now, restart the daemon<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/gmetad restart</span><br /><br /><b>Step 3</b> : Installing PHP Web-frontend dependent packages at "Master"<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev</span><br /><br />Check for /var/www/ganglia directory. If not existing, then copy the contents of /usr/share/ganglia-webfrontend to /var/www/ganglia.</span><br />
<span style="font-size: small;">You must be able to see graph.d and many other .php. Restart apache2.<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/apache2 restart</span><br /><br />Time to hit Web URL<br />http://Master/ganglia <br />in general, http://<hostname>/ganglia/<br /><br />You must be able to see some graphs.<br /><br /><br /><b>Step 4</b> : Configuring Hadoop-metrics with Ganglia.<br />On Master( Namenode, JobTracker)<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo vi /etc/hadoop-0.20/conf/hadoop-metrics.properties</span></span><br />
<blockquote>
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.fileName=/tmp/dfsmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> dfs.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> dfs.servers=Master:8650</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">mapred.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.fileName=/tmp/mrmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.servers=Master:8650</span></span><br />
<br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">jvm.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.fileName=/tmp/jvmmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> jvm.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> jvm.servers=Master:8650</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">rpc.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.fileName=/tmp/rpcmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> rpc.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> rpc.servers=Master:8650</span></span></blockquote>
<span style="font-size: small;"><br />Hadoop provides a way to access all the metrics with GangliaContext31 class.<br />Restart hadoop services at Master.<br />Restart gmond and gmetad.<br /><span style="font-family: "Courier New",Courier,monospace;">$ telnet Master 8649</span><br />will spit XML metrics of Hadoop<br /><br />On Slave1 (Secondary Namenode, Datanode, TaskTracker)<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo gedit /etc/hadoop-0.20.2/conf/hadoop-metrics.properties </span></span><br />
<blockquote>
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.fileName=/tmp/dfsmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> dfs.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> dfs.servers=Master:8650</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "dfs" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#dfs.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#mapred.fileName=/tmp/mrmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "mapred" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> mapred.servers=Master:8650</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">jvm.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#jvm.fileName=/tmp/jvmmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "jvm" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> jvm.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> jvm.servers=Master:8650</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for null</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.class=org.apache.hadoop.metrics.spi.NullContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for /metrics</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">rpc.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for file</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.class=org.apache.hadoop.metrics.file.FileContext</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;">#rpc.fileName=/tmp/rpcmetrics.log</span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: x-small;"># Configuration of the "rpc" context for ganglia</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> rpc.period=10</span></span><br />
<span style="font-size: small;"><span style="font-size: x-small;"> rpc.servers=Master:8650</span></span></blockquote>
<span style="font-size: small;">Restart the datanode, tasktracker with the command<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo service hadoop-0.20-tasktracker restart</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">$ sudo service hadoop-0.20-datanode restart</span></span><br /><span style="font-size: small;">Restart gmond<br /><span style="font-family: "Courier New",Courier,monospace;">$ telnet Master 8649</span><br />will spit the XML format of Hadoop metrics for this host Slave1<br /><br />Similar procedure done to Slave1 must be followed for Slave2 and then restarting the services of hadoop and gmond.<br /><br />On the Master, <br />Restart gmond and gmetad with<br /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/ganglia-monitor restart</span><br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">$ sudo /etc/init.d/gmetad restart</span><br style="font-family: "Courier New",Courier,monospace;" /><br />Hit the web URL<br />http://Master/ganglia<br />Check for Metrics, Grid, Cluster and all the nodes you configured for.<br />You can also witness the hosts up, hosts down, total CPUs. Lots more in store!<br /> </span><br />
<span style="font-size: small;">Enjoy monitoring your cluster! :)</span></div>
</div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-86759068796030894382012-07-01T15:16:00.000+05:302012-07-01T15:16:15.288+05:30Hadoop Hangover: Time To Yarn [!Yawn ;)] and Watchout for Kitten!<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
Well, we have seen new versions and new releases in software as days roll by. No wonder!<br />Same is the case with Hadoop! Past few months we saw new releases on the table.<br />I would like to suggest the following amazing article by Jim, explaining the <b>Evolving APIs</b>. <br />A <b>Good</b> read and a <b>MUST</b> read!<br /><a href="http://wiki.eclipse.org/Evolving_Java-based_APIs" style="color: #3465a4;">http://wiki.eclipse.org/Evolving_Java-based_APIs</a><br /><br />Releases?? Here it goes:<br /><ul>
<li dir="ltr">1.x series is the continuation of 0.20 release series</li>
<li dir="ltr">2.x series is the continuation of 0.23 release series</li>
</ul>
<br />The 1.x series are the stable ones and majority of commercial distribution market carry forward the same. This release is more matured in terms of authentication.<br />In security parlance, there are two major buzzwords:<br /><ul>
<li dir="ltr"><b>authentication</b> means that a person who is performing an operation is the person who claims to be. </li>
<li dir="ltr"><b>authorization</b> means what a person can do with a particular file or directory.</li>
</ul>
<br />Previously, it was just the authorization supported, giving a wide way for malicious users spoofing and gaining a free entry pass into the network cluster! The 1.x series now is more stable regarding this with the introduction of Kerberos, open-source network authentication protocol, to authenticate the user. Now, here there is a quite good handshake between these two open source fellows.<br />Kerberos, authenticates that the person is the one who he/she claims to be to Hadoop and then it is the work of Hadoop to do authorization checking on the permissions. It was in 0.20.20x series, the whole Kerberos was implemented.<br />To be known that its not so easy with this whole lotta Kerberos thingy, too many stuffs happening deep underneath! :)<br /><br />The 2.x series and 0.22 series are not greatly stable as 1.x are! But, they are good in their own ways. They own lots of new features. <br />1. <b>MapReduce2</b>: The classic runtime is replaced by a new runtime MapReduce2 also known as YARN [Yet Another Resource Negotiator], which is basically a resource management system running in your cluster in the distributed environment.<br />2. <b>High Availability</b>: The namenode is no more a SPOF, Single Point Of Failure. There are back up nodes to avoid it. Meaning, there are couple of namenode active at any moment.<br />3. <b>HDFS Federation</b>: As the namenode is the main repository to keep whole lots of metadata and to imagine a cluster can be of so many nodes and its storage is huge equally! So, this metadata which has the information about each file and the block, becomes a limiting factor in terms of memory! Thus, we have a pair of namenodes, managing part of the filesystem namespace.<br /><br /><b>MapReduce2 or YARN</b>:<br />If you have a large cluster it is very obvious that you can process large data sets. However, there was an issue with the classic runtime in the previous releases, wherein a cluster of >4000 machines hit a scalability issue. It is costly to have more number of small clusters. Thus, we need large number of machines in a cluster. This architecture will tend to increase the hardware utilization, need for other paradigms other than MapReduce, more agile.<br />To YARN is a beauty ;)<br />Actually, this architectural design is now more close to the Google Map Reduce paper. Reference: <a href="http://research.google.com/archive/mapreduce.html" style="color: #3465a4;">http://research.google.com/archive/mapreduce.html</a><br />Its an awesome Next Generation Hadoop Framework and MapReduce is just one application in that. There are many such applications written upon YARN.<br /><ul>
<li dir="ltr">Open MPI</li>
<li dir="ltr">Spark</li>
<li dir="ltr">Apache Giraph</li>
<li dir="ltr">Apache Hama</li>
<li dir="ltr">Apache HBase [Generic Co porocessors] etc.</li>
</ul>
<br />So, it does makes sense that it can handle other paradigms and also different versions of MapReduce. We all know how complex MapReduce can be and solving every problem in terms of MR is an uphill task!<br /><br />The classic runtime of previous release had a JobTracker, which had two important functions.<br /><ul>
<li dir="ltr">Resource Management</li>
<li dir="ltr">Job Scheduling and Job Monitoring</li>
</ul>
<br />But, this architecture is a little bit of re-design of the previous dealing with the splitting of JobTracker into two major separate components.<br /><ul>
<li dir="ltr"><b>Resource Manager</b>: It will compute the resources required for the applications globally and assign/ manage them.<b></b></li>
<li dir="ltr"><b>Per-Application Application Master</b>: The scheduling and monitoring done by JobTracker previously is done per application by the application master.</li>
</ul>
<br />These two are separate demons.<strike></strike>What does application mean here? <br />Application is nothing but a job in the classic runtime.<br />The Application master will talk to resource manager for the resources in the cluster ie. also termed as "Container" (i/o, CPU, Memory etc.). Container is the home where all the application processes run and its the heavy duty of the node managers to check on the applications not to rob more resources from the container.<br /><ul>
<li dir="ltr">The MapReduce program will run the job at the client node in the client JVM.</li>
<li dir="ltr">The Job will now get the application from the resource manager and then copy all those resources to the file system which is shared by the cluster and submit it back to the resource manager.</li>
<li dir="ltr">It is now the duty of the resource manager to start the container at the node manager and that launching the Application Master.</li>
<li dir="ltr">The Application Master will initialise the job at the node manager and get the input splits from the shared file system of the cluster, allocating resources at the resource manager node.</li>
<li dir="ltr">The Application Master will start the container at the node manager where task are to be run, in parallel.</li>
<li dir="ltr">The Node Manager will start the task JVM at the node manager node, where the YarnChild will retrieve all the resources from the shared file system for it to run the corresponding map or reduce task at the node manager node.</li>
</ul>
<br /><br />However, its not so simple as listed above. Too many stuffs happening! <br />Well, this was just an introduction and developers who wanna play with YARN, do have a look at <b>Kitten </b>( released just 5days before the day of writing this),</div>
<div>
Link: <a href="https://github.com/cloudera/kitten" style="color: #3465a4;">https://github.com/cloudera/kitten</a>, having a set of Java libraries and Lua-based configuration files that handle configuring, launching, and monitoring YARN applications, allowing developers to spend most of their time writing the logic of an application, not its configuration.<br />Will see more of all these in the coming posts.<br />Until then, Happy Learning!! :):)</div>
</div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-27894122226068746742012-06-27T15:58:00.000+05:302012-06-28T11:18:41.039+05:30Hadoop Hangover: Introduction To Apache Bigtop and Installing Hive, HBase and Pig<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: 'Trebuchet MS', sans-serif;">In the previous post we learnt how easy it was to install Hadoop with Apache Bigtop! </span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">We know its not just Hadoop and there are sub-projects around the table! So, lets have a look at how to install Hive, Hbase and Pig in this post.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Before rowing your boat</b>...</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Please follow the previous post and get ready with Hadoop installed!</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Follow the link for previous post:</span><br />
<a href="http://femgeekz.blogspot.in/2012/06/hadoop-hangover-introduction-to-apache.html" style="color: #3465a4;"><span style="font-family: 'Trebuchet MS', sans-serif;">http://femgeekz.blogspot.in/2012/06/hadoop-hangover-introduction-to-apache.html</span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">also, the same can be found at DZone, developer site: <span style="background-color: white;"></span><span style="background-color: white;"><a href="http://www.dzone.com/links/hadoop_hangover_introduction_to_apache_bigtop_and.html">http://www.dzone.com/links/hadoop_hangover_introduction_to_apache_bigtop_and.html</a></span></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>All Set</b>?? Great! Head On..</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Make sure all the services of Hadoop are running. Namely, JobTracker, SecondaryNameNode, TaskTracker, DataNode and NameNode. [standalone mode]</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Hive with Bigtop</b>:</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">The steps here are almost the same as Installing Hive as a separate project.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">However, few steps are reduced.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">The Hadoop installed in the previous post is Release 1.0.1</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">We had installed Hadoop with the following command</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">sudo apt-get install hadoop\*</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 1</b>:<b> Installing Hive</b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">We have installed Bigtop 0.3.0, and so issuing the following command installs all the hive components.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">ie. hive, hive-metastore, hive-server. The daemons names are different in Bigtop 0.3.0.</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">sudo apt-get install hive\*</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">This installs all the hive components. After installing, the scripts must be able to create <i>/tmp </i> and <i><a href="http://www.blogger.com/usr/hive/warehouse" style="color: #3465a4;">/usr/hive/warehouse</a></i><i></i>and HDFS doesn't allow these to be created while installing as it is unaware of the path to Java. So, create the directories if not created and grant the execute permissions. </span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">In the hadoop directory, ie. <a href="http://www.blogger.com/usr/lib/hadoop/" style="color: #3465a4;">/usr/lib/hadoop/</a></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><i>bin/hadoop fs -mkdir /tmp<br />bin/hadoop fs -mkdir <a href="http://www.blogger.com/user/hive/warehouse" style="color: #3465a4;">/user/hive/warehouse</a></i><i><br />bin/hadoop -chmod g+x /tmp<br />bin/hadoop -chmod g+x <a href="http://www.blogger.com/user/hive/warehouse" style="color: #3465a4;">/user/hive/warehouse</a></i></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><i></i><b>Step 2</b>: The alternative directories could be<i><a href="http://www.blogger.com/var/run/hive" style="color: #3465a4;">/var/run/hive</a></i><i></i>and<i><a href="http://www.blogger.com/var/lock/subsys" style="color: #3465a4;">/var/lock/subsys</a></i></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><i>sudo mkdir <a href="http://www.blogger.com/var/run/hive" style="color: #3465a4;">/var/run/hive</a></i><i><br />sudo mkdir <a href="http://www.blogger.com/var/lock/subsys" style="color: #3465a4;">/var/lock/subsys</a></i></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><i></i><b>Step 3</b>: Start the hive server, a daemon</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><i>sudo <a href="http://www.blogger.com/etc/init.d/hive-server" style="color: #3465a4;">/etc/init.d/hive-server</a></i><i> start</i></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image:</span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSXH5eTl9-qpR_o9UZgEaSN0qjiOPQQjEZyzKWpG5pLJVVTUggBVsqqKTvYU3MFDJiNtSWKHZQ1bGh2wHDtmCaWTS5gK1Ms-k2baVRA0GxksV_N23AtEap71UZxA9ODDwv0U88H50n97Pl/s1600/hiveserver.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="106" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSXH5eTl9-qpR_o9UZgEaSN0qjiOPQQjEZyzKWpG5pLJVVTUggBVsqqKTvYU3MFDJiNtSWKHZQ1bGh2wHDtmCaWTS5gK1Ms-k2baVRA0GxksV_N23AtEap71UZxA9ODDwv0U88H50n97Pl/s640/hiveserver.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">start hive-server</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 4</b>: Running Hive</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Go-to the directory <i><a href="http://www.blogger.com/usr/lib/hive" style="color: #3465a4;">/usr/lib/hive</a></i>.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">See the Image below:</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">bin/hive</span></i><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxQTdswnDt8Ntt0iajAtA0kpuh4rrnJIQ1gt6du1T3WOpCA8Pmwr-o69BC1sY7MqHlu-r_0mULFCOoJpuVY6um45sc0YSoreBOLrY_Ec1ybLyvLALQGVTy9ew3Gz_cq6tzw9fotzblmDKu/s1600/bin-hive.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxQTdswnDt8Ntt0iajAtA0kpuh4rrnJIQ1gt6du1T3WOpCA8Pmwr-o69BC1sY7MqHlu-r_0mULFCOoJpuVY6um45sc0YSoreBOLrY_Ec1ybLyvLALQGVTy9ew3Gz_cq6tzw9fotzblmDKu/s640/bin-hive.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">bin/hive</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b></b><br /><b></b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b><br /></b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b><br /></b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b><br /></b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b><br /></b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b><br /></b></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 5</b>: Operations on Hive</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgXXIMX36FLmDS3tJo_Sh3SbXkBfV6edyRJIXpKWfo3b1Oki4eemFjouK74GGnsW9wCQ6kQzeDJloiWXYMqpRQJ9lpRW5yqY-uZcXxQKWHS9c6fOXkkTy4Su5hSsDspSkLGEGzahRDH9Bs/s1600/hiveoperations.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="326" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgXXIMX36FLmDS3tJo_Sh3SbXkBfV6edyRJIXpKWfo3b1Oki4eemFjouK74GGnsW9wCQ6kQzeDJloiWXYMqpRQJ9lpRW5yqY-uZcXxQKWHS9c6fOXkkTy4Su5hSsDspSkLGEGzahRDH9Bs/s640/hiveoperations.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Basic hive operations</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>HBase with Bigtop</b>:</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Installing Hbase is similar to Hive.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 1</b>: Installing HBase</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">sudo apt-get install hbase\*</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgid0bXy2S5p0Tpr9UDkMzpyYIn4UrdT2ng9Yt_XGXmneMErjToHhbx3WzDOxv0cgB_61Dyrdn3i7G1Bf_EoYPq-ZKz5ZR89s_5xKtfrCzCPwqr0595_Anjr323EfnzkIht2S2nMKZy050u/s1600/hbase.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgid0bXy2S5p0Tpr9UDkMzpyYIn4UrdT2ng9Yt_XGXmneMErjToHhbx3WzDOxv0cgB_61Dyrdn3i7G1Bf_EoYPq-ZKz5ZR89s_5xKtfrCzCPwqr0595_Anjr323EfnzkIht2S2nMKZy050u/s640/hbase.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">hbase-0.92.0</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 2</b>: Starting HMaster</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">sudo service hbase-master start</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8GnF9wtJGW3vSG__iYnfjd4PXUQDoZZMLGjPrgoQxuEVVedRRZ1RsvlRmNnHRyPNnaOElwK0UZY9Lr3dPeYvwhal5g1Xi_Gs53jVeFAI5qHB-gMeDbHqmOdOZtEdY3-xqEEb-hatKaqCu/s1600/hbase+start.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8GnF9wtJGW3vSG__iYnfjd4PXUQDoZZMLGjPrgoQxuEVVedRRZ1RsvlRmNnHRyPNnaOElwK0UZY9Lr3dPeYvwhal5g1Xi_Gs53jVeFAI5qHB-gMeDbHqmOdOZtEdY3-xqEEb-hatKaqCu/s640/hbase+start.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Starting HMaster</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieY4YlXT2GooHE-jYVFCn0v_zYH29yhlzYtivUSV-Oa-ObRgqwsojfIgAolssvJSldiobrXppmpZPx1bfR4BO-9EQnUmvyYxYp_fn_iBR19wFX_Zo73afyy_vn8zdItOqXAFwtm7lkzJ9G/s1600/hbasejps.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieY4YlXT2GooHE-jYVFCn0v_zYH29yhlzYtivUSV-Oa-ObRgqwsojfIgAolssvJSldiobrXppmpZPx1bfR4BO-9EQnUmvyYxYp_fn_iBR19wFX_Zo73afyy_vn8zdItOqXAFwtm7lkzJ9G/s640/hbasejps.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">jps (HMaster started)</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 3</b>: Starting HBase shell</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">hbase shell</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvcYyIsWxon-F5D_3gbVYZvv6VIddc_DwOSIOGEWaTyInGKFJ3TzoXJdO7323aCFRWc35arRS0vRG5wOpWqLD6oyaUFqytT_EetkDHnmDJeQZlxQgKmjanPTV8LEbowRDvxEN_UZpC03jY/s1600/hbasetable.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvcYyIsWxon-F5D_3gbVYZvv6VIddc_DwOSIOGEWaTyInGKFJ3TzoXJdO7323aCFRWc35arRS0vRG5wOpWqLD6oyaUFqytT_EetkDHnmDJeQZlxQgKmjanPTV8LEbowRDvxEN_UZpC03jY/s640/hbasetable.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">start HBase shell</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 4</b>: HBase Operations</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwGB_dAQwH-SSpe-JXhdAD3aiAziDE54MlQ1-prVRYW7VK1f0g_2qW2qDnzQkAFEtA_v8XKERJ6_l4qZOx9wxBcZMxjHVV_FXX0mlnD8ZJf6-8sDsF5voibElGu_gS22_Pdm_hyOTibde7/s1600/hbasetabop.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwGB_dAQwH-SSpe-JXhdAD3aiAziDE54MlQ1-prVRYW7VK1f0g_2qW2qDnzQkAFEtA_v8XKERJ6_l4qZOx9wxBcZMxjHVV_FXX0mlnD8ZJf6-8sDsF5voibElGu_gS22_Pdm_hyOTibde7/s640/hbasetabop.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">HBase table operations</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibhgE7THk0Jol8jaJIJh3DY2RIDDe-rqHG5iPz9-F3VlDmYp_1VaiyPt8kX394F1XztpUFORTPc6sHiOW60ecWfB8Brii2PAYFH_KuY0eGYMR4g-_huqvipZdNSWuN1dib8s7fIV2zrmA2/s1600/hbasetabop2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="304" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibhgE7THk0Jol8jaJIJh3DY2RIDDe-rqHG5iPz9-F3VlDmYp_1VaiyPt8kX394F1XztpUFORTPc6sHiOW60ecWfB8Brii2PAYFH_KuY0eGYMR4g-_huqvipZdNSWuN1dib8s7fIV2zrmA2/s640/hbasetabop2.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">list,scan,get,describe In HBase</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<b><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></b><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><strike></strike><b>Pig with Bigtop</b>:</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Installing Pig is similar too.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 1</b>: Installing Pig</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">sudo apt-get install pig</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4HOZxw2j5eYhNEuQ5GqUXSv150p4NRZPO-MWVAhAzwNeEfrH7vjGiUQ8Pz1P0LXZAbh0ZwdWf42ZHLsh7nmCgJQH2L2d0PyTwBp6iHo7l1fy1ga4cFbFsyyxsPetluQh-E3acKf1edPOA/s1600/pig.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4HOZxw2j5eYhNEuQ5GqUXSv150p4NRZPO-MWVAhAzwNeEfrH7vjGiUQ8Pz1P0LXZAbh0ZwdWf42ZHLsh7nmCgJQH2L2d0PyTwBp6iHo7l1fy1ga4cFbFsyyxsPetluQh-E3acKf1edPOA/s640/pig.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Installing Pig</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 2</b>: Moving a file to HDFS</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdZlGYleY4qpxufHqDZGLS-WBsdp_wuFGrOjgFS90VBdTkUHYOGV5TVBGUrplfZ__bbde0AbwdqEHCzASOBHPspwsf_ZCcEa9n-rtz0ovyX1W8zk08dgPAMnHq5AkGf6UGWY4GFM3fEw4S/s1600/pighdfsfile.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdZlGYleY4qpxufHqDZGLS-WBsdp_wuFGrOjgFS90VBdTkUHYOGV5TVBGUrplfZ__bbde0AbwdqEHCzASOBHPspwsf_ZCcEa9n-rtz0ovyX1W8zk08dgPAMnHq5AkGf6UGWY4GFM3fEw4S/s640/pighdfsfile.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Moving a tab separated file "book.csv" to HDFS</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 3</b>: Installed Pig-0.9.2</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6S1l7VpcstOg6LIAXZGBIFMp2b0ySLQmQGZZlsJ3ujzLwoI9BBghZNnBn8F8eNUhUswC1NqV-FkUz9cF_Nyn-TonhHjLhn9NObwZHVG1wsOZSotqAw1IDCkvDrnb71ohQ5vmd8ZWrWVYZ/s1600/pigls.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6S1l7VpcstOg6LIAXZGBIFMp2b0ySLQmQGZZlsJ3ujzLwoI9BBghZNnBn8F8eNUhUswC1NqV-FkUz9cF_Nyn-TonhHjLhn9NObwZHVG1wsOZSotqAw1IDCkvDrnb71ohQ5vmd8ZWrWVYZ/s640/pigls.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Pig installed Pig-0.9.2</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 4</b>: Starting the grunt shell</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">pig</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image: </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnIydwdPQPbm5bgOguI4uoO1KnEwQ1_dZXfViVZ0wWehFX3T2AfjMSvdOMlrSB_rVoazSe61mzRnFFlsNEXjVpPrAwn6FtySdoQdwZj7TpXpaBGlQhf8CC-Ny2i6zcc4iezZIxdeBxDnf_/s1600/pigstart.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnIydwdPQPbm5bgOguI4uoO1KnEwQ1_dZXfViVZ0wWehFX3T2AfjMSvdOMlrSB_rVoazSe61mzRnFFlsNEXjVpPrAwn6FtySdoQdwZj7TpXpaBGlQhf8CC-Ny2i6zcc4iezZIxdeBxDnf_/s640/pigstart.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Starting Pig</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 5</b>: Pig Basic Operations</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image:</span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoTy-crXnW9iGaYhMAe9Nh36PmPfUWSit7M5arhcXRmb9aHtD5QJ9fI3Y32i3jg-PZhh7d6IrOvtrNQ9H77Yby6S52Uchyphenhyphens0Lv8oRe0SdxLo0Eny5-qy8H0i5qjefE6cKQxge2Wy1VC686/s1600/pigop.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoTy-crXnW9iGaYhMAe9Nh36PmPfUWSit7M5arhcXRmb9aHtD5QJ9fI3Y32i3jg-PZhh7d6IrOvtrNQ9H77Yby6S52Uchyphenhyphens0Lv8oRe0SdxLo0Eny5-qy8H0i5qjefE6cKQxge2Wy1VC686/s640/pigop.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Basic Pig Operations</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Image:</span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkexRKHwEBDLoWMBvV2T8g361iBgN4Mv4OSeinyYo_FVLX7oaDsEJeP5aKYu09MuiO8slL3-VQHONHjBLe91DBN3sAaFZ_-OSZe2Z6Il3XYtDxhbifw8zgz61BfU4jvnbUPkLK82QN2YSP/s1600/pigjob.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkexRKHwEBDLoWMBvV2T8g361iBgN4Mv4OSeinyYo_FVLX7oaDsEJeP5aKYu09MuiO8slL3-VQHONHjBLe91DBN3sAaFZ_-OSZe2Z6Il3XYtDxhbifw8zgz61BfU4jvnbUPkLK82QN2YSP/s640/pigjob.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;">Job Completion</span></td></tr>
</tbody></table>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">We saw that is it possible to install the subprojects and work with Hadoop, with no issues.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Apache Bigtop has its own spark! :)</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">There is a release coming BIGTOP-0.4.0 which is supposedly to fix the following issues:</span><br />
<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12318889&styleName=Html&projectId=12311420" style="color: #3465a4;"><span style="font-family: 'Trebuchet MS', sans-serif;">https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12318889&styleName=Html&projectId=12311420</span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Source and binary files:</span><br />
<a href="http://people.apache.org/~rvs/bigtop-0.4.0-incubating-RC0" style="color: #3465a4;"><span style="font-family: 'Trebuchet MS', sans-serif;">http://people.apache.org/~rvs/bigtop-0.4.0-incubating-RC0</span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Maven staging repo:</span><br />
<a href="https://repository.apache.org/content/repositories/orgapachebigtop-279" style="color: #3465a4;"><span style="font-family: 'Trebuchet MS', sans-serif;">https://repository.apache.org/content/repositories/orgapachebigtop-279</span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Bigtop's KEYS file containing PGP keys we use to sign the release:</span><br />
<a href="http://svn.apache.org/repos/asf/incubator/bigtop/dist/KEYS" style="color: #3465a4;"><span style="font-family: 'Trebuchet MS', sans-serif;">http://svn.apache.org/repos/asf/incubator/bigtop/dist/KEYS</span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Let us see how to install other sub-projects in the coming posts!</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Until then, Happy Learning!! :):)</span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-74197142886973534872012-06-22T19:30:00.001+05:302012-06-22T19:30:57.686+05:30Hadoop Hangover : Introduction To Apache Bigtop and Playing With It (Installing Hadoop)!<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span style="font-family: 'Trebuchet MS', sans-serif;">Ah!! The name is everywhere, carried with the wind. Apache Hadoop!!<br />The BIG DATA crunching platform!<br /> We all know how alien it can be at start too! Phew!! :o<br /><br />Its my personal experience, nearly 11 months before, I was trying to install HBase, I faced few issues! The problem was version compatibility. Ex: "HBase some x.version" with "Hadoop some y.version".<br />This is a real issue because you will never know which package of what version blends well with the other, unless, someone has tested it. This testing again depends on the environment where they have set up and could be another issue.<br />There was a pressing demand for the management of distributions and then comes an open source project which attempts to create a fully integrated and tested Big Data management distribution, "<b><i>Apache Bigtop</i></b>".<br /><br /><b>Goals of Apache Bigtop</b>:<br />-Packaging<br />-Deployment<br />-Integration Testing<br />of all the sub-projects of Hadoop. This project aims at system as a whole, than the individual project.<br /><br />I love the way Doug Cutting quoted in the Keynote, back then, wherein he expressed the similarity between Hadoop and Linux kernel,and the corresponding similarity between the big stack of Hadoop ( Hive, Hbase, Pig, Avro, etc.) and the fully operational operating systems with its distributions (RedHat, Ubuntu, Fedora, Debian etc.). This is an awesome analogy! :)<br /><br /><b>Life is made easy with Bigtop</b>:<br /> Bigtop Hadoop distribution artifacts won't make you feel that you live in an alien world! After installing, you will get a chance to blend a Hadoop cluster in any mode, with the sub-projects of it. Its all for you to garnish next! :)<br /><br /><b>Setup Of Bigtop and Installing Hadoop</b>:<br />It's time to welcome all your packages home. [I also mean /home/..] ;)<br />I've tested on Ubuntu 11.04 and here goes a quick and easy installation process.<br /><br /><b>Step 1</b>: Installing the GNU Privacy Guard key, a key management system to access all public key directories.<br /> <i>wget -O- <a href="http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/GPG-KEY-bigtop" style="color: #3465a4;">http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/GPG-KEY-bigtop</a></i><i> | sudo apt-key add -</i></span></div>
<div style="text-align: left;">
<i><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></i></div>
<div style="text-align: left;">
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 2</b>: Get the repo file from the link <a href="http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/ubuntu/bigtop.list" style="color: #3465a4;">http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/ubuntu/bigtop.list</a></span></div>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><i>sudo wget -O <a href="http://www.blogger.com/etc/apt/sources.list.d/bigtop.list" style="color: #3465a4;">/etc/apt/sources.list.d/bigtop.list</a></i><i><a href="http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/ubuntu/bigtop.list" style="color: #3465a4;">http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/ubuntu/bigtop.list</a></i></span></li>
</ul>
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><i>sudo gedit <a href="http://www.blogger.com/etc/apt/sources.list.d/bigtop.list" style="color: #3465a4;">/etc/apt/sources.list.d/bigtop.list</a></i><i></i><br />uncomment the mirror link near by. The first link worked for me. <br /> <i>deb <a href="http://apache.01link.hk/incubator/bigtop/stable/repos/ubuntu/" style="color: #3465a4;">http://apache.01link.hk/incubator/bigtop/stable/repos/ubuntu/</a></i><i> bigtop contrib</i></span></li>
</ul>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 3</b>: Updating the apt cache<br /><i> sudo apt-get update</i></span></div>
<div>
<i><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></i><div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 4</b>: Checking in the artifacts<br /><i> sudo apt-cache search hadoop</i><br />Image: </span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifAtPkGx3NUQz5yinh_gJep3A6c7aqzcsaeJDpeKDU_aizgweKBt7D_WkTdRLwP0Tis10_iyheNdjX37tL39CYlPRLtEvTCft1I9208FFRliAHS4r1T5Bbg7SMOrX6s8kmMoWdtpJCldFY/s1600/search_hadoop.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="460" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifAtPkGx3NUQz5yinh_gJep3A6c7aqzcsaeJDpeKDU_aizgweKBt7D_WkTdRLwP0Tis10_iyheNdjX37tL39CYlPRLtEvTCft1I9208FFRliAHS4r1T5Bbg7SMOrX6s8kmMoWdtpJCldFY/s640/search_hadoop.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">Search in the apt cache<br /></span></td></tr>
</tbody></table>
<div>
<b style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></b></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 5</b>: Set your JAVA_HOME <br /> <i>export JAVA_HOME=path_to_your_Java</i><br />export $JAVA_HOME in <a href="http://www.blogger.com/~/.bashrc" style="color: #3465a4;">~/.bashrc</a></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 6</b>: Installing the complete Hadoop stack <br /><i> sudo apt-get install hadoop\*</i><br />Image: (above)<br /><b style="background-color: white;"><br /></b></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Running Hadoop</b>:<br /><br /><b>Step 1</b>: Formatting the namendoe<br /><i> sudo -u hdfs hadoop namenode -format</i><br />Image : </span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3y8A1JqJTAy0ABdeDrmdA8XSnYqSoShAAasm4TzjxlHpxrOCaAOGA_st8-5B2xvXnfKW5ksfFuM7QzYRxvy1Is5OSC3uq0kIs_Ps6BvBZpbmHEH8kYUbrKuDmbXHM5iDssNaREzME4JSk/s1600/bigtop1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="276" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3y8A1JqJTAy0ABdeDrmdA8XSnYqSoShAAasm4TzjxlHpxrOCaAOGA_st8-5B2xvXnfKW5ksfFuM7QzYRxvy1Is5OSC3uq0kIs_Ps6BvBZpbmHEH8kYUbrKuDmbXHM5iDssNaREzME4JSk/s640/bigtop1.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">Formatting the namenode</span></td></tr>
</tbody></table>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /><b>Step 2</b>: Starting the Namenode, Datanode, Jobtracker, Tasktracker of Hadoop<br /><i> for i in hadoop-namenode hadoop-datanode hadoop-jobtracker hadoop-tasktracker ; do sudo service $i start ; done</i><br /> Now, the cluster is up and running.<br />Image :</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtHY9NCLFAcbNDqDdeFJTZLzud9WPtaTNWiay1qeX79E2BoXOF2cKY7mrlRGzsS8IRgHTGooYEd3EsuJG229RwTHSD9IqGoGP7WbtAzJuHmt8BqFTMXFAQAznqpdMFOw3oyBHHakuqIyXk/s1600/bigtop2.png" imageanchor="1" style="margin-left: auto; margin-right: auto; text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtHY9NCLFAcbNDqDdeFJTZLzud9WPtaTNWiay1qeX79E2BoXOF2cKY7mrlRGzsS8IRgHTGooYEd3EsuJG229RwTHSD9IqGoGP7WbtAzJuHmt8BqFTMXFAQAznqpdMFOw3oyBHHakuqIyXk/s640/bigtop2.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">Start all the services</span></td></tr>
</tbody></table>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 3</b>: Creating a new directory in hdfs<br /><i> sudo -u hdfs hadoop fs -mkdir <a href="http://www.blogger.com/user/" style="color: #3465a4;">/user/</a></i><a href="http://www.blogger.com/bigtop" style="color: #3465a4;">bigtop</a><br />bigtop is the directory name in the user $USER<br /><i> sudo -u hdfs hadoop fs -chown $USER <a href="http://www.blogger.com/user/" style="color: #3465a4;">/user/</a></i><a href="http://www.blogger.com/bigtop" style="color: #3465a4;">bigtop</a><br />Image :</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJDGaFe8V3cLqnSwsgVbdDHwL1p3XeWbIw45tBv4pQNBwX6uJl-e-VkLvO4kEaet3b8bIepG6tyHeKygx7uiSALBE47iGifhDXOxX4Jt7jYuGtrWWApMN4W53ymV7ZKYVjtBTIFwZjKLHK/s1600/bigtop3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJDGaFe8V3cLqnSwsgVbdDHwL1p3XeWbIw45tBv4pQNBwX6uJl-e-VkLvO4kEaet3b8bIepG6tyHeKygx7uiSALBE47iGifhDXOxX4Jt7jYuGtrWWApMN4W53ymV7ZKYVjtBTIFwZjKLHK/s640/bigtop3.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">Create a directory in HDFS</span></td></tr>
</tbody></table>
<div>
<span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></span></div>
<div>
<b style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></b></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 4</b>: List the directories in file system<br /><i> hadoop fs -lsr /</i><br />Image :</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht1FsG6QitEvfXkf4qWliSb-rg6EscJ_bHiGxrWhvJdiHUIfMaxVM7xQHz9KEjrq9bUoPKbwDtHqx37Zhpv7CYx9Rs-zmlpSL524XBn8wt0OedG3WtrcGnxj1ohgQz2dqaIEQSjtpnJ5aj/s1600/filesystem.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht1FsG6QitEvfXkf4qWliSb-rg6EscJ_bHiGxrWhvJdiHUIfMaxVM7xQHz9KEjrq9bUoPKbwDtHqx37Zhpv7CYx9Rs-zmlpSL524XBn8wt0OedG3WtrcGnxj1ohgQz2dqaIEQSjtpnJ5aj/s640/filesystem.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">HDFS directories</span></td></tr>
</tbody></table>
<div>
<span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Step 5</b>: Running a sample pi example<br /><i> hadoop jar <a href="http://www.blogger.com/usr/lib/hadoop/hadoop-examples.jar" style="color: #3465a4;">/usr/lib/hadoop/hadoop-examples.jar</a></i><i> pi 10 1000</i><br />Image :</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFa21lUefUCAoRdLHeuok_QbqkLHyCaUb43t4WWfLw3nMr7sTbf3HuXOIJOJ6RkEeJH1Ky9yXGNaZt_zh5X-v9aD_d1fAfN3CH7EYrlLJL0I85HLGBh2brtqiAB_lUFugadRzzfOtpIwSM/s1600/bigtop4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="496" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFa21lUefUCAoRdLHeuok_QbqkLHyCaUb43t4WWfLw3nMr7sTbf3HuXOIJOJ6RkEeJH1Ky9yXGNaZt_zh5X-v9aD_d1fAfN3CH7EYrlLJL0I85HLGBh2brtqiAB_lUFugadRzzfOtpIwSM/s640/bigtop4.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">Running a sample program</span></td></tr>
</tbody></table>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br style="background-color: white;" /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirbJHVyA8-ZC4SHL51ei5_T1zCLEeTtDsn1C-H0IJaYQ5EqNEiwwQ5bfB4RxjwsTta6yPatDTeFMXuJn3us0NkWIPHaN7D06BA8xIUuAQsjf5OnOZOMJQQB7hG4sj7dsZDjyWappvutDuG/s1600/bigtop6.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirbJHVyA8-ZC4SHL51ei5_T1zCLEeTtDsn1C-H0IJaYQ5EqNEiwwQ5bfB4RxjwsTta6yPatDTeFMXuJn3us0NkWIPHaN7D06BA8xIUuAQsjf5OnOZOMJQQB7hG4sj7dsZDjyWappvutDuG/s640/bigtop6.png" width="640" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: small;">Job Completed!</span></td></tr>
</tbody></table>
<div>
<span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br />Enjoy with your cluster! :)</span></div>
</div>
</div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">We shall see what more blending could be done with Hadoop (with Hive, Hbase, etc.) in the next post!</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">Until then, </span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">Happy Learning!! :):)</span></div>
</div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com4tag:blogger.com,1999:blog-6187879032785791598.post-77613299505238318812012-06-18T23:20:00.000+05:302012-06-18T23:37:55.153+05:30Start Darting...(Contd.)<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: 'Trebuchet MS', sans-serif;">Back again! :)<br />The first part of "Start Darting" is at the following link:<br /><a href="http://www.dzone.com/links/start_darting_.html" style="color: #3465a4;">http://www.dzone.com/links/start_darting_.html</a> (or)<br /><a href="http://femgeekz.blogspot.in/2012/06/start-darting.html" style="color: #3465a4;">http://femgeekz.blogspot.in/2012/06/start-darting.html</a><br /><br />Well, to start afresh and get started, no place is as great as <a href="http://www.dartlang.org/docs/editor/getting-started/" style="color: #3465a4;">http://www.dartlang.org/docs/editor/getting-started/</a><br />The steps given below are more detailed in the aforementioned link. Please do go through them. :)<br />Step 1: Download the dart editor. Its nearly 110MB .zip file, available for Windows, Linux and Mac.<br />Step 2: Launch the editor by double clicking the executable.<br />Step 3: Running a dart code.<br />We saw in the previous post a Hello World sample. The same sample can be run.<br />If your code has web app, it will launch it in Dartium (Chromium with the Dart VM).</span><br />
<ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>New Features about Dart</b>:</span><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Snapshots</b>: Modern browsers need to parse a web app’s source code before that app can run. Snapshots here mean that the state and the code is recorded at certain point of time, which will increase the speed at startup.</span></li>
</ul>
</li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Isolates</b>: These are the processes without any overhead. Each isolate has its own memory and code, which can’t be affected by any other isolate. The only way an isolate can communicate with another isolate is by way of messages. Isolates allow a single app to use multi-core computers effectively. Another use for isolates is running code from different origins in the same page, without compromising security.</span></li>
</ul>
</li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Interfaces with default implementations</b>: Dart interface can have a default implementation—a class (usually private) that is the default way to create objects that implement the interface. </span></li>
</ul>
</li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Easy Generics</b>: Dart takes a new approach by designing a generics system that’s more understandable.</span></li>
</ul>
</li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Optional typing</b>: Its optional to use type in Dart and it won't change the way your app executes, but they can help developers and programming tools to understand your code. You might not bother with types while you’re developing a prototype, but you might add types when you’re ready to commit to an implementation. An emerging pattern is to add types to interfaces and method signatures,and omit types inside methods.</span></li>
</ul>
</li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>HTML library</b>: We also took a fresh look at how you should use the HTML DOM. DOM is short for Document Object Model; it’s the interface that lets you pro-grammatically update the content, structure, and style of a web page.) By creating a native Dart library (dart:html) to access and manipulate the DOM. There are elements, attributes, and nodes to work with.</span></li>
</ul>
</li>
</ul>
</li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;"><b>Un-boxing Dart</b>:<br /> Dart is more of a platform than a language. It has</span><br />
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Libraries: The core libraries are like math, server side, I/O, JSON. </span></li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Virtual Machine: Dart Virtual Machine is used, being run on the browsers as embedded or on the command line.</span></li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Compiler to java script: The Frog compiler is used to compile the Dart code to java script.</span></li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Language Specifications: The language looks very familiar like Java or java script because Google wanted mass adoption of the language. If it was a standing new language, only few people would have adopted it. Since, it looks very familiar, people feel comfortable with the optional typing and also isolates.</span></li>
</ul>
</li>
</ul>
<ul style="text-align: left;">
<li style="list-style-type: none;"><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Dart Editor: Its an editor that helps to build complete end to end app. It helps with syntax highlighting, auto completion and running the code.</span></li>
</ul>
</li>
</ul>
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Compiling dart code to java script:</b></span></li>
<li style="list-style-type: none;"><ul><span style="font-family: 'Trebuchet MS', sans-serif;">The Frog Compiler which come with the SDK, compiles a given Dart code to java script, this could be on the client side. A future version of Chrome will ship with an embedded Dart VM, allowing your Dart code to run directly in Chrome without first being compiled to Java Script.</span>
</ul>
</li>
</ul>
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Libraries to play with</b>:</span><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><b>dart:core</b> This is the basic library used in majority of the scripts.</span><ul>
<li style="list-style-type: none;"><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">print() -> display basic text </span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Collection, Set, List, Queue -> group objects in collections</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Map -> manipulate key-value pairs </span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Date, TimeZone, Duration, Stopwatch -> specify dates and times</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">String, Pattern, RegExp -> use strings </span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">num, int, double, Math -> use numbers</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Future -> return a value before your operation is complete</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Comparable -> compare similar objects</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Iterable -> perform an operation on each item in a multi-item object </span></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<ul style="text-align: left;">
<ul>
</ul>
</ul>
<ul>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>dart:io</b> This library is used on the server side to connect with the outside world.</span><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">InputStream and OutputStream : Reading and writing data</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Files and directories : Open and close files</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Sockets: Connection to the network</span></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>dart:html</b>: API's for producing UI for webapps.</span><ul>
<li dir="" style="list-style-type: none;"><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"> Document and window: get global objects</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Element query and queryAll: find HTML elements</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Elements On property: add or remove events</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Collection Interfaces to operate on group of objects</span></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">There exists many more libraries and all you can see when you download the editor and play with it.</span><br />
<ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Good Pointers</b>:</span><ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">This is the first site you have to look for : <a href="http://www.dartlang.org/" style="color: #3465a4;">http://www.dartlang.org/</a></span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">All general Dart discussions happen over here. Have a problem?? Post it here. A helping hand will always be there from an open source community. Link: <a href="https://groups.google.com/a/dartlang.org/group/misc/topics?pli=1" style="color: #3465a4;">https://groups.google.com/a/dartlang.org/group/misc/topics?pli=1</a></span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Want some new features for Dart?? Want to contribute?? Want to file a bug?? Go here: <a href="http://dart.googlecode.com/" style="color: #3465a4;">http://dart.googlecode.com</a></span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Reference about API : <a href="http://api.dartlang.org/" style="color: #3465a4;">http://api.dartlang.org/</a></span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Great blog about Dart: <a href="http://dartosphere.org/" style="color: #3465a4;">http://dartosphere.org/</a></span></li>
</ul>
</li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;"> Happy Learning! :)</span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-65103950738829206622012-06-15T18:30:00.003+05:302012-06-15T18:32:26.213+05:30Start Darting ...<div dir="ltr" style="text-align: left;" trbidi="on">
<i><span style="font-family: 'Trebuchet MS', sans-serif;">Well, before Starting to Dart.. I wanna say few words.. :)<br />" Its really encouraging to see the audience and the amount of traffic here, in the pre-init phase of my new just born blog! :)<br />Awesome...!!! :):)<br />It really amazes, urges, kindles me to read, experiment and write, more and more and more. I wanna thank you all for the same " :)</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><i><br /></i><b>On Pitch</b>....!! ;)</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">It was at the GOTO Conference, (International Software Development Conference) 2011 @Aarhus, something unusual was revealed!</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">It was Dart (or Dash), a web based programming and the catch is "Its a replacement to Java script", by Google !!</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Programming languages are hot spicy topics for debate. ;)</span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif;">Why dump Java script and move on with a new platform? What are its demerits?</span></i><br />
<blockquote class="tr_bq">
<span style="font-family: 'Trebuchet MS', sans-serif;"><i><b>" ...</b></i><i> Writing a web app can be lots of fun, especially at the beginning when you experience<br />instant gratification: code, reload, repeat.<br />Unfortunately, finishing and maintaining a web app are not so fun. Java Script is great<br />for small scripts, and it has the performance chops to run large apps. But when a script<br />evolves into a large web app, debugging and modifying that app can be a nightmare,<br />especially when you have a large team <b>...</b></i><i>instant gratification: code, reload, repeat.</i><i>Unfortunately, finishing and maintaining a web app are not so fun. Java Script is great</i><i>for small scripts, and it has the performance chops to run large apps. But when a script</i><i>evolves into a large web app, debugging and modifying that app can be a nightmare,</i><i>especially when you have a large team <b>... </b></i><i>"</i></span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Trebuchet MS', sans-serif;"> says a member of Google’s Developer Relations Team!</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Trebuchet MS', sans-serif;">To be really frank here, this is the memo leaked from the employee of Google, for a pressing need of dart and replacement of Java Script.</span></blockquote>
<span style="font-family: 'Trebuchet MS', sans-serif;">Here, it goes:</span><br />
<a href="http://markmail.org/message/uro3jtoitlmq6x7t" style="color: #3465a4;"><span style="font-family: 'Trebuchet MS', sans-serif;">http://markmail.org/message/uro3jtoitlmq6x7t</span></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">The memo goes into some detail on what Dart would be, but doesn't go into much detail on why Dart should be. In other words, it doesn't explicitly state the deficiencies in Java Script.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Need a reason for "The Need" ?</span><br />
<br />
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Well, to recollect,</span></li>
</ul>
<br />
<br />
<span style="font-family: 'Trebuchet MS', sans-serif;">- Google with its V8 engine</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">- Apple with its Nitro engine</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">- Microsoft with its Chakra</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">- Mozilla with its Monkeys { No Offense ;) } </span><span style="font-family: 'Trebuchet MS', sans-serif;">etc.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"> have really struggled a lot to make serious improvements in Java Script execution. So, the order of magnitude is high in the modern browsers with the technique being used is JIT (Just In Time) compilation instead of interpretation. But, its really very challenging to write a high performance scripting engine specially one for java script like this and for all the effort and resources that's put in, the yield from java script is not up to the brim. Call it not "NEED FOR SPEED/ PERFORMANCE" ? (Dart Game.. Not a Car Game) ;)</span><br />
<ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Java script, being a prototype based language is like a boon and a bane! If the engine developers or the users do not want to follow prototypes, then java script should be no where around.</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Java script has features too and the problem that is conceived while importing third party libraries and the conflicts that occurs, the variable scopes is mind boggling. Java script, sometimes I feel has no good features, speaking accordingly to features. ;)</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Java script and its holy dynamic typing, has disadvantages too. Just, think of a situation where in the errors that are not detected at compile time, but soon in the browser, as the browser engine is left astray, pops up on the user interface! Ah, man!! This is not a pleasurable experience at all! So, its better to know the pre defined types, it also helps in faster code generation. As, the saying goes.. " It's better to be safe, than to be sorry! " :)</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">No hard feelings.. Java script has its own community of die hard fans.. when people say, Java script cannot do on the server side, they came up with node.js. So on...</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">It has all the zeal to do anything and everything, but we must wonder at the cost of what!! :)</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<b><span style="font-family: 'Trebuchet MS', sans-serif;">Play Darts:</span></b><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Okay! Have you played Darts??</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Let's Start Darting :) It's very similar to the game. (at least with terms) ;)</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">=> Wanna Say "Hello! Darts" ??</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Nice, Goto: <a href="http://try.dartlang.org/" style="color: #3465a4;">http://try.dartlang.org/</a> , and your code should look like this</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<i><span style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">// Welcome to try.dartlang.org<br />//<br />// Here you can try out the Dart Language from the comfort of your own<br />// *modern* web browser. When you run the code … it is submitted to AppEngine,<br />// translated into the equivalent JavaScript, and run right in your browser window.<br />//<br />// Ok, your turn!<br />main() {<br /> print('Hello! Darts');<br />}</span></i><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Now, Run the code and see in the console for your output. :)</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">The code looked similar to something. Feeling familiar ?? </span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Ah!! Well, Google met its requirement :p </span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Yes!! Google's main intent was to make the language look very familiar like Java or java script.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">What to play with ?</span><br />
<ul style="text-align: left;">
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Dartboard tool which lets you write and run Dart code in your browser.</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Dart Editor which lets you create,modify, and run Dart apps.</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> SDK contains command-line tools such as a Dart-to-Java Script compiler (which produces Java Script that you can put in any modern browser) and </span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Dart Virtual Machine (the VM, which lets you run Dart code on servers)</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">The latest tool to become available is a build of the Chromium browser, nicknamed Dartium, that contains a built-in Dart VM.</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">Well, lots more in store...</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Next blog post...</span><br />
<ul>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">New things in Dart</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Un boxing of Dart</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">Sample codes, libraries, running java script from dart</span></li>
<li dir="ltr"><span style="font-family: 'Trebuchet MS', sans-serif;">References and where to learn more and more?</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">Until then,</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;">Happy Learning! :)</span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-67184849535159990882012-06-08T18:53:00.001+05:302012-06-08T18:55:19.921+05:30Women & Tech World ...<div dir="ltr" style="text-align: left;" trbidi="on">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Social revolutions rarely are, especially when you're a woman trying to break into the boys' club that is "TECH WORLD". <br />I have been through stages from being a mundane person to a wannabe and a newbie, learning and very jealous to know more... and I will love to be in this newbie phase forever :)<br />In the course, Yeah !! I have felt that it is male dominated world. No doubt!<br /><br />But, we have to dispel the notion aforementioned and move ahead. The emergence of young female tech founders and executives reflects sweeping change in the worlds of start-up companies and angel funding, where wealthy investors give money in return for a stake in a company. Also, if we observe deeply, we are the heroines! :)<br />TELL ME WHY ?? ;)<br />Imagine, the Internet based companies, e-commerce based companies! Don't you notice that they are like ever-super-fast-growing and 24*7 engaged??<br />Yes, they are! :):)<br />It underscores the enormous purchasing prowess of women on-line that is transforming the Web economy. As more consumers reach for their smart phones and tablets to shop and communicate, there is a pressing need for commerce sites that cater to women, who control majority of on-line purchases worldwide.<br />So, I say we are the unsung heroines ;) <br />Cheers !! :):)<br /><br />On this positive note, head on... :)<br />If you are thinking to be an entrepreneur, you can always be one!<br />Before, it was very difficult for a female entrepreneur to start the company, as funding was equally a formidable task.<br />Of late, its a thrilling and bumpy ride for female entrepreneurs in tech. <br />It's cheaper to start a company today because Web servers and other equipment cost hundreds of thousands of dollars, not the millions once required for high-end computer servers. Newer technologies, such as cloud computing, reduce infrastructure costs. And coding isn't as onerous as it once was. Those changes have allowed entrepreneurs to build products faster and land funding sooner.<br /><br />We, can notice the influx of female entrepreneurs in this tech world. Its really awesome to be a women in tech and keeping very busy!<br />There are large number of female led groups whom you can really rely upon for the guidelines and also, there are superwomen in this field :):)<br />Its good to follow them!<br /><br />But it is a long journey, and there are many stages to that journey and a lot to juggle with, to be SUCCESSFUL. :)<br />I hope this post will be an inspiring note :)<br />All The Very Best! :)</span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-48785809077095270212012-06-06T21:24:00.001+05:302012-06-07T12:57:47.106+05:30Getting Started With ANTLR:Basics<div dir="ltr" style="text-align: left;" trbidi="on">
<span class="Apple-style-span" style="color: #fff2cc; font-family: Verdana, sans-serif;"></span><br />
<div style="line-height: 20px;">
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Yeah! It's after a lapse of a month or so that there is a post in this blog! :)<br />Well, this post drives you towards the basics of ANTLR. Previously, we had learnt about setting up of ANTLR as an external tool.<br />RECAP! It's here: <a href="http://femgeekz.blogspot.in/2012/06/antlr-as-external-tool-in-eclipse-on.html">ANTLR External Tool</a><br />:)<br />So, here we go.... <br /><b> </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>What is ANTLR?</b>• ANother Tool for Language Recognition, is a language tool that provides a framework for constructing recognizers, interpreters, compilers, and translators from grammatical descriptions containing actions.</span><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b> </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>What can be the target languages? </b></span><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"> </span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• Action Script, Ada </span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• C<br />• C#; C#2<br />• C#3<br />• D<br />• Emacs ELisp<br />• Objective C<br />• Java<br />• Java Script<br />• Python<br />• Ruby<br />• Perl6<br />• Perl<br />• PHP<br />• Oberon<br />• Scala<br /><b> </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>What does ANTLR support?</b>• Tree construction<br />• Error recovery<br />• Error handling<br />• Tree walking<br />• Translation<br /><b> </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>What environment does it support? </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">ANTLRWorks is the IDE for ANTLR. It is the graphical </span><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">grammar editor and debugger, written by Jean Bovet using Swing.<br /><b><br /></b></span></div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>What for ANTLR can be used? </b><br />• ""REAL"" programming languages </span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• domain-specific languages [DSL]<br /><b> </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Who is using ANTLR?</b><br />• Programming languages :Boo, Groovy, Mantra, Nemerle, XRuby etc. </span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• Other Tools: HIbernate, Intellij IDEA, Jazillian, JBoss Rules, Keynote(Apple), WebLogic(Oracle) etc.<b><br /></b></span></div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
</div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"> </span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Where is that you can look for ANTLR? </b></span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">You can always follow here<a href="http://www.blogger.com/%20http://www.antlr.org%20" style="text-decoration: none;"> http://www.antlr.org </a>• to download ANTLR and ANTLRWorks, which are free and open source</span></div>
<div style="font-size: 15px;">
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• docs,articles,wiki,mailing list,examples....</span></div>
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><span class="Apple-style-span" style="font-size: 15px;"> You can catch everything here!</span><br /><br /><b>Row your Boat....</b></span><br />
<div style="font-size: 15px;">
</div>
</div>
<ul style="line-height: 1.4; list-style-image: initial; list-style-position: initial; list-style-type: disc; margin-bottom: 0.5em; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 2.5em; padding-right: 2.5em; padding-top: 0px; text-align: left;">
<li style="border-bottom-style: none; border-color: initial; border-left-style: none; border-right-style: none; border-top-color: initial; border-top-style: none; border-width: initial; margin-bottom: 0.25em; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-indent: 0px;"><u><span style="font-family: Georgia, 'Times New Roman', serif;">Basic terms</span></u></li>
</ul>
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• <b>Lexer</b> : converts a stream of characters to a stream of tokens.<br />• <b>Parser</b> : processes of tokens, possibly creating AST<br />• <b>Abstract Syntax Tree(AST)</b>: an intermediate tree representation of the parsed input that is simpler to process than the stream of tokens. It can as well be processed multiple times.<br />• <b>Tree Parser</b>: It processes an AST<br />• <b>String Template</b>: a library that supports using templates with placeholders for outputting text </span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><br /></span><br />
<ul style="line-height: 1.4; list-style-image: initial; list-style-position: initial; list-style-type: disc; margin-bottom: 0.5em; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 2.5em; padding-right: 2.5em; padding-top: 0px; text-align: left;">
<li style="border-bottom-style: none; border-color: initial; border-left-style: none; border-right-style: none; border-top-color: initial; border-top-style: none; border-width: initial; margin-bottom: 0.25em; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-indent: 0px;"><u><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">General Steps</span></u></li>
</ul>
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">• Write Grammar in one or more files<br />• Write string templates[optional]<br />• Debug your grammar with ANTLRWorks<br />• Generate classes from grammar<br />• Write an application that uses generated classes<br />• Feed the application text that conforms to the grammar<br /><b><br /></b></span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>A Bit Further....</b><br /></span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Lets write a simple grammar which consists of<br />• Lexer<br />• Parser<br />Lexer: Breaks the input stream into tokens<br />Lets take the example of simple declaration type in C of the form "int a,b;" or "int a;" and same with float.</span><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">As we see we can write lexer as follows:</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">//TestLexer.g<br /><i>grammar TestLexer; </i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><i>ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'.'|'a'..'z'|'A'..'Z')*; COMMA: ','; </i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><i>SEMICOLON:';'; </i></span><br />
<i><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">DATATYPE: 'int' | 'float';</span></i><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><i><br /></i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><i><br /></i></span><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">As we could see, these were the characters that were to be converted to tokens. </span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">So, now lets write some rules which processes these tokens generated and may it create a parse tree accordingly.</span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /><span class="Apple-style-span" style="font-size: x-small;">//TestParser.g<br />grammar TestParser;<br />options {language : Java;}<br />decl:DATATYPE ID (',' ID)* ; </span> </span><br />
<br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Running ANTLR on the grammar just generates the lexer and parser,TestParser and TestLexer. To actually try the grammar on some input, we</span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">need a test rig with a main( ) method as follows:</span><br />
<br />
<span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">// Test.java<br />import org.antlr.runtime.*;<br />public class Test {<br /><br />public static void main(String[] args) throws Exception {<br /><br />// Create an input character stream from standard in<br />ANTLRFileStream input = new ANTLRFileStream("input"); // give path to the file input<br /><br />// Create an ExprLexer that feeds from that stream<br />TestLexer lexer = new TestLexer(input);<br />// Create a stream of tokens fed by the lexer<br />CommonTokenStream tokens = new CommonTokenStream(lexer);<br />// Create a parser that feeds off the token stream<br />TestParser parser = new TestParser(tokens);<br />// Begin parsing at rule decl<br />parser.decl();<br />}</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">}</span></span><br />
<br />
<br />
We shall see how to create<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"> an AST and walk over the tree in the next blog post...</span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Happy learning....! :)</span><br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif; font-size: x-small;"><br /></span>
<br />
<br />
<br />
<br />
<div style="color: #333333; font-size: 15px; line-height: 20px; text-align: left;">
</div>
<span class="Apple-style-span" style="color: #fff2cc; font-family: Verdana, sans-serif;">
</span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-60657251362286359512012-06-06T21:17:00.002+05:302012-06-07T13:01:09.968+05:30ANTLR as an external tool in eclipse on ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">This tutorial tells how to setup the ANTLR in your eclipse.<br /><b>STEP 1:</b><br />Download the jar file antlrworks-1.4.2.jar from <a href="http://www.antlr.org/download." style="text-decoration: none;">http://www.antlr.org/download.</a><br />Further details about ANTLRWorks: The ANTLR GUI Development Environment, follow the link : <a href="http://www.antlr.org/works/index.html" style="text-decoration: none;">http://www.antlr.org/works/index.html</a><br /><b>STEP 2:</b><br />Create a java project in eclipse as follows:<br />File->New->Project<br />Select Java and Java project.<br />Click on Next.<br />Name the project as "TestANTLR"<br />Press Finish.<br />Add the antlrworks-1.4.2.jar to the project classpath.<br />Right click on "TestANTLR" project .<br />Select Properties->Libraries.<br />Click on "Add External jar"<br />Select the complete path of the "antlrworks-1.4.2.jar" and press Ok.<br /><b>STEP 3: make it as an external tool</b><br />Goto Run->External Tools->Configure<br />Click on New.<br />Name: <i>ANTLR Compiler</i><br />Tool Location:<i> /usr/lib/jvm/java-6-sun-1.6.0.26/bin/java</i><br />// this must be the complete path to your java<br />Tool Arguments: -classpath complete_path_to_antlrworks-1.4.2.jar </span></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">org.antlr.Tool<i> ${resource_name}</i><br />Working Directory: <i>${container_loc}</i><br />Here, org.antlr.Tool is the main class which would take the ${resource_name} for processing.<br /><i>${resource_name}</i> and <i>${container_loc} </i>can be selected with "Browse Variables" option too.</span></span></div>
<div style="text-align: left;">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;"><br /></span></span></div>
<div style="text-align: left;">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">Going ahead :<br />*<b>Creating a grammar file</b><br />Create a grammar file with .g extension. Say, Example.g<br /><b><br /></b></span></span></div>
<div style="text-align: left;">
<span class="Apple-style-span" style="line-height: 20px;"><b><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">//sample code</span></b><br /><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">grammar Example;<br />start : 'hello' ID ';' {System.out.println("hiii... "+$ID.text);} ;<br />ID: 'a'..'z' + ;<br />WS: (' ' |'\n' |'\r' )+<br />{$channel=HIDDEN;}</span><br /><b style="font-family: Verdana, sans-serif; font-size: 15px;"><br /></b></span></div>
<div style="text-align: left;">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;"><b>*Running the above code:</b><br />Run->External Tools->ANTLR Compiler<br />Press F5 or right click on the project and "refresh"<br />all you can see is a lexer and parser files generated with the tokens.</span></span></div>
<div style="text-align: left;">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;"><br /></span></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">In our example,<br />ExampleLexer.java , ExampleParser.java and Example.tokens<br /></span></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">Create Main.java program in the same project with the following code:<br /></span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small; line-height: 20px;">import org.antlr.runtime.*;<br />public class Main {<br />public static void main(String[] args) throws Exception {<br />// create a CharStream that reads from standard input<br />ANTLRInputStream input = new ANTLRInputStream(System.in);<br />// create a lexer that feeds off of input CharStream<br />ExampleLexer lexer = new ExampleLexer(input);<br />// create a buffer of tokens pulled from the lexer<br />CommonTokenStream tokens = new CommonTokenStream(lexer);<br />// create a parser that feeds off the tokens buffer<br />ExampleParser parser = new ExampleParser(tokens);<br />// begin parsing at rule start<br />parser.start();<br />}</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small; line-height: 20px;">} </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;"><br /></span></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">Set the arguments in the Run configurations and click on Apply and Run.</span></span></div>
<div style="text-align: left;">
</div>
<div style="text-align: left;">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">Now you have the output at console.</span></span></div>
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;">:)</span></span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0tag:blogger.com,1999:blog-6187879032785791598.post-66415401103079747412012-06-06T18:00:00.003+05:302012-06-07T13:02:00.720+05:30Say Hello To ANDROID: [Part I] Introduction<div dir="ltr" style="text-align: left;" trbidi="on">
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;"> </span><span class="Apple-style-span" style="font-size: 15px; line-height: 20px;"> Its an era of mobile, tablets and a very exciting time for developers!! Mobile phones have never been more popular and powerful. Smart Phones have become very stylish, versatile packing hardware features like GPS, accelerometers, etc. are an enticing platform which kindles the developers to create some innovative mobile applications.<br /><br /> With the existing mobile development built on proprietary operating systems that restrict third-party applications, Android offers an open and the best alternative. Without artificial barriers, Android developers are free to write applications that take full advantage of increasingly powerful mobile hardware. As a result, developer interest in Android devices has made their 2008 release a hugely anticipated mobile technology event.<br /><br /> The open philosophy of android is most welcomed by majority of the developers as it features very powerful SDK libraries. Experienced mobile developers can now tinker with android and explore the platform, leveraging the unique features to enhance existing products or create more innovative ones.<br /><br /><b>ANDROID</b><br /> Its an open source software stack that includes<br />• Operating System<br />• Middle ware<br />• Key Applications + set of API Libraries<br />which changes the look, feel and function of the mobile.<br /><br /> In Android, native and third-party applications are written using the same APIs and executed on the same run time. These APIs feature hardware access, location-based services, support for background services, map-based activities, relational databases, inter-device peer-to-peer messaging and 2D and 3D graphics.<br /><br /><b>Just A Flashback </b>...<br /><br /> Historically, the developers had to code in low level C or C++ and had to learn the specific hardware feature upon which they were coding. But, as the hardware features enhanced this became more cumbersome. More recently, Symbian was far successful in giving a room for the developers to better leverage the hardware available. However, it required writing complex C/C++ code and making heavy use of proprietary APIs that are notoriously difficult to use. This difficulty was amplified when developing applications that must work on different hardware implementations and particularly true when developing for a particular hardware feature like GPS.<br /><br /> Then came the Java hosted MIDIlets that are executed on the same JVM, abstracting the underlying hardware and letting the developers create apps that run on wide variety of hardware that supports Java run time.<br /><br /> So our Andy is here... :) Ouch! forgot about Apple's iPhone, Windows mobiles ?? ;) No offence!<br /> They provide richer UI, UX and simplified development environment! "BUT" they’re built on proprietary operating systems that often prioritize native applications over those created by third parties and restrict communication among applications and native phone data.<br /><br /> Third-party and native Android applications are written using the same APIs and are executed on the same run time. Users can remove and replace any native application with a third-party developer alternative; even the dialer and home screens can be replaced...{ Wanna Try ?? :p }<br /><br /> <b>Google</b> describes <b>Android</b> as:<br />"The first truly open and comprehensive platform for mobile devices, all of the software to run a mobile phone but without the proprietary obstacles that have hindered mobile innovation."<br /> <a href="http://googleblog.blogspot.com/2007/11/wheres-my-gphone.html" style="text-decoration: none;">http://googleblog.blogspot.com/2007/11/wheres-my-gphone.html</a> :)<br /><br /> The FLASHBACK is incomplete...... without the introduction of OHA { Oh Haa ?? ;) }<br /><br /><b>OHA</b>: <b>Open Handset Alliance</b> comprises of<br />• Developers<br />• Hardware Manufacturers<br />• Mobile Carriers<br /><br />The tech companies involved prominently are Motorola, HTC, T-Mobile, Qualcomm and their words:<br />"A commitment to openness, a shared vision for the future, and concrete plans to make the vision a reality. To accelerate innovation in mobile and offer consumers a richer, less expensive, and better<br />mobile experience."<br /> <a href="http://www.openhandsetalliance.com/oha_faq.html" style="text-decoration: none;">http://www.openhandsetalliance.com/oha_faq.html</a> :)<br /><br /> Android offers an excellent enterprise platform, has targeted developers making their lives far simple with Google and the OHA betting that the way to deliver<br />better mobile software to consumers is by making it easier for developers to write it themselves. This openness and power ensure that anyone with the inclination can bring a vision to life at minimal cost.<br /><br /> Use Open source softwares, share them and give back something better to the community..... !! :)<br /><br /> Next post...<br />• Merits of android<br />• unboxing SDK features, Software stack and lots more ..... :)<br /><br /><br /> Well, <a href="http://developer.android.com/" style="text-decoration: none;">http://developer.android.com</a> is a great reference site!<br /> <br /> Happy Learning! :)</span></span></div>Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0