First of all - I would like to thank you all for following my posts, for all those lovely interactions on and off the blog and encouraging me to do better, all this time!
Its time to say goodbye to the Blogger, as the very experience of writing posts here has become such a pain!! Adding to it, there has been no update to the blogspot, since a long time! I understand that they are busy changing logos! Sure! so be it!!
So, for the obvious reasons - its time to move on!
I've moved all the posts on this site to my new site : http://femgeekz.github.io/, based on a simple jekyll theme.
The site is still a work in progress, but that shouldn't stop me from posting new content, with all new variety!
This site is no longer being updated. Please visit : http://femgeekz.github.io/
Thanks for stopping by! :)
Adieu!
- Swatz
Wednesday, 2 September 2015
Saturday, 3 January 2015
Apache Mesos : Writing your own distributed framework
In the previous post, we saw what mesos is, how is it useful and getting started with it. In this post, we shall see how to write your own framework on mesos.
(In mesos, a framework is any application running on it.)
This post explains about a framework called "mesos-pinspider" which fetches the user profile information and user board information of a pinterest page of a user.
- Driver which submits the tasks to the framework
- Scheduler which registers with the master to be offered resources, takes the tasks and runs them on executor
- Executor process that is launched on slave nodes to run the framework’s tasks
You may check the code here on github. Let's break it down to PinDriver, PinScheduler and PinUserProfileExecutor.
Note: Please note that two ExecutorInfo are used ie. one for fetching user profile information and the other one for user board information for demonstration. This explanation involves only one executorinfo - userProfileExecutorInfo
Executor is a callback interface which is implemented by frameworks' executors. In our implementation, let us concentrate on launchTask()
Scheduler is a callback interface to be implemented by frameworks' schedulers. In our implemenation, let us concentrate on resourceOffers(), statusUpdate() and frameworkMessage()
Happy Learning! :)
(In mesos, a framework is any application running on it.)
This post explains about a framework called "mesos-pinspider" which fetches the user profile information and user board information of a pinterest page of a user.
Mesos Framework
In general, a Mesos framework has three basic components.- Driver which submits the tasks to the framework
- Scheduler which registers with the master to be offered resources, takes the tasks and runs them on executor
- Executor process that is launched on slave nodes to run the framework’s tasks
Pinspider Framework Example
You may check the code here on github. Let's break it down to PinDriver, PinScheduler and PinUserProfileExecutor.
Driver
The driver component of the framework is PinDriver.- Create Executor Info
Protos.ExecutorInfo userProfileExecutorInfo = Protos.ExecutorInfo.newBuilder().setExecutorId(Protos.ExecutorID.newBuilder().setValue("PinUserProfileExecutor")).setCommand(commandInfoUserProfile).setName("PinUserProfileExecutor Java").setSource("java").build();
- Create Framework Info
Protos.FrameworkInfo.Builder frameworkBuilder = Protos.FrameworkInfo.newBuilder().setFailoverTimeout(120000).setUser("").setName("Pinspider Framework");
- Instantiate Scheduler
Scheduler scheduler = args.length == 1 ?new PinScheduler(userProfileExecutorInfo,userBoardExecutorInfo) : new PinScheduler(userProfileExecutorInfo, userBoardExecutorInfo, Integer.parseInt(args[1]), args[2]);
Note: Please note that two ExecutorInfo are used ie. one for fetching user profile information and the other one for user board information for demonstration. This explanation involves only one executorinfo - userProfileExecutorInfo
- Starting the mesos scheduler driver.
MesosSchedulerDriver schedulerDriver = new MesosSchedulerDriver(scheduler,frameworkBuilder.build(), args[0]);int status = schedulerDriver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1;schedulerDriver.stop();System.exit(status);
Executor Implementation
The Executor component of the framework is PinUserProfileExecutor.Executor is a callback interface which is implemented by frameworks' executors. In our implementation, let us concentrate on launchTask()
@Override public void launchTask(final ExecutorDriver executorDriver, final Protos.TaskInfo taskInfo) {}
- Set the task status by setting the ID and the state with a builder pattern.
Protos.TaskStatus taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId()).setState(Protos.TaskState.TASK_RUNNING).build();
- Send the status update to the framework scheduler retrying as necessary until an acknowledgement has been received or the executor is terminated, in which case, a TASK_LOST status update will be sent.
executorDriver.sendStatusUpdate(taskStatus);
- Get the data from the tasks and run your logic.
try { message = ("userprofile :" + getUserProfileInfo(url)).getBytes(); } catch (IOException e) { LOGGER.error("Error parsing the Pinterest URL :" + e.getMessage()); }
- Send the framework the message.
executorDriver.sendFrameworkMessage(message);
- Mark the state of the task as finished and send the status update to the framework scheduler.
taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId()) .setState(Protos.TaskState.TASK_FINISHED).build();executorDriver.sendStatusUpdate(taskStatus);
- main() method to create an instance of MesosExecutorDriver and run
mesosExecutorDriver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1
Scheduler Implementation
The Scheduler component of the framework is PinScheduler.Scheduler is a callback interface to be implemented by frameworks' schedulers. In our implemenation, let us concentrate on resourceOffers(), statusUpdate() and frameworkMessage()
- Constructor : construct with the executor information and the number of launch tasks.
public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor , Protos.ExecutorInfo pinUserBoardExecutor ) {
this(pinUserProfileExecutor,pinUserBoardExecutor, 5, "http://www.pinterest.com/techcrunch");
}
public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor,Protos.ExecutorInfo pinUserBoardExecutor, int totalTasks, String url) { this.pinUserProfileExecutor = pinUserProfileExecutor; this.pinUserBoardExecutor = pinUserBoardExecutor; this.totalTasks = totalTasks; this.crawlQueue = Collections.synchronizedList(new ArrayList<String>()); this.crawlQueue.add(url); }
- Resource Offers
- A resource offer can be resources like CPU, memory etc. From the offers list, get the scalar value of the resources. We need to give our requirements of resources for the tasks while setting the task info.
for (Protos.Offer offer : list) { List<Protos.TaskInfo> taskInfoList = new ArrayList<Protos.TaskInfo>(); double offerCpus = 0; double offerMem = 0; for (Protos.Resource resource : offer.getResourcesList()) { if (resource.getName().equals("cpus")) { offerCpus += resource.getScalar().getValue(); } else if (resource.getName().equals("mem")) { offerMem += resource.getScalar().getValue(); } } LOGGER.info("Received Offer : " + offer.getId().getValue() + " with cpus = " + offerCpus + " and mem =" + offerMem);
- Create task ID.
Protos.TaskID taskID = Protos.TaskID.newBuilder().setValue(Integer.toString(launchedTasks++)).build();
- Create task info by setting task ID, adding resources, setting data and setting executor.
Protos.TaskInfo pinUserProfileTaskInfo = Protos.TaskInfo.newBuilder().setName("task " + taskID.getValue()).setTaskId(taskID).setSlaveId(offer.getSlaveId()).addResources(Protos.Resource.newBuilder().setName("cpus").setType(Protos.Value.Type.SCALAR).setScalar(Protos.Value.Scalar.newBuilder().setValue(CPUS_PER_TASK))).addResources(Protos.Resource.newBuilder().setName("mem").setType(Protos.Value.Type.SCALAR).setScalar(Protos.Value.Scalar.newBuilder().setValue(MEM_PER_TASK))).setData(ByteString.copyFromUtf8(crawlQueue.get(0))).setExecutor(Protos.ExecutorInfo.newBuilder(pinUserProfileExecutor)).build();
- Launch the tasks through the SchedulerDriver.
... taskInfoList.add(pinUserProfileTaskInfo);taskInfoList.add(pinUserBoardTaskInfo);}schedulerDriver.launchTasks(offer.getId(), taskInfoList);
- Status update
@Override public void statusUpdate(SchedulerDriver schedulerDriver, Protos.TaskStatus taskStatus) {... }
- Stop the SchedulerDriver if tasks are finished
if (taskStatus.getState() == Protos.TaskState.TASK_FINISHED) { finishedTasks++; LOGGER.info("Finished tasks : " + finishedTasks); if (finishedTasks == totalTasks) { schedulerDriver.stop(); } }
- Abort the SchedulerDriver if the tasks are killed, lost or failed
if (taskStatus.getState() == Protos.TaskState.TASK_FAILED
|| taskStatus.getState() == Protos.TaskState.TASK_KILLED
|| taskStatus.getState() == Protos.TaskState.TASK_LOST) {
LOGGER.error("Aborting because the task " + taskStatus.getTaskId().getValue() + " is in unexpected state : "
+ taskStatus.getState().getValueDescriptor().getName() + "with reason : " + taskStatus.getReason()
.getValueDescriptor()
.getName()
+ " from source : " + taskStatus.getSource().getValueDescriptor().getName() + " with message : "
+ taskStatus.getMessage());
schedulerDriver.abort();
}
- Framework Message
- Handle your message
Complete code is available here with the instructions to run and sample output.@Override public void frameworkMessage(SchedulerDriver schedulerDriver, Protos.ExecutorID executorID, Protos.SlaveID slaveID, byte[] bytes) { String data = new String(bytes); System.out.println(data); LOGGER.info("User Profile Information : " + data); }
Happy Learning! :)
Labels:
apache mesos,
distributed,
distributed systems,
framework,
linux,
mesos,
pinterest
Location:
Los Angeles, CA, USA
Saturday, 20 December 2014
Introduction and getting started with Apache Mesos
Introduction to Apache Mesos
In this era of distributed computing, where we spin up clusters for Hadoop, Storm, Jenkins, Cassandra, etc separately, we are not making effective use of the resources. There would be long pauses in the cluster after entering a burst of information, thus making it very in-efficient.Now, what if all these frameworks shared the same set of machines and resources, then small slices of time spent waiting for some resources could be granted to other frameworks. This is the concept of Time Sharing.
Apache Mesos is a datacenter operating system and it shares the same philosophy of time-sharing. Mesos is called a datacenter because it hosts different frameworks under a single roof. It is called an operating system because shares many concepts of Linux.
1. Isolation : Linux creates isolation through processes where, each of these processes has its own file descriptors and its own address space. This is achieved by Linux Containers (wiki:LXC) in Mesos
2. Process Scheduler : The processes have accesses to the system resources by balancing the work loads across multiple computing resources, thereby optimizing resources, maximize the throughput, minimizing the response time and avoiding the overhead by any one resource. There are various scheduling algorithms to execute more than one process at a time (wiki:Multitasking) and also transmit multiple data streams simultaneously across a single physical channel(wiki:Multiplexing). Mesos uses such scheduling algorithms.
3. Common Infrastructure : Linux has a set of calls irrespective of filesystems, drivers etc. Similarly, Mesos has a common set of calls which helps in the execution of tasks.
4. Package Manager : Linux has apt-get, aptitude, synaptic, yum etc that helps in the automation of the process of installing, upgrading, configuring, and removing software. Similarly, Mesos has a recent support for Docker(wiki:Docker)
Traditionally, distributed systems has 2 components in a non-peer to peer systems.
1. Coordinator : Generate tasks, send the tasks to worker and receive results from Worker.
2. Worker : Execute the tasks and send the status and results back to Worker.
With Mesos, there are three levels Coordinator, Mesos master and Mesos slaves where coordinator negotiates with mesos master and then master decides on partitioning the cluster to distribute the tasks. Thus, we can schedule jobs across the machines, thereby running hadoop, cassandra, spark etc.
All the distributed systems that run on Mesos are called applications or frameworks and the coordinator is called as scheduler in Mesos vocabulary.
How does Mesos work?
In summary, Mesos works on a request/offer based model. Whenever, you want to run a job, you send a request. These requests are simplified subset of specification like number of GPUs, RAM etc, at that point of time. Mesos, checks for the request specification and it will reply back with the resource offers of what resources are available on a set of machines. This is non-blocking and has two level of scheduling : Offering and Scheduling.Mesos master: Control the resource allocation to the schedulers
Scheduler: Uses the resource offers to decide which tasks to run and which one to run next.
More information on Mesos architecture is here : Mesos Architecture
Getting started with mesos
1. Download the tarball from the Mirror Apache Mesos v0.21.0 and untar it.
tar -zvxf mesos-0.21.0.tar.gz
cd mesos-0.21.0/
2. Install the dependencies
sudo apt-get update
sudo apt-get install build-essential openjdk-6-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev
3. Building Mesos
Please make sure it has appropriate permissions while building.
mkdir build
cd build
../configure
make
make check
make install
Start Mesos Master
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos
Start Mesos Slave
./bin/mesos-slave.sh --master=127.0.0.1:5050
Web UI
http://127.0.0.1:5050
Running a test framework in Java
./src/examples/java/test-framework 127.0.0.1:5050
![]() |
| Mesos Home |
![]() |
| Mesos Frameworks |
![]() |
| Mesos executor tasks |
![]() |
| Mesos Slave Nodes |
Happy Learning! :)
Labels:
apache mesos,
clusters,
distributed systems,
hadoop,
jenkins,
mesos,
open source,
spark,
storm,
ubuntu
Location:
Los Angeles, CA, USA
Tuesday, 5 August 2014
Raspberry Pi B+ : Connect to your Pi with no display monitor, keyboard or mouse
The new Raspberry Pi B+ was released on July 14th 2014. The following link does all the justice to know about the improved features of B+. Please refer it here : Raspberry Pi B+
The moment I had this credit card sized thing on my palm, I was so excited to get started with it. I downloaded Raspbian from Raspberry Pi Downloads Page and unzipped it. This is the image file 2014-06-20-wheezy-raspbian.img you get after the unzip operation. I used the SanDisk 8GB Class 4 Micro SD card and dd tool to write to it. Refer Installing images on Linux here ( Please note, I did bs=1M with dd tool while writing). This was one of the guide I was following Quick Start Guide
But then, wait! Do you need a HDMI TV or display monitor, USB keyboard and a USB mouse to connect to it? Oh really!? I was not interested to invest any further.
Here is what I did:
The moment I had this credit card sized thing on my palm, I was so excited to get started with it. I downloaded Raspbian from Raspberry Pi Downloads Page and unzipped it. This is the image file 2014-06-20-wheezy-raspbian.img you get after the unzip operation. I used the SanDisk 8GB Class 4 Micro SD card and dd tool to write to it. Refer Installing images on Linux here ( Please note, I did bs=1M with dd tool while writing). This was one of the guide I was following Quick Start Guide
But then, wait! Do you need a HDMI TV or display monitor, USB keyboard and a USB mouse to connect to it? Oh really!? I was not interested to invest any further.
Here is what I did:
- I inserted the Micro SD card into the slot behind.
- I had a micro USB cable my for Nexus 5. I used the same for the 5V power supply
- I had an Ethernet crossover cable already with me for some VPN stuffs. I took one of them and connected one end to the router and the other end to the Pi Ethernet Port (next to 4 USB Ports)
- Next, I used my laptop to connect to my Pi in the same network.
- On my laptop,
- Opened Terminal. Typed in $ ifconfig to get to know the IP address.
- For ex: if inet addr:192.168.0.101 then our Pi should have inet addr:192.168.0.*
- Next, for my rescue came the nmap tool. If you do not have it already installed, $ sudo apt-get install nmap
- Execute $ nmap -T4 -F 192.168.0.* on the terminal ( as per the example considered ) This will scan all the hosts which are up and then lists out the open ports. You will find Pi as one among them.
- If you like GUI Based, you may like Zenmap and can be installed by $ sudo apt-get install zenmap.
- After installing, type $ sudo zenmap
- Type Target as 192.168.0.*
- Choose Profile as Quick Scan
- Click on Scan
- This will list the Raspberry Pi MAC Address and the IP Address
- Now, you know the IP Address of Pi. The default login for Raspbian is username "pi" with the password "raspberry"
- Time to ssh!
- Need to change your configuration settings after login? Type $ sudo raspi-config and change it accordingly!
Happy hacking! :)
Labels:
model B+,
no display monitor,
no keyboard,
no mouse,
pi,
raspberry pi,
ubuntu
Location:
Los Angeles, CA, USA
Tuesday, 18 June 2013
One Cap to rule 'em all ...
Wondering which cap it could be?
....Well its Capistrano :D
I am a fan of Capistrano from way back and we use it for almost all kind of deployments - Hadoop, MongoDB clusters and so on.
If you have not tried Capistrano, you must try it and figure out how you can use for deployments in your environment.
Its highly configurable - so capify your stuffs!
Checkout the capified scripts to Deploy a replicated sharded MongoDB cluster on AWS EC2 instances in the following link
https://github.com/SwathiMystery/deploy_shard_mongodb
Feel free to experiment, report bugs/ issues and contribute back.
For more details, follow the link below :
https://github.com/SwathiMystery/deploy_shard_mongodb/blob/master/README.md#deploy-replicated-sharded-mongodb-cluster
....Well its Capistrano :D
I am a fan of Capistrano from way back and we use it for almost all kind of deployments - Hadoop, MongoDB clusters and so on.
If you have not tried Capistrano, you must try it and figure out how you can use for deployments in your environment.
Its highly configurable - so capify your stuffs!
Checkout the capified scripts to Deploy a replicated sharded MongoDB cluster on AWS EC2 instances in the following link
https://github.com/SwathiMystery/deploy_shard_mongodb
Feel free to experiment, report bugs/ issues and contribute back.
For more details, follow the link below :
https://github.com/SwathiMystery/deploy_shard_mongodb/blob/master/README.md#deploy-replicated-sharded-mongodb-cluster
Labels:
aws,
cap,
capistrano,
deployments,
EC2,
Hadoop cluster,
mongodb,
open source,
replica sets,
scripts,
sharding,
ubuntu
Location:
United States
Monday, 15 April 2013
Monitoring S3 uploads for a real time data
If you are working on Big Data and its bleeding edge technologies like Hadoop etc., the primary thing you need is a "dataset" to work on. So, this data can be reviews, blogs, news, social media data (Twitter, Facebook etc), domain specific data, research data, forums, groups, feeds, fire hose data etc. Generally, companies reach the data vendors to fetch such kind of data.
Normally, these data vendors dump the data into a shared server kind of environment. For us to use this data for processing with MapReduce and so forth, we move them to S3 for storage first and processing next. Assume, the data belong to social media such as Twitter or Facebook, then the data can be dumped according to the date format directory. Majority of the cases, its the practice.
Also assuming 140-150GB /day being dumped in a hierarchy like 2013/04/15 ie. yyyy/mm/dd format, stream of data, how do you
- upload them to s3 in the same hierarchy to a given bucket?
- monitor the new incoming files and upload them?
- save the space effectively on the disk?
- ensure the reliability of uploads to s3?
- clean if the logging is enabled to track?
- re-try the failed uploads?
These were some of the questions, running at the back of my mind, when I wanted to automate the uploads to S3. Also, I wanted 0 human intervention or at-least the least!
So, I came up with
- s3sync / s3cmd.
- the python Watcher script by Greggory Hernandez, here https://github.com/greggoryhz/Watcher
A big thanks! This helped me with monitoring part and it works so great!
- few of my own scripts.
What are the ingredients?
Normally, these data vendors dump the data into a shared server kind of environment. For us to use this data for processing with MapReduce and so forth, we move them to S3 for storage first and processing next. Assume, the data belong to social media such as Twitter or Facebook, then the data can be dumped according to the date format directory. Majority of the cases, its the practice.
Also assuming 140-150GB /day being dumped in a hierarchy like 2013/04/15 ie. yyyy/mm/dd format, stream of data, how do you
- upload them to s3 in the same hierarchy to a given bucket?
- monitor the new incoming files and upload them?
- save the space effectively on the disk?
- ensure the reliability of uploads to s3?
- clean if the logging is enabled to track?
- re-try the failed uploads?
These were some of the questions, running at the back of my mind, when I wanted to automate the uploads to S3. Also, I wanted 0 human intervention or at-least the least!
So, I came up with
- s3sync / s3cmd.
- the python Watcher script by Greggory Hernandez, here https://github.com/greggoryhz/Watcher
A big thanks! This helped me with monitoring part and it works so great!
- few of my own scripts.
What are the ingredients?
- Installation of s3sync. I have just used one script of s3cmd here and not s3sync in real. May be in future -- so I have this.
- Installation of Watcher.
- My own wrapper scripts.
- cron
Next, having set up of the environment ready, lets make some common "assumptions".
- Data being dumped will be at /home/ubuntu/data/ -- from there it could be 2013/04/15 for ex.
- s3sync is located at /home/ubuntu
- Watcher repository is at /home/ubuntu
Getting our hands dirty...
- Goto Watcher and set the directory to be watched for and corresponding action to be undertaken.
- Create a script called monitor.sh to upload to s3 in s3sync directory as below.
- The variables you may like to change is s3bucket path in "s3path" in monitor.sh
- This script will upload the new incoming file detected by the watcher script in the reduced redundancy storage format. (you can remove the header -- provided you are not interested to store in RRS format)
- The script will call s3cmd ruby script to upload recursively and thus maintains the hierarchy ie. yyyy/mm/dd format with files *.*
- It will delete the file successfully uploaded to s3 from the local path -- to save the disk space.
- The script would not delete the directory, as it will be taken care by yet another script re-upload.sh, which acts as a backup for the failed uploads to be uploaded again to s3.
- Create a script called re-upload.sh which will upload the failed file uploads.
- This script ensures that the files that are left over from monitor.sh (failed uploads -- this chance is very less. May be 2-4 files/day. -- due to various reasons.), will be uploaded to s3 again with the same hierarchy in RRS format.
- Post successful upload, deletes the file and hence the directory if empty.
- Now, more dirtiest work -- Logging and cleaning logs.
- All the "echo" created in monitor.sh can be found in ~/.watcher/watcher.log when the watcher.py is running.
- This log helps us initially and may be later too, to backtrack errors or so.
- Call of duty - Janitor for cleaning logs. To do this, we can use cron to run a script at sometime. I was interested to run - Every Saturday at 8.00 AM
- Create a script to clean log as "clean_log.sh" in /home/ubuntu/s3sync
- Time for cron
- All set! logging clean happens every Saturday 8.00 AM and re-upload script runs for the previous day, to check if files exist and does the cleaning accordingly.
- Let's start the script
So, this assures successful uploads to S3.
My bash-fu with truth! ;)
Happy Learning! :)
Labels:
automation,
monitor files,
monitoring,
real time data,
s3,
s3cmd,
s3sync,
ubuntu,
upload to s3,
uploads,
watcher
Location:
Bangalore, Karnataka, India
Tuesday, 19 March 2013
Phabricator - pretty()
Ah pretty() -- liking it too much! These days I'm into MongoDB so.. :P
Its like 3am ish now and I know, I'm blabbering ... Gotta sleep!
Ah! please bear with rest of my ramblings!
Last month, I was looking into some of the code review tools!
You know - reviewing the review tools! ;)
Well, I experimented with reviewboard, gerrit, barkeep.
Gerrit does not support post-commit review. I was looking for, a way to conduct code review, after the commit is pushed to git.
Reviewboard is cool. Its existing from a longer time. I experimented a lot on the cloud. Well, the post-review process was little painful with command line.
Barkeep installation is kind of different. I did not invest much time. But it looks interesting!
I went with Phabricator, released by Facebook and Open source too.
Wanna contribute ? Fork here https://github.com/facebook/phabricator/ : else go http://phabricator.org/
Okay I don't wanna geeko-Fi more with operators ;)
Well, Phabricator is used by many companies like Quora (unsure if they are presently using it or not, have a doubt , as I read in some quora thread), DropBox, Path (Android app), Disqus (online commenting system), deviantArt( Painting app -- famous in Chrome web store) .... Ah well, lots more!!
It has complete software tools. Ok!! You want ...
Its like 3am ish now and I know, I'm blabbering ... Gotta sleep!
Ah! please bear with rest of my ramblings!
Last month, I was looking into some of the code review tools!
You know - reviewing the review tools! ;)
Well, I experimented with reviewboard, gerrit, barkeep.
Gerrit does not support post-commit review. I was looking for, a way to conduct code review, after the commit is pushed to git.
Reviewboard is cool. Its existing from a longer time. I experimented a lot on the cloud. Well, the post-review process was little painful with command line.
Barkeep installation is kind of different. I did not invest much time. But it looks interesting!
I went with Phabricator, released by Facebook and Open source too.
Wanna contribute ? Fork here https://github.com/facebook/phabricator/ : else go http://phabricator.org/
Okay I don't wanna geeko-Fi more with operators ;)
Well, Phabricator is used by many companies like Quora (unsure if they are presently using it or not, have a doubt , as I read in some quora thread), DropBox, Path (Android app), Disqus (online commenting system), deviantArt( Painting app -- famous in Chrome web store) .... Ah well, lots more!!
- What's cool about Phabricator?
It has complete software tools. Ok!! You want ...
- Code Review ? Differential
- Post Commit Review ? Audits
- Code Browser ? Diffusion
- Bug Tracking ? Maniphest
- Wiki ? Phriction
- Want more ? It's under development... So you gonna get more too!
Well, personally, I prefer using Audits, Diffusion, Maniphest.
Wiki - is used in Confluence/BitBucket -- so for now not essential.
Code Review -- Pre-commit reveiw process stalls the development in agile kinda environment. So not going with it.
- How's the installation?
I read a question in Quora that why is it difficult to install Phabricator. I believe its not so difficult too.
May be I will post -- how to install/configure everything with Phabricator.
Please bear with the mentions about Quora, I'm a quor(a)ddict! :D
- How to start with Audits?
To get started with the workflow, it does take time, probably because its still under development and documentation demands update. But the greatest thing, I love about Phabricator is, its still under the development and yet, functions -- as it purports to be!! That's Amazing! The UI is lovely too!
I faced this issue -- If you have huge repository, where codereview has never happened before, how do you start the process with previous commits?
Unfortunately, you cannot do with existing commits! The commits that occur after installation/configuration of Phabricator with that repository, audits are possible.
You will have to create some rules that trigger post commit audits. Its called Herald in Phabricator.
I found a workaround for the old commits -- if you wanna review -- You can browse the repository with Diffusion, goto the module / code you are interested in and click on the commit number.
Upon clicking on commit number, you have the freedom to post in-line comments/review comments, where even the diff is visible.
After finishing your commenting process, you can cook it with Raise Concern. I also prefer mapping this with a task tracker -- I mean bug tracker ( Maniphest ) -- add the assignee, link to the comments you gave in description and assign.
This will send a mail to the team.
Email configuration is one big task.
I used Postfix with SMTP as configuring outbound mail and have updated in github Phabricator.
Mapping Your Projects/Audits/Team
This takes a lot of time and effort too.
- First, you need to create accounts for all.
- You will have to configure your remote repository (using git).
- Map the modules with the module owners and team lead.
- Create herald rule which triggers audits.
So, you can set up completely in 2days. But its fun!! Audits can be real fun..!! :)
I will really recommend Phabricator! :D
Happy Tweaking! :)
Wednesday, 20 February 2013
Books that every developer must read!
I have a rack-space of books! I just wanted to make a wishlist of all the books which I've read and yet to read! :)
1.
Design Patterns: Elements of Reusable Object-Oriented Software
Erich Gamma Richard Helm Ralph Johnson John Vlissides

2.
Structure And Interpretation Of Computer Programs, Second Edition : By Harold Abelson and Gerald Jay Sussman

3.
Refactoring to Patterns
Joshua Kerievsky

4.
Types and Programming Languages
Benjamin C. Pierce

5.
Code: The Hidden Language of Computer Hardware and Software
Charles Petzold

6.
Object-Oriented Analysis and Design with Applications (2nd Edition)
Grady Booch

7.
Code Complete: A Practical Handbook of Software Construction, Second Edition
Steve McConnell

8.
The Design of the UNIX Operating System [Prentice-Hall Software Series]
Maurice J. Bach

9.
The Pragmatic Programmer: From Journeyman to Master
Andrew Hunt David Thomas

10.
Practical API Design: Confessions of a Java Framework Architect
Jaroslav Tulach

11.
The Practice of Programming (Addison-Wesley Professional Computing Series)
Brian W. Kernighan Rob Pike

12.
Programming Pearls (2nd Edition)
Jon Bentley

13.
Writing Secure Code, Second Edition
Michael Howard David LeBlanc

14.
The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition)
Frederick P. Brooks Jr.

15.
Patterns of Enterprise Application Architecture
Martin Fowler

16.
Introduction to Functional Programming (Prentice Hall International Series in Computing Science)
Richard Bird

17.
The Art of Computer Programming
Donald E. Knuth

18.
Effective Java (2nd Edition)
Joshua Bloch

19.
Thinking in Java (4th Edition)
Bruce Eckel

20.
Programmers at Work: Interviews With 19 Programmers Who Shaped the Computer Industry (Tempus)
Susan Lammers

21.
1.
Design Patterns: Elements of Reusable Object-Oriented Software
Erich Gamma Richard Helm Ralph Johnson John Vlissides

2.
Structure And Interpretation Of Computer Programs, Second Edition : By Harold Abelson and Gerald Jay Sussman

3.
Refactoring to Patterns
Joshua Kerievsky

4.
Types and Programming Languages
Benjamin C. Pierce

5.
Code: The Hidden Language of Computer Hardware and Software
Charles Petzold

6.
Object-Oriented Analysis and Design with Applications (2nd Edition)
Grady Booch

7.
Code Complete: A Practical Handbook of Software Construction, Second Edition
Steve McConnell

8.
The Design of the UNIX Operating System [Prentice-Hall Software Series]
Maurice J. Bach

9.
The Pragmatic Programmer: From Journeyman to Master
Andrew Hunt David Thomas

10.
Practical API Design: Confessions of a Java Framework Architect
Jaroslav Tulach

11.
The Practice of Programming (Addison-Wesley Professional Computing Series)
Brian W. Kernighan Rob Pike

12.
Programming Pearls (2nd Edition)
Jon Bentley

13.
Writing Secure Code, Second Edition
Michael Howard David LeBlanc

14.
The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition)
Frederick P. Brooks Jr.

15.
Patterns of Enterprise Application Architecture
Martin Fowler

16.
Introduction to Functional Programming (Prentice Hall International Series in Computing Science)
Richard Bird

17.
The Art of Computer Programming
Donald E. Knuth

18.
Effective Java (2nd Edition)
Joshua Bloch

19.
Thinking in Java (4th Edition)
Bruce Eckel

20.
Programmers at Work: Interviews With 19 Programmers Who Shaped the Computer Industry (Tempus)
Susan Lammers

21.
Coders at Work: Reflections on the Craft of Programming Peter Seibel
Well, I compiled all the aforementioned from Amazon. I will keep appending to list, if I remember :)
Happy Learning! :)
Labels:
architecture,
Best books,
books,
code,
design,
developers,
must-read,
practices,
programmers,
software
Monday, 11 February 2013
Hadoop Hangover: How-to launch a hadoop cluster CDH4 [MRv1 / YARN + Ganglia] using Apache Whirr
This post is about how-to launch a CDH4 MRv1 or CDH4 Yarn cluster on EC2 instances. It's said that you can launch a cluster with the help of Whirr and in a matter of 5 minutes! This is very true if and only if everything works out well! ;)
Hopefully, this article helps you in that regard.
So, let's row the boat...
- Download the stable version of Apache Whirr ie. whirr-0.8.1.tar.gz from the following link whirr-0.8.1.tar.gz
- Extract from the tarball and generate the key
- Generate the key
- Make a properties file to launch the cluster with that configuration.
- Now let me tell you how to avoid getting headaches!
- cluster name: Keep your cluster name simple. Avoid testCluster, testCluster1 etc. ie. No Caps, numerics..
- Decide on the number of datanodes you want judiciously.
- Your launch may not be successful, if java is not installed. Make sure the image has Java. However, this properties file takes care of that.
- It will be good to go ahead with MRv1 for now and later switch to MRv2, when we get a production stable release.
- This is the minimal set of configurations for launching a Hadoop cluster. But, you can do a lot performance tuning upon this.
- I had launched this cluster from an ec2 instance, Initially i faced errors, regarding user. Setting the configuration below, solved the problem.
- Set proper permissions for ~/.ssh and whirr-0.8.1 folder before launching.
- Well, we are ready to launch the cluster. Name the properties file as "whirr_cdh.properties".
In the console you can see, links to Namenode and JobTracker Web UI. It also prints how to ssh to the instances in the end.
Happy Learning! :)
- Now, you should be having the files generated. You will be able to see these files: instances, hadoop-proxy.sh and hadoop-site.xml
- Starting the proxy
- Open another terminal, and type
- You should be able to access the HDFS.
- You can alternatively download hadoop tarball and launch with
- Okay! So I know that you will not be satisfied unless you a web UI
- If you want to launch MRv2, use this.
Happy Learning! :)
Labels:
Apache Hadoop,
Apache Whirr,
aws,
CDH4,
configuration,
EC2,
Firefox,
FoxyProxy,
Ganglia,
Hadoop cluster,
Install Hadoop,
MRv1,
proxy,
Web UI,
Yarn
Tuesday, 18 December 2012
FUSE on Amazon S3
FUSE: File System In User Space, hosted on sourceforge, a well known open source project http://fuse.sourceforge.net/
You either put the files in S3 bucket directly or in the mount point, both will always be in the same hierarchy and in Sync. The best thing is that any arbitrary program can just point to this mount point and perform simple/ normal commands, rather than file system specific commands.
Here is a small documentation about how we can achieve this.
1. Check out the code from google code.
2. Switch to the working directory
3. Now same old ritual of configure , make and install.
To run the subsequent command you need autoconf. So make sure you have it by running the following command.
Getting back...
4. Done with the Installation process.
Cross-check:
5. Add the following line to your ~/.bashrc file and source it.
6. Install s3cmd. Many of you must be using this tool to interact with s3.
Configuring FUSE
1. First set use_allow_other for others to use. Uncomment in fuse.conf
2. Set the AcessKey:SecretKey in the format in passwd-s3fs file
5. With s3fs, as a root user.
6. I created a directory structure of
/mnt/s3Sync/
-> 2012/12/18$ more test.txt
This is a check file to sync with the s3dir-sync.
Blah..!
The same is synced in the bucket "s3dir-sync"
Cross-Check:
Happy Learning! :)
You either put the files in S3 bucket directly or in the mount point, both will always be in the same hierarchy and in Sync. The best thing is that any arbitrary program can just point to this mount point and perform simple/ normal commands, rather than file system specific commands.
Here is a small documentation about how we can achieve this.
1. Check out the code from google code.
$ svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs
2. Switch to the working directory
$ cd s3fs
$ ls
AUTHORS autogen.sh ChangeLog configure.ac COPYING doc INSTALL Makefile.am NEWS README src test
3. Now same old ritual of configure , make and install.
To run the subsequent command you need autoconf. So make sure you have it by running the following command.
$ sudo apt-get install autoconf
$ autoreconf --install
It is silently notifying you that you lack the libraries. Time to get them installed...
$ sudo apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support
Getting back...
$ ./configure --prefix=/usr
$ make
$ sudo make install
4. Done with the Installation process.
Cross-check:
$ /usr/bin/s3fs
s3fs: missing BUCKET argumentUsage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...
5. Add the following line to your ~/.bashrc file and source it.
export s3fs=/usr/bin/s3fs
$source ~/.bashrc$ s3fs s3fs: missing BUCKET argumentUsage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...
6. Install s3cmd. Many of you must be using this tool to interact with s3.
$ sudo apt-get install s3cmd$ s3cmd --configureThis will configure with the S3 account using Access and Secret Key.
Configuring FUSE
1. First set use_allow_other for others to use. Uncomment in fuse.conf
$ vi /etc/fuse.conf
2. Set the AcessKey:SecretKey in the format in passwd-s3fs file
$ sudo vi /etc/passwd-s3fs
$ sudo chmod 640 /etc/passwd-s3fs3. Created a bucket called "s3dir-sync" for this experiment.
$ s3cmd ls2012-12-18 09:23 s3://s3dir-sync4. Creating a mount point where you want to dump/place the files and keep them in sync with the S3 bucket. Create as root user.
$ sudo mkdir -p /mnt/s3Sync$ sudo chmod 777 /mnt/s3Sync
5. With s3fs, as a root user.
$ sudo s3fs s3dir-sync -o default_acl=public-read -o allow_other /mnt/s3Sync/Cross-check:
$ mount -ls3fs on /mnt/s3Sync type fuse.s3fs (rw,nosuid,nodev,allow_other)If you try mounting again, you will get the following Warning
mount: according to mtab, s3fs is already mounted on /mnt/s3Sync
6. I created a directory structure of
/mnt/s3Sync/
-> 2012/12/18$ more test.txt
This is a check file to sync with the s3dir-sync.
Blah..!
The same is synced in the bucket "s3dir-sync"
Cross-Check:
$ s3cmd ls s3://s3dir-sync
DIR s3://s3dir-sync/2012/
2012-12-18 09:57 0 s3://s3dir-sync/2012
Labels:
Amazon Web Services,
aws,
file system,
FUSE,
open source,
precise pangolin,
s3,
s3cmd,
s3fs,
source forge,
sync,
ubuntu
Monday, 10 December 2012
Get your wireless working on DELL Inspiron 5220
I brought a new DELL Inspiron 5220. It's amazing!
Configuration :
- 3rd Generation i5 Processor
- 4GB RAM
- 1TB Hard Disk
- 15" Screen
- 1GB Graphics
It ships with Windows 8! ;)
However, made a dual boot upon it. Although BIOS looked different this time!!
Well,
I'm working on Ubuntu :)
Release : 11.10 (Oneiric)
Kernel Linux : 3.0.0-28-generic
GNOME 3.2.1
But Wi-Fi was not getting detected. This was not unusual, as I had set this up in earlier Dell models.
Well, the remedy is easy.
Step 1: Make sure that you can witness the Device Card and the ID. Especially, the Network !!
Type in the following command.
$ lspci -nnk | grep Network
08:00.0 Network controller [0280]: Intel Corporation Device [8086:0887] (rev c4)
Step 2: Figure out which kernel version. This is because the driver which we will be installing works on 2.6.37 or higher
$ uname -a
Linux Swathi 3.0.0-28-generic #45-Ubuntu SMP Wed Nov 14 21:57:26 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Step 3: Install the network manager
$ sudo apt-get install network-manager*
Step 4: Install few packages
$ sudo apt-get install build-essential linux-headersStep 5: Check the output of
$ dmesg
If it outputs the failure of firmware file, then its time to download the .ucode and place it in /lib/firmware.
Reboot.
It should be working. If not, try Step 6.
Step 6: Download the compat wireless tarball from this location
http://linuxwireless.org/download/compat-wireless-2.6/compat-wireless-2012-05-10-p.tar.bz2
Extract from the tarball
$ tar -xvf <path_to_compat_wireless_bz2>
$ cd <extracted_path_compat_wireless>
Installing the packages
$ make
$ sudo make install
After this command it will show on the console the command to disable bluetooth, ethernet and Wi-Fi. Type in the 3 commands.
Place this module into the kernel.
$ sudo modprobe alxThe Ethernet LAN should be detected.
Add the driver module into this file : /etc/modules
Append the following lines. Don't touch the rest. This will enable this module while restarting the system as it loads the module.
$ sudo vi /etc/modules
#E2200 support
alx
Reboot your machine.
You must witness "Wi-Fi Networks Available!" notification on you desktop :)
Happy Learning! :)
Labels:
8086:0887,
compat,
Dell Inspiron 5220,
linux,
ubuntu,
ubuntu 11.10,
wifi,
wireless
Subscribe to:
Posts (Atom)




