no parking (go biking)

I see this board every where in Bangalore these days, and is one of the reason i love biking (aka cycling) instead of a car. There is also a bigger reason to worry about – auto’s going at the speed of rajanikanth, speeding call center cabs, heavy duty bus which occupies entire road …. all rushing thru city limits, making living here less joyful.

Am sure as a child we would have done loads of biking. Rekindle your inner child and start again. Its actually good fun cycling, first we dont have to worry about “no parking” boards, second cops wont stop you, third you burn unwated calories in your body which results in you enjoying every meal that you eat , fourth is i can still travel in city volvos to any where and do the last mile in bike (aka cycle), fifth is we are definetly more green (burn ur fat not fuel ), sixth you make tons of friend on you way, seventh is there is still a chance of seeing sky,cloud,stars,sun,moon,people, etc.

For sure biking (aka cycling)keep you fit, gives you some time to listen to music or some podcast and make your life bit more enjoyable.

(Veloscope a initiative started by couple of us to promote biking & photography in and around Bangalore, India. Visit our facebook page)

Published with Blogger-droid v1.6.5

Access AWS from command line – quick reference

One of cool feature of Amazon AWS is ability to talk to AWS ecosystem thru command line. Why do i think that this is cool, because you can script it. Let’s take a look at setting it up.This is not a detailed guide, its just a quick reference.

We are gonna use the command line from a shell. Our goal right now is to get ‘values’ for the key listed below. Line starting with ‘#’ are to be ignored and meant for explanation.

#JAVA_HOME -> where you java installation, the below path works for Mac. For linux and Windows it would be different
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
# ec2-api-tools, this need to be downloaded from http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip and place it where ever you want.
export EC2_HOME=/Applications/ec2/ec2-api-tools 
# both the .pem has to be generated and download, and if you mis you will have to regenerate. I normally keep this is my dropbox.
export EC2_PRIVATE_KEY=~/.ec2/pk-xxxxx.pem
export EC2_CERT=~/.ec2/cert-xxxxx.pem  
# available from the http://aws.amazon.com/account/
export EC2_ACCNO=1111-2222-3333
export ACCESS_KEY=BABBIO4XTFPTE6UYD3BQ
export SECRET_KEY=jcbsdsadcusjkh867HJk+897bnHygTQoEHj
# for easy access
export PATH=$PATH:$EC2_HOME/bin

Login to your account and click security credentials:

Get your Access and Secret Keys: 

Generate your certificate and download pk***.pem and cert-***.pem

Get you ec2 account number:

Reference:
http://aws.amazon.com/documentation/

Detailed reference of commnad line
http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/

Scaling from a local server to cloud using Amazon AWS

Goal
Migrate from existing local network server set up to a cloud appliance and scale up using Amazon AWS platform/ecosystem/tools to meet customer demand s.

Intro
Am writing this document with few assumptions, the reader is familiar with Amazon AWS, Cloud in general. It will be nice to know some of the following key words like AMI, EC2, EBS, Volume, Ephemeral Store, Monitoring, Basic Networking, SSH, VPN & VPC. We will be using most of them below to explain our solution.  Its not a must to know everything but a general idea of what it is will be really helpful. As always we can refer the repository of documentation at aws.amazon.com for some the above mentioned technology & keywords. If confused am at your service to clarify. 

 
base OS
For any product to run successfully we need a base OS, a solid base OS image that it can safely run and get critical OS & security updates over a period of time (at least 2-3 years).  My personal recommendation would be a Ubuntu 10.04 LTS which has 5 year support and would suffice our need to start with. Of course we could choose Gentoo/CentOS/RHeL or other alternate distribution based on specific needs that you have.  Since we are looking at running production instance in AWS, it make sense to have our root partition (aka ‘/’ ) in Elastic Block Store (aka EBS).  EBS allows us to have store data ranging from 1GB to 1TB and is re sizable (yes you hears it right). EBS allows us to create volume of various size and mount it to instance to be used a partition (say web root of our Apache for example). EBS has a incredible feature to take snapshot where in the snapshot taken are stored in S3 a highly reliable data store with 99.999% up time. In case of a crash (which rarely happens), recovering would be few click away (or few lines of script away). Initially we can start with taking a snapshot say every midnight and as we push more and more changes we can increase the frequency of snapshot. Remember when we take multiple snapshot of same volume it only stores incremental changes and thus saves substantial storage cost.

2 ways of setting up LAMP

1. LAMP

• The traditional way of setting up a LAMP is installing the require packages from the OS repository which would involve setting up Apache, MySQL and PHP. (reference: https://help.ubuntu.com/community/ApacheMySQLPHP). Here the entire stack remains in one instance. This mean you will have to do the database tweaking and administration yourself apart from app development.

2. LA(M)P -> LA(RDS)P

If you see the above heading, i have replace ‘M’ with ‘RDS’. Amazon AWS provides relational database capability as well, its called Relational Database Services (aka RDS). The advantage of using this is entire MySQL database administration overhead of taken away from you and allows you to stay focused what you really want to be doing, developing and improving your application instead of database administration. I highly recommend going in for RDS which takes care of our scaling issue as we attract more customers (and data)

Things to take care while setting up Apache/PHP
As said earlier EBS allows you to scale from GB -> TB and is re sizable. I recommend to create a EBS volume  (start with XGB based on what is currently used) and attach this instance. Upon attaching a EBS volume to a instance we can access it as a separate disk drive. Format with ext{3,4} and mount it. Use the newly mounted partition as your Apache Root folder.  This approach has multiple advantages as well.

• Separation of OS data vs application data
• In case of crash, you can boot another instance and connect the volume to new instance, yes data in EBS is persistent
• We can schedule more frequent snapshot for data volume.
• The snapshots are stored in S3 (a highly reliable data store ) and is replicated across multiple availability zone for higher redundancy, a huge ++++ for us.
• Creating a experimental setup with mock data is as simple as taking a snapshot of your vol and cloning it.

A reason to smile, a little bit of cost saving here
By default EBS volume snapshot are stored on to S3 and is replicated to multiple availability zone. But in some case we might have loads of data which are not critical but is generated at run time or during the session. Here we could choose to use Reduced Redundancy Storage (aka RRS). So go ahead create a EBS volume, set the attribute to make it RRS, attach to your instance and let all your transient data reside in this new low cost volume/partition.

how do we know if we should increase the instance size (RAM, CPU, IO, Network, etc..)
As we have started our implementation with basic instance type which matches our local hardware, we need to know if we are required to increase the CPU,RAM, etc… This brings in CloudWatch which allows us to monitor the instance very closly and take calculated decisions. Based on the monitoring details we can choose more memory or more cpu (more about instance type here).
Update from small to medium/large to give you users a better experience. Else downgrade  from large to medium/small to save some bucks for your company.

What next ?? customer is king, let’s give him the best experience.
Now that we have taken care of our basic need with respect to base OS, database and storage. Now let’s looks at some other aspect w.r.t reducing latency of the content that we serve users. Internet is something which is consumed globally and need to serve the content (be it images, video, etc.) to every user around the world becomes the next challenges. Amazon AWS provide CloudFront, a web service for global content delivery with focus on reducing the latency while serving any user any where in the world. Let your customer reside anywhere in the world, CloudFront is there to lead from front.

faster, faster & faster …. but f-a-t-a-l error and system crashed !!!
In the above section we saw that we have handled the Content Delivery part of it, we started monitoring and know if load is high or not. Being a start-up every user who is visiting us is a potential customer and he leaving with a bad impression is the last thing to happen. What is you server faulted and is down at the middle of night or while you were travelling ? Wont it be nice to have another instance serving your customer and make him happy ?

Don’t worry we have Elastic Load Balancer. The load balancing can kick in based on multiple parameter which are configurable. For example in our case we want to make sure if a instance is bad then route all the data to new instance. Elastic Load Balance uses Cloud Watch behind the scene to get the stats of instance and act upon it based on what we have configured.  

ok all well, now i have million of user at middle of night !!!
Cool, am happy to hear that more people are using your service, but sad to know that you need to awake to check when to allocate more resource. What if i give a option to do this dynamically ?? am sure your family will be really to have you home at mid night. Here we have Auto Scaling. Again choice is left to you to decide when to scale up and when to scale down based on parameters (like time of day, load, etc)

all is well, but how to achieve seamless working from my office and Amazon AWS
Since your office is gonna be connected to Amazon AWS network for managing the instance, copying critical data, testing a new piece of code, etc… you don’t want every employee to connect separately to EC2 machines. Also while transmitting critical data you don’t want a network hacker (or a competitor) stealing your critical data. To save you from all these head aches of seamless and safer connection, AWS has Virtual Private Cloud (aka VPC). This will make sure your IT infrastructure and Cloud is connected via an encrypted VPN connection. Once your VPC is enabled and configured your cloud is ready to serve you locally. Remember to tweak you security group to enable/disable network connections.

What have we done so far, a quick glance:
Our goal is to get a robust, reliable, secure, scalable, on demand computing resources for the next .com that you are building on top of LAMP stack. With ec2 running Linux is a click away. By choosing appropriate distribution (one which has Life Time Support like Ubuntu, CentOS,RHeL, etc.) you can choose to stay focused on you application development. By storing your ‘/’, ‘/data’ on EBS make sure you have safe backup available in multiple availability zone (powered by S3). By storing ‘/non_critical_data’ in EBS (with RRS) we make sure we save money (reducing redundancy). With monitoring (Cloud Watch) enabled you know what is you system doing and when w.r.t CPU, IO, Network, Memory, etc .. Once you have become popular around the globe, the Cloud Front (the content delivery network) make sure the latency is reduced to minimum and as a result customer experiencing best possible experience using your product. Just to make sure you sleep well and stay focused, the elastic load balancing system keeps a tab on faulty system and redirect traffic to new instance when required. Same is the case when you have more users hitting your system, the Auto Scaling does the magic behind the scene. Security group make sure that you are safe in the Internet by allowing only needed protocol/ports to public. Finally our VPC make sure that data is secure while your employees connect from office to all aws resource (ec2 instance, etc..) which make the machine in the Amazon AWS network transparent.

happy aws’ing

for any help am just a email away

Launch Amazon AWS, EC2 instance in singapore in less than 10 clicks

With launch of new region in Singapore, people in APAC get benefited by launching their instance close to their country (earlier we had only EU and US).

In this blog post i will cover steps to launch a instance in less than 10 steps.

Step1:
Visit https://console.aws.amazon.com/ec2/home and login

Step 2:
Choose your region, in our case Asia Pacific

Step 3:
Decide which AMI to boot from. For those who are new to AMI, Visualize it as a OS template. For this example we will use an AMI listed in alestic.com

Step 4:
Create a Security Group (equivalent of firewall ). Let’s just enable SSH at the minimum and if we are planning to expose http then port 80 as well.

Step 5:
Create a key pair (The ssh connections to AMI is key based and hence this step). Store the key pair in your home folder (i created a ~/.ec2 folder to keep all ec2 stuff)

Step 6:
Launch Instance (choose ami, instance type, security group, key pair) and finally fire it.

Step 7:
ssh to the machine and see if its working.

In case if you are interested you can setup the command line API to connect to Amazon AWS (will cover this as a separate post)

happy aws’ing

wisdom gone

After much comtemplation ( and pain ) i decided to get rid of wisdom tooth both the upper and lower. According to doctor its normally not used for chewing and can be safely removed. As always i was curious to know how ancient people managed to get rid of wisdom tooth. According to some of last generation people there used to be people who can pluck any tooth (not certified docs) and give you some ayurvedic medicine to heal and things will be alright in few days (oh yes there are cases where piece of teeth still remained inside ). These days things have definetely improved with x-rays and locals which make sure that we do it right in a less painful way.

I got mine wisdom tooth removed on a wednesday. The x-ray showed that i have a straight root and it should be relatively easy to pull it out but it turned out to be a disaster (had to cut in pieces before taking it out). Also the doc had to cut the tissues around for a safe removal. Had stiched it to make sure that i heal faster.

Day 0 night was crazy, i had to take injection for pain and antibiotic. Both on my left and right thigh. I managed to eat some thing, dont remember what but thanks to mom. Normally i sleep on a room in first floor (my parents in ground floor) and in the middle of night i had to come down for ice cubes (reduces swelling), my legs trembled and phat…i lay flat on ground. Immediately i got my conscience back and i could remember my enemies very well (took that as a sign that nothing serious happened).

Day 1 morning, i had developed severe neck pain and i assumed that it because of fall. Later i realized that it because of locals. Bleeding continued and was a vampire for entire day 1 (bleeding almost stopped end of day 1)

Day 2 could brush my teeth and felt a bit okay than previous days. Could even walk for few km (very slowly tho)

Day 3 am writing this blog

I continued to eat very well (grinded the food before eating) thanks to mom. Just after the medicine i dont go out or do things (feels giddy for almost an hour) and i gargle at least 3/4 times and occationally eat some ice cream if pain aggravates.

Overall it painful, but hoping that my tomorrow is gonna be better.

I know this is nothing to what other go thru in life when there is no medical support or any else to take care of them. When i read how Lance Armstrong recovered from cancer with mere 2% survival chance to winning tour de france this is jus nothing.

Experience is great teacher and feeling greatful to all master/patriots/freedom fighters/soliders who were/is ready to sacrifice there happiness today for a better tomorrow.

Oops too much gyan for today !

Published with Blogger-droid v1.6.5

android looks super cool (especially bash aka terminal)

Its been a while i wanted to post a blog about android. I was hooked onto android from day one. A collegue of mine had a basic LG GW620 and he got himself upgraded (or may be downgraded) to a blackberry(may god bless him ). First impression of Droid 1.5 was not that great, its like just another phone with some interface. My goal was to get froyo up and after a bit of research i found few guys at openetna.com are actually building a free firmware to android 2.2-* for the phone that i have got. Got really excited and installed froyo (still keeps updating to latest RC* release every two weeks). Found some really really cools apps but all it matters to me is a shell, that’s where i belong to

A screenshot of bash in adroid is attached. There are some basic apps that will make ur life simpler while using any droid phone. Like for example evernote for capturing any notes anywhere, a blogger client or google cli (helps u to post a blog from command line), google sync (comes as part of gapps package from google directly), sms back up+ (which backup all received sms and latest version it syncs call logs as well), this means all ur contact + sms + call logs are in safe hands (google god ). The other app which is must is titanium backup, the only app i paid some money (150/- indian rupee or $3) and really worth paying for. It has a cool feature to backup to dropbox (2gb free and if u r smart u can get upto 8gb free).

If you are planning to buy a phone and get it updated like the way i have done with openenta (or any other firmware) be ready with a standby phone and a new sim card, so that ur exisiting sim can be used in the spare phone and the temporary sim can be used in android phone till u set it up completely (i got a temporary uninor 6gb data plan and got everything tested before i started using my actual sim). Also be ready to go thru some hardship of reading, reading and reading some android forums post in case if you are stuck. The best part is android community in general r()cks.

Published with Blogger-droid v1.6.5

Announcing – OpenLinuxBook Project

Its been a little frustrating to see that there is lack of trust worthy documentation for Linux. There are many sites which has tons of documentation but is not very well maintained (and is not the latest) and joining the volunteer committee is not that easy (separate registration, approval from board, etc). Born out of desire to start a documentation project or rather a single source of truth for Linux documentation. A documentation which covers the depth and breadth of Linux with latest and greatest technical details for begineers, intermediates and advances users. That means that we might write something refer from existing documentation projects and soon.

The are few things i want to achieve:

• Use existing infrastructure like Google docs or something similar where people can collaborate easily and work together with a single standard tool chain.  (with zero or no approval cycle)
• There are loads of awesome documentation scattered around, need to figure out a way to scrape it and make it part of the this documentation.
• Should be easy to includes mindmaps and other visual charts (i suppose gdocs has some support for these already)
• A comprehensive documentation which can be source of reference for beginners and experts (build the required gdocs to support feature we want)
• Zero administrative maintenance/support (like for example the domain name, ftp, etc). Use as much as public domain tools (say like facebook for communication, drobbox public folder for sharing, gdocs for sharing/writing documents, etc).  

This is born out of a personal desire, i need to figure out way (irc, forums, posters, facebook page) to contact/interact people and evole the mission and goals for this project.

What to expect next:
I will perhaps put together a campaign which we can announce in multiple Linux forums and get some feedback. If you are reading this and going thru a similar thought feel free to drop in a email to me, perhaps u can be the first to volunteer. 

A sneak peak – Package Management Systems

The first and foremost aspect of a software development process is deciding on what user wants, then get it implemented using some programming language and final stage is to make it available to consumer or customer. In the world of web its either a website or web service api which user access. But behind the scene there are tons of work involved to make sure user see the latest and greatest version (in certain cases latest and most stable version of software;-) ).

1. (An) Idea
2. Implement (the Idea, ofcourse using a programming language of ur choice )
3. Test/Validate/etc (exception if you are a awesome coder)
4. Package (or bundle) It (3 and 4 could be interchanged and depended on organization)
5. Take the packaged software to server or customer box and install it  (push it in package repository)
6. Provision the installed package (take the package from repo and untar it and apply system wide configuration changes)
7. back to step 2, with additional inputs to improve the software (add a beta tag, because this never stops, iterative development)
8. push it to update server
9. continue to use it or delete it

The 3 main activities we are focused here are Packaging or Bundling, Installation, Provisioning the already installed package, Upgrading existing installed software and finally deleting the package either from locally installed machine or from the software repository itself.

Some time the best (may be crude) approach is tar it (or zip it) and untar or unzip w.r.t particular directory (tar -C in Linux) and untar/unzip in destination host. This would be most primitive approach and could work clean in some scenarios. Other simpler approach is using a rsync from you pre-production environment to production environment. All the above mentioned steps has some dis advantages when it comes to large scale systems (more than 100+ servers), w.r.t loosing track what is installed, upgrading OS, selective update, etc…

A good packaging system will help us get the following:

1. Name of the package Installed
2. A brief description of what is suppose to do
3. A version number identifier (some companies obfuscate this )
4. Checksum, to avoid over the network tamppering 
5. List of other dependent packages 
6. After installation, the details of packages are normally kept in a local package database

What does a package management brings it a Operating System ?
Its help maintain things independently and co-evolve different components/libraries of Operating System, otherwise would have become a single monolithic design in nature.

Lifecycle of Package Management System:

In general, it will great to have the following principles as foundation principle for any package management system:

• Install & Deletion of package has to be first functionality
• Verification of a already implemented package has to be smooth, an verification of source of packages also will be good to implement
• Package builder is the king, me it simpler for him. If he is not happy we will not have packages generated
• Let the package builder refer to original source (otherwise can become a hell managing it)
• Make sure that the core package manager system works in multiple architecture, this means the package format can be safely built and used in multiple arch (no relearning of tools for different arch)

Provisioning Systems in Linux

According to wikipedia, provisioning is the process of preparing and equipping a network to allow it to provide (new) services to its users.

If you are one of the developer, who’s life revolves around amazon ec2 servers (or any cloud services) for production, development, testing then you have used provisioning systems already. When we issue, ec2-start-instance , the system working behind this command is a provisioning system. The ami-id is a unique identifier which is mapped to a specific template which has instruction to boot specific version of kernel, network, OS, etc…

In a physical machine after the BIOS POST it will start PXE and gets itself a IP address. Making use of various protocols like tftp it gets an Operating System image (Windows, Linux, Solaris, etc) and boots the kernel (which initializes the rest of the System).

In a virtual environment, the user (or web services api in amazon ec2) request the host operating system to provision the guest machine.

In theory provisioning systems are nothing but templates (or pre configured machine instructions) that a machine will execute while booting up (or during power on). In traditional system where we have a physical machine next to you and if you have a installation CD in hands things were easy, mount the CD and fire install and few minutes later you configuration screen pops up and you create users, set network, login, etc … Now with Cloud Computing (aka On demand) people have figured out way’s to automate things so that we dont have the repeat the same thing again and again while installing or configuration 1000+ of systems (imagine installing 1000+ servers using CD, its a nightmare, less productive and cannot scale up). From the age old days we had kickstart (find /root -name ‘ks.cfg’) from redhat which anaconda installed used to provision multiple systems with same configuration. 

With changes in way OS is deployed the need to have centrally managed easily configurable and which is extend able evolved over a period is the need of hour. (especially when we have On demand system based on XEN and other Virtualization technologies)

The beauty of Open Source/Free Software is that you think about it and by the time you have finished thinking someone else has already executed it

Secret (or sacred ) weapon for doing provision is Cobbler (i dont want to getting into a fight of which is better one, feel free to choose what works for you)

So has cobbler reinvented the wheel, according to me no because it does follow the famous unix principle, “get simpler tools work together” (eg: ls -l | grep name where ls, grep does it job and pipe connects it both). As a whole its a complete Provisioning Application with its api exposed as web services.

The bigger picture (than me writing the all the nuts and bolts)
 

One feature which i personally like is installation over SSH, especially useful for distributions like gentoo where we build things from scratch. Its also provides a neat built-in configuration management tool or can be used alon side with puppet. Some of the power feature Cobbler come with are
Power Features:

• Non program can get started easily using cheetah templates, http://www.cheetahtemplate.org/
• reusing common blocks using snippets, https://fedorahosted.org/cobbler/wiki/KickstartSnippets
• Yum Mirroring
• Advanced Networking Configuration — bonding, vlans
• DHCP Management
• DNS Management
• Power Management
• Configuration Management Hooks (puppet), u can choose built in conf management as well
• BIOS/Firmware upgrades (ex: Dell, HP), personally i have not used it
• Remotely erase machines (DBAN), http://www.dban.org/
• Extensive anaconda monitoring features, really helpful to see the remote logs, https://fedorahosted.org/cobbler/wiki/AnaMon
• Multi-server replication, for redundancy and high availability. Will be handy if you want to expand to other geography. 

Worth taking a look at Cobbler, try it out in your next project and drop in your feedback.

reference
http://mdehaan.fedorapeople.org/presentations/cobbler/index.html#%2812%29
https://fedorahosted.org/cobbler/
https://fedorahosted.org/cobbler/wiki/UserDocs

mailing list
https://fedorahosted.org/mailman/listinfo/cobbler


irc
#cobbler and #cobbler-devel in irc.freenode.net

A good friend happened to invite me to take pictures of Conty Club, India meet. Although I knew these guys were meeting the second time in a row, I didn’t quite expect this level of maturity from a start up club.

To start with the entry formalities were so well orchestrated and so was the parking at the venue. The assembly area was sanitised by the army for Conty Club members. As the Conty’s rolled in, the other formalities were done and CCI was playing fantastic host. Getting members to know each other and their cars and looking inside out of each other cars was the norm of the hour.

Soon after the 10th car arrived, members were requested to proceed to have their refreshments. The refreshments itself was a brilliant arrangement and the army had set up the spread with such opulence, that it seemed more like a small wedding reception. Although there wasn’t a huge breakfast spread at you disposal, the stuff was substantial for a brunch. The military chai still lingers in my taste buds.

The photo sessions went on with members & families including children simply having a good time with 19 Conty’s in one place plus the sun playing hide and seek with a gentle drizzle to add to the aura of the event.

Everything was with clock work precision. A small talk by the hosts about the meet, the philosophy of the club in bringing people together for a purpose, the love of the car and members sharing their experiences with loads of laughter, fun and frolic.

The cars were finally flagged off by a senior army official and split in different directions to avoid disruptions to normal traffic from a big convoy and prevent inconvenience to other road users. Again a very thoughtful gesture by CCI.

The team regroup at Residency road parking with all the 19 cars for a round of chilled Red Bull and was greeted by the Scooter club making a cheerful pass, also by some of CCI’s friends from the Fiat Club. The cars were parked for public viewing and again an instant hit. The teams dispersed on time and the Sunday couldn’t have been spent in a better way.

I thoroughly enjoyed my time at the CCI meet and have volunteered to take pictures of the next meet also as their official photographer. The show they put up was simply awesome. This is by far the best car or auto meet by a group/club I have ever been to.

Some pics here

ContyClub_Meet