Scaling from a local server to cloud using Amazon AWS
Migrate from existing local network server set up to a cloud appliance and scale up using Amazon AWS platform/ecosystem/tools to meet customer demand s.
Am writing this document with few assumptions, the reader is familiar with Amazon AWS, Cloud in general. It will be nice to know some of the following key words like AMI, EC2, EBS, Volume, Ephemeral Store, Monitoring, Basic Networking, SSH, VPN & VPC. We will be using most of them below to explain our solution. Its not a must to know everything but a general idea of what it is will be really helpful. As always we can refer the repository of documentation at aws.amazon.com for some the above mentioned technology & keywords. If confused am at your service to clarify.
base OS
For any product to run successfully we need a base OS, a solid base OS image that it can safely run and get critical OS & security updates over a period of time (at least 2-3 years). My personal recommendation would be a Ubuntu 10.04 LTS which has 5 year support and would suffice our need to start with. Of course we could choose Gentoo/CentOS/RHeL or other alternate distribution based on specific needs that you have. Since we are looking at running production instance in AWS, it make sense to have our root partition (aka ‘/’ ) in Elastic Block Store (aka EBS). EBS allows us to have store data ranging from 1GB to 1TB and is re sizable (yes you hears it right). EBS allows us to create volume of various size and mount it to instance to be used a partition (say web root of our Apache for example). EBS has a incredible feature to take snapshot where in the snapshot taken are stored in S3 a highly reliable data store with 99.999% up time. In case of a crash (which rarely happens), recovering would be few click away (or few lines of script away). Initially we can start with taking a snapshot say every midnight and as we push more and more changes we can increase the frequency of snapshot. Remember when we take multiple snapshot of same volume it only stores incremental changes and thus saves substantial storage cost.
2 ways of setting up LAMP
- LAMP
- The traditional way of setting up a LAMP is installing the require packages from the OS repository which would involve setting up Apache, MySQL and PHP. (reference: https://help.ubuntu.com/community/ApacheMySQLPHP). Here the entire stack remains in one instance. This mean you will have to do the database tweaking and administration yourself apart from app development.
- LA(M)P -> LA(RDS)P
Things to take care while setting up Apache/PHP
As said earlier EBS allows you to scale from GB -> TB and is re sizable. I recommend to create a EBS volume (start with XGB based on what is currently used) and attach this instance. Upon attaching a EBS volume to a instance we can access it as a separate disk drive. Format with ext{3,4} and mount it. Use the newly mounted partition as your Apache Root folder. This approach has multiple advantages as well.
- Separation of OS data vs application data
- In case of crash, you can boot another instance and connect the volume to new instance, yes data in EBS is persistent
- We can schedule more frequent snapshot for data volume.
- The snapshots are stored in S3 (a highly reliable data store ) and is replicated across multiple availability zone for higher redundancy, a huge ++++ for us.
- Creating a experimental setup with mock data is as simple as taking a snapshot of your vol and cloning it.
A reason to smile, a little bit of cost saving here
By default EBS volume snapshot are stored on to S3 and is replicated to multiple availability zone. But in some case we might have loads of data which are not critical but is generated at run time or during the session. Here we could choose to use Reduced Redundancy Storage (aka RRS). So go ahead create a EBS volume, set the attribute to make it RRS, attach to your instance and let all your transient data reside in this new low cost volume/partition.
how do we know if we should increase the instance size (RAM, CPU, IO, Network, etc..)
As we have started our implementation with basic instance type which matches our local hardware, we need to know if we are required to increase the CPU,RAM, etc… This brings in CloudWatch which allows us to monitor the instance very closly and take calculated decisions. Based on the monitoring details we can choose more memory or more cpu (more about instance type here).
Update from small to medium/large to give you users a better experience. Else downgrade from large to medium/small to save some bucks for your company.
What next ?? customer is king, let’s give him the best experience.
Now that we have taken care of our basic need with respect to base OS, database and storage. Now let’s looks at some other aspect w.r.t reducing latency of the content that we serve users. Internet is something which is consumed globally and need to serve the content (be it images, video, etc.) to every user around the world becomes the next challenges. Amazon AWS provide CloudFront, a web service for global content delivery with focus on reducing the latency while serving any user any where in the world. Let your customer reside anywhere in the world, CloudFront is there to lead from front.
faster, faster & faster …. but f-a-t-a-l error and system crashed !!!
In the above section we saw that we have handled the Content Delivery part of it, we started monitoring and know if load is high or not. Being a start-up every user who is visiting us is a potential customer and he leaving with a bad impression is the last thing to happen. What is you server faulted and is down at the middle of night or while you were travelling ? Wont it be nice to have another instance serving your customer and make him happy ?
Don’t worry we have Elastic Load Balancer. The load balancing can kick in based on multiple parameter which are configurable. For example in our case we want to make sure if a instance is bad then route all the data to new instance. Elastic Load Balance uses Cloud Watch behind the scene to get the stats of instance and act upon it based on what we have configured.
ok all well, now i have million of user at middle of night !!!
Cool, am happy to hear that more people are using your service, but sad to know that you need to awake to check when to allocate more resource. What if i give a option to do this dynamically ?? am sure your family will be really to have you home at mid night. Here we have Auto Scaling. Again choice is left to you to decide when to scale up and when to scale down based on parameters (like time of day, load, etc)
all is well, but how to achieve seamless working from my office and Amazon AWS
Since your office is gonna be connected to Amazon AWS network for managing the instance, copying critical data, testing a new piece of code, etc… you don’t want every employee to connect separately to EC2 machines. Also while transmitting critical data you don’t want a network hacker (or a competitor) stealing your critical data. To save you from all these head aches of seamless and safer connection, AWS has Virtual Private Cloud (aka VPC). This will make sure your IT infrastructure and Cloud is connected via an encrypted VPN connection. Once your VPC is enabled and configured your cloud is ready to serve you locally. Remember to tweak you security group to enable/disable network connections.
What have we done so far, a quick glance:
Our goal is to get a robust, reliable, secure, scalable, on demand computing resources for the next .com that you are building on top of LAMP stack. With ec2 running Linux is a click away. By choosing appropriate distribution (one which has Life Time Support like Ubuntu, CentOS,RHeL, etc.) you can choose to stay focused on you application development. By storing your ‘/’, ‘/data’ on EBS make sure you have safe backup available in multiple availability zone (powered by S3). By storing ‘/non_critical_data’ in EBS (with RRS) we make sure we save money (reducing redundancy). With monitoring (Cloud Watch) enabled you know what is you system doing and when w.r.t CPU, IO, Network, Memory, etc .. Once you have become popular around the globe, the Cloud Front (the content delivery network) make sure the latency is reduced to minimum and as a result customer experiencing best possible experience using your product. Just to make sure you sleep well and stay focused, the elastic load balancing system keeps a tab on faulty system and redirect traffic to new instance when required. Same is the case when you have more users hitting your system, the Auto Scaling does the magic behind the scene. Security group make sure that you are safe in the Internet by allowing only needed protocol/ports to public. Finally our VPC make sure that data is secure while your employees connect from office to all aws resource (ec2 instance, etc..) which make the machine in the Amazon AWS network transparent.
happy aws’ing
for any help am just a email away
