ALL about AWS EBS Provisioned IOPS - feature and resources

AWS has recently announced EBS Provisioned IOPS feature, a new Elastic Block Store volume type for running high performance databases in the cloud. Provisioned IOPS are designed to deliver predictable, high performance for I/O intensive workloads, such as database applications, that rely on consistent and fast response times. With Provisioned IOPS, you can flexibly specify both volume size and volume performance, and Amazon EBS will consistently deliver the desired performance over the lifetime of the volume.

Simple comparison between the standard volumes and provisioned IOPS volumes:

Amazon EBS Standard volumes

• Offer cost effective storage for applications with moderate or bursty I/O requirements.
• Deliver approximately 100 IOPS on average with a best effort ability to burst to hundreds of IOPS.
• Are also well suited for use as boot volumes, where the burst capability provides fast instance start-up times.
• $0.10 per GB-month of provisioned storage
• $0.10 per 1 million I/O requests

Amazon EBS Provisioned IOPS volumes

• Provisioned IOPS volumes are designed to deliver predictable, high performance for I/O intensive workloads such as databases.
• Amazon EBS currently supports up to 1000 IOPS per Provisioned IOPS volume, with higher limits coming soon.
• Provisioned IOPS volumes are designed to deliver within 10% of the provisioned IOPS performance 99.9% of the time.
• $0.125 per GB-month of provisioned storage
• $0.10 per provisioned IOPS-month

AWS has compiled some interesting resources for the users:

Our recent release of the EBS Provisioned IOPS feature (blog post, explanatory video, EBS home page) has met with a very warm reception. Developers all over the world are already making use of this important and powerful new EC2 feature. I would like to make you aware of some other new resources and blog posts to help you get the most from your EBS volumes.

• The EC2 FAQ includes answers to a number of important performance and architecture questions about Provisioned IOPS.
• The EC2 API tools have been updated and now support the creation of Provisioned IOPS volumes. The newest version of the ec2-create-volume tool supports the --type and --iops options. For example, the following command will create a 500 GB volume with 1000 Provisioned IOPS:
  $ ec2-create-volume --size 500 --availability-zone us-east-1b --type io1 --iops 1000
• Eric Hammond has written a detailed migraton guide to show you how to convert a running EC2 instance to an EBS-Optimized EC2 instance with Provisioned IOPS volumes. It is a very handy post, and it also shows off the power of programmatic infrastructure.
• I have been asked about the applicability of existing EC2 Reserved Instances to the new EBS-Optimized instances. Yes, they apply, and you pay only the additional hourly charge. Read our new FAQ entry to learn more.
• I have also been asked about the availability of EBS-Optimized instances for more instance types. We intend to support other instance types based on demand. Please feel free to let us know what you need by posting a comment on this blog or in the EC2 forum.
• The folks at CloudVertical have written a guide to understanding new AWS I/O options and costs.
• The team at Stratalux wrote a very informative blog post, Putting Amazon's Provisoned IOPS to the Test. Their conclusion:
  "Based upon our tests PIOPS definitely provides much needed and much sought after performance improvements over standard EBS volumes. I’m glad to see that Amazon has heeded the calls of its customers and developed a persistent storage solution optimized for database workloads."

We have also put together a new guide to benchmarking provisioned IOPS volumes. The guide shows you how to set up and run high-quality, repeatable benchmarks on Linux and Windows using the fio, Oracle Orion, and SQLIO tools. The guide will walk you through the following steps:

• Launching an EC2 instance.
• Creating Provisioned IOPS EBS volumes.
• Attaching the volumes to the instance.
• Creating a RAID from the volumes.
• Installing the appropriate benchmark tool.
• Benchmarking the I/O performance of your volumes.
• Deleting the volumnes and terminate the instance.

Since I like to try things for myself, I created six 100 GB volumes, each provisioned for 1000 IOPS:

Then I booted up an EBS-Optimized EC2 instance, built a RAID, and ran fio. Here's what I saw in the AWS Management Console's CloudWatch charts after the run. Each volume was delivering 1000 IOPS, as provisioned:

Here's an excerpt from the results:

fio_test_file: (groupid=0, jobs=32): err= 0: pid=23549: Mon Aug 6 14:01:14 2012

read : io=123240MB, bw=94814KB/s, iops=5925 , runt=1331000msec

clat (usec): min=356 , max=291546 , avg=5391.52, stdev=8448.68

lat (usec): min=357 , max=291547 , avg=5392.91, stdev=8448.68

clat percentiles (usec):

| 1.00th=[ 418], 5.00th=[ 450], 10.00th=[ 478], 20.00th=[ 548],

| 30.00th=[ 596], 40.00th=[ 668], 50.00th=[ 892], 60.00th=[ 1160],

| 70.00th=[ 3152], 80.00th=[10432], 90.00th=[20864], 95.00th=[26752],

| 99.00th=[29824], 99.50th=[30336], 99.90th=[31360], 99.95th=[31872],

| 99.99th=[37120]

Read the benchmarking guide to learn more about running the benchmarks and interpreting the results.

SOURCE for resources : http://aws.typepad.com/aws/2012/08/ebs-provisioned-iops-some-interesting-resources.html

Announcing AWS Elastic Beanstalk support for Python, and seamless database integration

It’s a good day to be a Python developer: AWS Elastic Beanstalk now supports Python applications! If you’re not familiar with Elastic Beanstalk, it’s the easiest way to deploy and manage scalable PHP, Java, .NET, and now Python applications on AWS. You simply upload your application, and Elastic Beanstalk automatically handles all of the details associated with deployment including provisioning of Amazon EC2 instances, load balancing, auto scaling, and application health monitoring.

Elastic Beanstalk supports Python applications that run on the familiar Apache HTTP server and WSGI. In other words, you can run any Python applications, including your Django applications, or your Flask applications. Elastic Beanstalk supports a rich set of tools to help you develop faster. You can use eb and Git to quickly develop and deploy from the command line. You can also use the AWS Management Console to manage your application and configuration.

The Python release brings with it many platform improvements to help you get your application up and running more quickly and securely. Here are a few of the highlights below:

Integration with Amazon RDS

Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud, making it a great fit for scalable web applications running on Elastic Beanstalk.

If your application requires a relational database, Elastic Beanstalk can create an Amazon RDS database instance to use with your application. The RDS database instance is automatically configured to communicate with the Amazon EC2 instances running your application.
 

A console screenshot showing RDS configuration options when launching a newAWS Elastic Beanstalk environment.

Once the RDS database instance is provisioned, you can retrieve information about the database from your application using environment variables:

import os if 'RDS_HOSTNAME' in os.environ: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': os.environ['RDS_DB_NAME'], 'USER': os.environ['RDS_USER'], 'PASSWORD': os.environ['RDS_PASSWORD'], 'HOST': os.environ['RDS_HOSTNAME'], 'PORT': os.environ['RDS_PORT'], } }

To learn more about using Amazon RDS with Elastic Beanstalk, visit “Using Amazon RDS with Python” in the Developer Guide.

Customize your Python Environment

You can customize the Python runtime for Elastic Beanstalk using a set of declarative text files within your application. If your application contains a requirements.txt in its top level directory, Elastic Beanstalk will automatically install the dependencies using pip.

Elastic Beanstalk is also introducing a new configuration mechanism that allows you to install packages from yum, run setup scripts, and set environment variables. You simply create a “.ebextensions” directory inside your application and add a “python.config” file in it. Elastic Beanstalk loads this configuration file and installs the yum packages, runs any scripts, and then sets environment variables. Here is a sample configuration file that syncs the database for a Django application:

commands: syncdb: command: "django-admin.py syncdb --noinput" leader_only: true option_settings: "aws:elasticbeanstalk:application:python:environment": DJANGO_SETTINGS_MODULE: "mysite.settings" "aws:elasticbeanstalk:container:python": WSGIPath: "mysite/wsgi.py"

Snapshot your logs

To help debug problems, you can easily take a snapshot of your logs from the AWS Management console. Elastic Beanstalk aggregates the top 100 lines from many different logs, including the Apache error log, to help you squash those bugs.
 

The snapshot is saved to S3 and is automatically deleted after 15 minutes. Elastic Beanstalk can also automatically rotate the log files to Amazon S3 on an hourly basis so you can analyze traffic patterns and identify issues. To learn more, visit “Working with Logs” in the Developer Guide.

Support for Django and Flask

Using the customization mechanism above, you can easily deploy and run your Django and Flask applications on Elastic Beanstalk.

• To get started with Django, visit “Deploy a Django Application to AWS Elastic Beanstalk” in the Developer Guide.
• To get started with Flask, visit “Deploy a Flask Application to AWS Elastic Beanstalk” in the Developer Guide.

For more information about using Python and Elastic Beanstalk, visit the Developer Guide.

SOURCE

yoyoclouds: The Cloud Revolution

The Cloud Revolution Cloud computing spending will account for 25% of annual IT expenditure growth by 2012 and nearly a third of the growth the following year. 

 “The battle for Cloud dominance is heating up, with the release of Office 365, it will be very interesting to see where the next big play comes from.”

Read more here.

Big Data Infographic: The 2012 London Summer Games

Big Data Infographic: The 2012 London Summer Games

Recent years have seen a lot of development in the cloud computing sphere. Big data is believed by many to be here to stay, and a lot of real investment is touted to happen in this particular area. Such a trend is quite exciting, as new, better and more powerful infrastructure will be needed to support all this. So,  a lot of further development is on the way to accommodate these computing perimeters.

Does agility matter?

Your company might have been in the business for many years, or it may be a newcomer to the field – JIT (Just in Time) deployment of services is very important for both types of organizations. This is also critical to the success of any of the types of company described above. While small business owners may sometimes think that there is a cost control through traditional IT, they really need to consider the agility that the cloud brings to their businesses. Cloud computing, when moved and executed properly, can help companies tap market opportunities to the best possible effect, due to the extended flexibility and agility it has to offer. The recent acquisition of Cloud dot com by Citrix clearly proves that there has been an increase in interest in cloud computing technology. It is also touted in the networking space that many emerging SDN players are bound to be acquisition targets for major companies keeping an eye on the developments in the sphere.

Read more here.

SOURCE