Amazon's AWS re:Invent

Amazon held it’s first ever global customer and partner conference called re:Invent for it’s AWS (Amazon Web Services) business.  Six-thousand people from 60 countries were in attendance.  

Most people have heard of Amazon’s cloud service and the more highly technical amongst you are familiar with their compute and storage offerings.   AWS has come a long way, a very long way and they’re not stopping anytime soon.

At the core, AWS is a set of building blocks so organizations can put their infrastructure “In the Cloud” and deploy their applications into virtual environments.   AWS is now a BIG set of building blocks.   It would behoove any infrastructure engineer, software developer or product manager to become familiar with their services.    The fundamental principles of deploying in the cloud are similar to a private virtualized environment,  but there’s a lot more choice (which means although there is much more flexibility, only knowledge of all the services and design patterns will lead to the best solution) plus it’s a different language.  Knowledge of architecting in AWS will provide the appropriately robust and cost-effective solution.  AWS most certainly facilitates deploying for developers.

Here’s a very valid solution in AWS:  Launch EC2 with EBS behind ELB with your domain on Route 53 and your videos on Cloudfront, back up to S3 and your db on RDS with Multi-AZ…..   makes perfect sense… right?  Different language?

For those interested, I’ve listed the core AWS building block services (think Lego’s for data architects) at the end, in the areas of compute, storage, database, deployment & management, CDN, application services and network.   That will help unlock the key to the AWS language example given above.

The keynotes and sessions we attended were fascinating.    The hour plus keynotes can be seen here from Jeff Bezos, Werner Vogels (CTO), and Andy Jassy (SVP).

Powerpoint presentations from the various sessions can be found here.

Here’s an interesting statistic.  In 2003 when amazon.com was a $5B retail business the total amount of servers to support the back-end of that business is the same as what they deploy every single day now for AWS!   In S3 they have about 1 Trillion objects stored and they handle 800K requests per second. Talk about scale !  

Here’s another tidbit related to new products and feature releases.  In 2007, the first full year of AWS operations, there were 9 releases.  In 2011, there were 82 releases.  For 2012 there will be 150 new releases.  In terms of software releases into AWS that happens every 12 seconds..   yes, 12 seconds!

One thing that struck me was the amount of enterprise and government agencies that are using AWS.  AWS now has hundred’s of thousands of customers from 190 countries.   They have 300 government agency customers and 1500 educational institutions in addition to thousands of enterprise customers.  Their vast partner ecosystem has helped fuel their growth.    The keynotes and sessions included presentations from customers including Netflix, Nasdaq, SAP, Pinterest and more.

Major announcements on day 1 included a 25% cost reduction on S3 and two new services.    The first called Redshift, is their first data warehouse solution that scales to Petabytes in the cloud.  Compared to conventional data warehouse solutions Redshift should be ten times lower in cost with high performance.  The second is a new workflow service called Data Pipeline which is more focused on data-related workflows like backups, running analytics, generating reports, etc.

The link to the Jeff Bezos keynote above is actually a one hour fireside chat between Bezos and Werner Vogels.   The main theme from Jeff was focus on your customers and think long-term.   Towards the end he spoke about innovation and how the ability to spin up instances on demand allows entrepreneurs to increase their experimentation rate.   He also offered the following advice to entrepreneurs and those within existing companies:

-         -   Don’t chase today’s fads

-         -   Be passionate about what you do

-         -   Be a missionary not a mercenary

-         -  Start with your customer and work backwards.

Werner Vogel, Amazon’s CTO, took the stage to Nirvana’s music playing on day 2.

He said that AWS was first developed because Amazon needed to scale and they had difficulty dealing with peaks and valleys (think Black Friday).   Werner showed a chart which illustrated that to provision their “old” infrastructure for their peaks, on a yearly basis they were wasting about 40% of their server capacity.  If they considered 15% above peak for November they were wasting almost 76% of their capacity.    This is the primary benefit of the cloud.   You provision what you need on demand and scale up and down accordingly.

Other advice from Werner included:

-         -   Don’t think about resources in the “old” way; an EC2 instance is not a server

-        -    Decompose into small loosely coupled stateless building blocks

-         -   Automate your application and processes

-         -   Let business levers control the system

-         -   Architect with cost in mind

-         -   Protect your customer as a first priority

-        -    In production, deploy to at least two availability zones

-         -   Integrate security into your application from the ground up

-        -    Build, test, integrate and deploy continuously

-        -    Don’t treat failure (equip, system,etc) as an exception, treat it as a normal possible state

So all in all, it was a fascinating few days attending Amazon’s AWS re:Invent

Amazon’s core building block services include:

Compute:

   EC2 (Elastic Compute Cloud) for pay as you go compute capacity

   EMR (Elastic MapReduce) to cost effectively process vast amounts of data

   Auto-Scaling to automatically scale your EC2 capacity up or down (this is huge!)

   ELB (Elastic Load Balancing) automatically distributes traffic across EC2 instances

Storage:

   S3 (Simple Storage Service) fully redundant data storage

   Glacier is a very low cost storage for archive because retrieval is 3 to 5 hours

   EBS (Elastic Block Storage) for block level storage volumes with EC2

   Import/Export for getting data in/out of AWS the ‘old way’ on disk drives, etc.

   Storage Gateway for connecting on-premise appliances to cloud storage

Database:

   RDS (Relational Database Service) for MySQL, SQL server or Oracle set-up

   DynamoDB is a fully managed NoSQL db service

   SimpleDB managed NoSQL db service for smaller datasets

   ElasticCache to deploy, operate and scale an in-memory cache

   Redshift is a data warehouse solution

Deployment & Management

   IAM (Identity and Access Management)

   Cloudwatch for monitoring AWS services starting with EC2

   Elastic Beanstalk simplifies deployment and management through automation

   CloudFormation simplifies provisioning through templates and version control

   Data Pipeline is a workflow solution for data related jobs (backup, analytics, etc.)

CDN (Content Delivery Network)

   Cloudfront is the AWS CDN similar to Akamai or Limelight

Application Services

   CloudSearch is a managed search service

   SWF (Simple Workflow Service) is a workflow engine for your execution state

   SQS (Simple Queue Service) provides a hosted queue for storing messages

   SNS (Simple Notification Service) to manage notifications from the cloud

   SES (Simple Email Service) for scalable, bulk & transactional email sending

   AWS Marketplace to buy software that runs in the cloud

  

Networking

   Amazon Route 53 is a robust DNS (Domain Name Service)

   VPC (Virtual Private Cloud) lets you provision a private, isolated section of AWS

   Direct Connect simplifies establishing a dedicated network connection to AWS

And more………

End of day tech party...  Amazon style