Tag Archives: Amazon Aurora

AWS Hot Startups – August 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-hot-startups-august-2016/

Back with her second guest post, Tina Barr talks about four more hot startups!


Jeff;


This month we are featuring four hot AWS-powered startups:

  • Craftsvilla – Offering a platform to purchase ethnic goods.
  • SendBird – Helping developers build 1-on-1 messaging and group chat quickly.
  • Teletext.io – A solution for content management, without the system.
  • Wavefront – A cloud-based analytics platform.

Craftsvilla
Craftsvilla was born in 2011 out of sheer love and appreciation for the crafts, arts, and culture of India. On a road trip through the Gujarat region of western India, Monica and Manoj Gupta were mesmerized by the beautiful creations crafted by local artisans. However, they were equally dismayed that these artisans were struggling to make ends meet. Monica and Manoj set out to create a platform where these highly skilled workers could connect directly with their consumers and reach a much broader audience. The demand for authentic ethnic products is huge across the globe, but consumers are often unable to find the right place to buy them. Craftsvilla helps to solve this issue.

The culture of India is so rich and diverse, that no one had attempted to capture it on a single platform. Using technological innovations, Craftsvilla combines apparel, accessories, health and beauty products, food items and home décor all in one easily accessible space. For instance, they not only offer a variety of clothing (Salwar suits, sarees, lehengas, and casual wear) but each of those categories are further broken down into subcategories. Consumers can find anything that fits their needs – they can filter products by fabric, style, occasion, and even by the type of work (embroidered, beads, crystal work, handcrafted, etc.). If you are interested in trying new cuisine, Craftsvilla can help. They offer hundreds of interesting products from masalas to traditional sweets to delicious tea blends. They even give you the option to filter through India’s many diverse regions to discover new foods.

Becoming a seller on Craftsvilla is simple. Shop owners just need to create a free account and they’re able to start selling their unique products and services. Craftsvilla’s ultimate vision is to become the ‘one-stop destination’ for all things ethnic. They look to be well on their way!

AWS itself is an engineer on Craftsvilla’s team. Customer experience is highly important to the people behind the company, and an integral aspect of their business is to attain scalability with efficiency. They automate their infrastructure at a large scale, which wouldn’t be possible at the current pace without AWS. Currently, they utilize over 20 AWS services – Amazon Elastic Compute Cloud (EC2), Elastic Load Balancing, Amazon Kinesis, AWS Lambda, Amazon Relational Database Service (RDS), Amazon Redshift, and Amazon Virtual Private Cloud to name a few. Their app QA process will move to AWS Device Farm, completely automated in the cloud, on 250+ services thanks to Lambda. Craftsvilla relies completely on AWS for all of their infrastructure needs, from web serving to analytics.

Check out Craftsvilla’s blog for more information!

SendBird
After successfully exiting their first startup, SendBird founders John S. Kim, Brandon Jeon, Harry Kim, and Forest Lee saw a great market opportunity for a consumer app developer. Today, over 2,000 global companies such as eBay, Nexon, Beat, Malang Studio, and SK Telecom are using SendBird to implement chat and messaging capabilities on their mobiles apps and websites. A few ways companies are using SendBird:

  • 1-on-1 messaging for private messaging and conversational commerce.
  • Group chat for friends and interest groups.
  • Massive scale chat rooms for live-video streams and game communities.

As they watched messaging become a global phenomenon, the SendBird founders realized that it no longer made sense for app developers to build their entire tech stack from scratch. Research from the Localytics Data Team actually shows that in-app messaging can increase app launches by 27% and engagement by 3 times. By simply downloading the SendBird SDK (available for iOS, Android, Unity, .NET Xamarin, and JavaScript), app and web developers can implement real-time messaging features in just minutes. SendBird also provides a full chat history and allows users to send chat messages in addition to complete file and data transfers. Moreover, developers can integrate innovative features such as smart-throttling to control the speed of messages being displayed to the mobile devices during live broadcasting.

After graduating from accelerator Y Combinator W16 Batch, the company grew from 1,000,000 monthly chat users to 5,000,000 monthly chat users within months while handling millions of new messages daily across live-video streaming, games, ecommerce, and consumer apps. Customers found value in having a cross-platform, full-featured, and whole-stack approach to a real-time chat API and SDK which can be deployed in a short period of time.

SendBird chose AWS to build a robust and scalable infrastructure to handle a massive concurrent user base scattered across the globe. It uses EC2 with Elastic Load Balancing and Auto Scaling, Route 53, S3, ElastiCache, Amazon Aurora, CloudFront, CloudWatch, and SNS. The company expects to continue partnering with AWS to scale efficiently and reliably.

Check out SendBird and their blog to follow their journey!

Teletext.io
Marcel Panse and Sander Nagtegaal, co-founders of Teletext.io, had worked together at several startups and experienced the same problem at each one: within the scope of custom software development, content management is a big pain. Even the smallest correction, such as a typo, typically requires a developer, which can become very expensive over time. Unable to find a proper solution that was readily available, Marcel and Sander decided to create their own service to finally solve the issue. Leveraging only the API Gateway, Lambda functions, Amazon DynamoDB, S3, and CloudFront, they built a drop-in content management service (CMS). Their serverless approach for a CMS alternative quickly attracted other companies, and despite intending to use it only for their own needs, the pair decided to professionally market their idea and Teletext.io was born.

Today, Teletext.io is called a solution for content management, without the system. Content distributors are able to edit text and images through a WYSIWYG editor without the help of a programmer and directly from their own website or user interface. There are just three easy steps to get started:

  1. Include Teletext.io script.
  2. Add data attributes.
  3. Login and start typing.

That’s it! There is no system that needs to be installed or maintained by developers – Teletext.io works directly out of the box. In addition to recurring content updates, the data attribution technique can also be used for localization purposes. Making a website multilingual through a CMS can take days or weeks, but Teletext.io can accomplish this task in mere minutes. The time-saving factor is the main benefit for developers and editors alike.

Teletext.io uses AWS in a variety of ways. Since the company is responsible for the website content of others, they must have an extremely fast and reliable system that keeps website visitors from noticing external content being loaded. In addition, this critical infrastructure service should never go down. Both of these requirements call for a robust architecture with as few moving parts as possible. For these reasons, Teletext.io runs a serverless architecture that really makes it stand out. For loading draft content, storing edits and images, and publishing the result, the Amazon API Gateway gets called, triggering AWS Lambda functions. The Lambda functions store their data in Amazon DynamoDB.

Read more about Teletext.io’s unique serverless approach in their blog post.

Wavefront
Founded in 2013 and based in Palo Alto, Wavefront is a cloud-based analytics platform that stores time series data at millions of points per second. They are able to detect any divergence from “normal” in hybrid and cloud infrastructures before anomalies ever happen. This is a critical service that companies like Lyft, Okta, Yammer, and Box are using to keep running smoothly. From data scientists to product managers, from startups to Fortune 500 companies, Wavefront offers a powerful query engine and a language designed for everyone.

With a pay-as-you-go model, Wavefront gives customers the flexibility to start with the necessary application size and scale up/down as needed. They also include enterprise-class support as part of their pricing at no extra cost. Take a look at their product demos to learn more about how Wavefront is helping their customers.

The Wavefront Application is hosted entirely on AWS, and runs its single-tenant instances and multi-tenant instances in the virtual private cloud (VPC) clusters within AWS. The application has deep, native integrations with CloudWatch and CloudTrail, which benefits many of its larger customers also using AWS. Wavefront uses AWS to create a “software problem”, to operate, automate and monitor clouds using its own application. Most importantly, AWS allows Wavefront to focus on its core business – to build the best enterprise cloud monitoring system in the world.

To learn more about Wavefront, check out their blog post, How Does Wavefront Work!

Tina Barr

AWS Week in Review – August 22, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-august-22-2016/

Here’s the first community-driven edition of the AWS Week in Review. In response to last week’s blog post (AWS Week in Review – Coming Back With Your Help!), 9 other contributors helped to make this post a reality. That’s a great start; let’s see if we can go for 20 this week.


Monday

August 22

Tuesday

August 23

Wednesday

August 24

Thursday

August 25

Friday

August 26

Sunday

August 28

New & Notable Open Source

New SlideShare Presentations

Upcoming Events

Help Wanted

Stay tuned for next week, and please consider helping to make this a community-driven effort!

Jeff;

AWS Webinars – August, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-webinars-august-2016/

Everyone on the AWS team understands the value of educating our customers on the best ways to use our services. We work hard to create documentation, training materials, and blog posts for you! We run live events such as our Global AWS Summits and AWS re:Invent where the focus is on education. Last but not least, we put our heads together and create a fresh lineup of webinars for you each and every month.

We have a great selection of webinars on the schedule for August. As always they are free, but they do fill up and I strongly suggest that you register ahead of time. All times are PT, and each webinar runs for one hour:

August 23

August 24

August 25

August 30

August 31


Jeff;

PS – Check out the AWS Webinar Archive for more great content!

 

Amazon Aurora Update – Create Cluster from MySQL Backup

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-aurora-update-create-cluster-from-mysql-backup/

After potential AWS customers see the benefits of moving to the cloud, they often ask about the best way to migrate their applications and their data, including large amounts of structured information stored in relational databases.

Today we are launching an important new feature for Amazon Aurora. If you are already making use of MySQL, either on-premises or on an Amazon EC2 instance, you can now create a snapshot backup of your existing database, upload it to Amazon S3, and use it to create an Amazon Aurora cluster. In conjunction with Amazon Aurora’s existing ability to replicate data from an existing MySQL database, you can easily migrate from MySQL to Amazon Aurora while keeping your application up and running.

This feature can be used to easily and efficiently migrate large (2 TB and more) MySQL databases to Amazon Aurora with minimal performance impact on the source database. Our testing has shown that this process can be up to 20 times faster than using the traditional mysqldump utility. The database can contain both InnoDB and MyISAM tables; however, we do encourage you to migrate from MyISAM to InnoDB where possible.

Here’s an outline of the migration process:

  1. Source Database PreparationEnable binary logging in the source MySQL database and ensure that the logs will be retained for the duration of the migration.
  2. Source Database Backup – Use Percona’s Xtrabackup tool to create a “hot” backup of the source database. This tool does not lock database tables or rows, does not block transactions, and produces compressed backups. You can direct the tool to create one backup file or multiple smaller files; Amazon Aurora can accommodate either option.
  3. S3 Upload – Upload the backup to S3. For backups of 5 TB or less, a direct upload via the AWS Management Console or the AWS Command Line Interface (CLI) is generally sufficient. For larger backups, consider using AWS Import/Export Snowball.
  4. IAM Role – Create an IAM role that allows Amazon Relational Database Service (RDS) to access the uploaded backup and the bucket it resides within. The role must allow RDS to perform the ListBucket and GetBucketLocation operations on the bucket and the GetObject operation on the backup (you can find a sample policy in the documentation).
  5. Create Cluster – Create a new Amazon Aurora cluster from the uploaded backup. Click on Restore Aurora DB Cluster from S3 in the RDS  Console, enter the version number of the source database, point to the S3 bucket and choose the IAM role, then click on Next Step. Proceed through the remainder of the cluster creation pages (Specify DB Details and Configure Advanced Settings) in the usual way:

    Amazon Aurora will process the backup files in alphabetical order.

  6. Migrate MySQL Schema – Migrate (as appropriate) the users, permissions, and configuration settings in the MySQL INFORMATION_SCHEMA.
  7. Migrate Related Items – Migrate the triggers, functions, and stored procedures from the source database to the new Amazon Aurora cluster.
  8. Initiate Replication – Begin replication from the source database to the new Amazon Aurora cluster and wait for the cluster to catch up.
  9. Switch to Cluster – Point all client applications at the Amazon Aurora cluster.
  10. Terminate  Replication – End replication to the Amazon Aurora cluster.

Given the mission-critical nature of a production-level relational database, a dry run is always a good idea!

Available Now
This feature is available now and you can start using it today in all public AWS regions with the exception of Asia Pacific (Mumbai). To learn more, read Migrating Data from an External MySQL Database to an Amazon Aurora DB Cluster in the Amazon Aurora User Guide.


Jeff;

 

Now Open – AWS Asia Pacific (Mumbai) Region

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-open-aws-asia-pacific-mumbai-region/

We are expanding the AWS footprint again, this time with a new region in Mumbai, India. AWS customers in the area can use the new Asia Pacific (Mumbai) Region to better serve end users in India.

New Region
The new Mumbai region has two Availability Zones, raising the global total to 35. It supports Amazon Elastic Compute Cloud (EC2) (C4, M4, T2, D2, I2, and R3 instances are available) and related services including Amazon Elastic Block Store (EBS), Amazon Virtual Private Cloud, Auto Scaling, and  Elastic Load Balancing.

It also supports the following services:

There are now three edge locations (Mumbai, Chennai, and New Delhi) in India. The locations support Amazon Route 53, Amazon CloudFront, and S3 Transfer Acceleration. AWS Direct Connect support is available via our Direct Connect Partners (listed below).

This is our thirteenth region (see the AWS Global Infrastructure map for more information). As usual, you can see the list of regions in the region menu of the Console:

Customers
There are over 75,000 active AWS customers in India, representing a diverse base of industries. In the time leading up to today’s launch, we have provided some of these customers with access to the new region in preview form. Two of them (Ola Cabs and NDTV) were kind enough to share some of their experience and observations with us:

Ola Cabs’ mobile app leverages AWS to redefine point-to-point transportation in more than 100 cities across India. AWS allows OLA to constantly innovate faster with new features and services for their customers, without compromising on availability or the customer experience of their service. Ankit Bhati (CTO and Co-Founder) told us:

We are using technology to create mobility for a billion Indians, by giving them convenience and access to transportation of their choice. Technology is a key enabler, where we use AWS to drive supreme customer experience, and innovate faster on new features & services for our customers. This has helped us reach 100+ cities & 550K driver partners across India. We do petabyte scale analytics using various AWS big data services and deep learning techniques, allowing us to bring our driver-partners close to our customers when they need them. AWS allows us to make 30+ changes a day to our highly scalable micro-services based platform consisting of 100s of low latency APIs, serving millions of requests a day. We have tried the AWS India region. It is great and should help us further enhance the experience for our customers.


NDTV, India’s leading media house is watched by millions of people across the world. NDTV has been using AWS since 2009 to run their video platform and all their web properties. During the Indian general elections in May 2014, NDTV fielded an unprecedented amount of web traffic that scaled 26X from 500 million hits per day to 13 billion hits on Election Day (regularly peaking at 400K hits per second), all running on AWS.  According to Kawaljit Singh Bedi (CTO of NDTV Convergence):

NDTV is pleased to report very promising results in terms of reliability and stability of AWS’ infrastructure in India in our preview tests. Based on tests that our technical teams have run in India, we have determined that the network latency from the AWS India infrastructure Region are far superior compared to other alternatives. Our web and mobile traffic has jumped by over 30% in the last year and as we expand to new territories like eCommerce and platform-integration we are very excited on the new AWS India region launch. With the portfolio of services AWS will offer at launch, low latency, great reliability, and the ability to meet regulatory requirements within India, NDTV has decided to move these critical applications and IT infrastructure all-in to the AWS India region from our current set-up.

 


Here are some of our other customers in the region:

Tata Motors Limited, a leading Indian multinational automotive manufacturing company runs its telematics systems on AWS. Fleet owners use this solution to monitor all vehicles in their fleet on a real time basis. AWS has helped Tata Motors become to more agile and has increased their speed of experimentation and innovation.

redBus is India’s leading bus ticketing platform that sells their tickets via web, mobile, and bus agents. They now cover over 67K routes in India with over 1,800 bus operators. redBus has scaled to sell more than 40 million bus tickets annually, up from just 2 million in 2010. At peak season, there are over 100 bus ticketing transactions every minute. The company also recently developed a new SaaS app on AWS that gives bus operators the option of handling their own ticketing and managing seat inventories. redBus has gone global expanding to new geographic locations such as Singapore and Peru using AWS.

Hotstar is India’s largest premium streaming platform with more than 85K hours of drama and movies and coverage of every major global sporting event. Launched in February 2015, Hotstar quickly became one of the fastest adopted new apps anywhere in the world. It has now been downloaded by more than 68M users and has attracted followers on the back of a highly evolved video streaming technology and high attention to quality of experience across devices and platforms.

Macmillan India has provided publishing services to the education market in India for more than 120 years. Prior to using AWS, Macmillan India has moved its core enterprise applications — Business Intelligence (BI), Sales and Distribution, Materials Management, Financial Accounting and Controlling, Human Resources and a customer relationship management (CRM) system from an existing data center in Chennai to AWS. By moving to AWS, Macmillan India has boosted SAP system availability to almost 100 percent and reduced the time it takes them to provision infrastructure from 6 weeks to 30 minutes.

Partners
We are pleased to be working with a broad selection of partners in India. Here’s a sampling:

  • AWS Premier Consulting Partners – BlazeClan Technologies Pvt. Limited, Minjar Cloud Solutions Pvt Ltd, and Wipro.
  • AWS Consulting Partners – Accenture, BluePi, Cloudcover, Frontier, HCL, Powerupcloud, TCS, and Wipro.
  • AWS Technology Partners – Freshdesk, Druva, Indusface, Leadsquared, Manthan, Mithi, Nucleus Software, Newgen, Ramco Systems, Sanovi, and Vinculum.
  • AWS Managed Service Providers – Progressive Infotech and Spruha Technologies.
  • AWS Direct Connect Partners – AirTel, Colt Technology Services,  Global Cloud Xchange, GPX, Hutchison Global Communications, Sify, and Tata Communications.

Amazon Offices in India
We have opened six offices in India since 2011 – Delhi, Mumbai, Hyderabad, Bengaluru, Pune, and Chennai. These offices support our diverse customer base in India including enterprises, government agencies, academic institutions, small-to-mid-size companies, startups, and developers.

Support
The full range of AWS Support options (Basic, Developer, Business, and Enterprise) is also available for the Mumbai Region. All AWS support plans include an unlimited number of account and billing support cases, with no long-term contracts.

Compliance
Every AWS region is designed and built to meet rigorous compliance standards including ISO 27001, ISO 9001, ISO 27017, ISO 27018, SOC 1, SOC 2, and PCI DSS Level 1 (to name a few). AWS implements an information Security Management System (ISMS) that is independently assessed by qualified third parties. These assessments address a wide variety of requirements which are communicated to customers by making certifications and audit reports available, either on our public-facing website or upon request.

To learn more; take a look at the AWS Cloud Compliance page and our Data Privacy FAQ.

Use it Now
This new region is now open for business and you can start using it today! You can find additional information about the new region, documentation on how to migrate, customer use cases, information on training and other events, and a list of AWS Partners in India on the AWS site.

We have set up a seller of record in India (known as AISPL); please see the AISPL customer agreement for details.


Jeff;

 

Guest Post – Zynga Gets in the Game with Amazon Aurora

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/guest-post-zynga-gets-in-the-game-with-amazon-aurora/

Long-time AWS customer Zynga is making great use of Amazon Aurora and other AWS database services. In today’s guest post you can learn about how they use Amazon Aurora to accommodate spikes in their workload. This post was written by Chris Broglie of Zynga.


Jeff;


Zynga has long operated various database technologies, ranging from simple in-memory caches like Memcached, to persistent NoSQL stores like Redis and Membase, to traditional relational databases like MySQL. We loved the features these technologies offered, but running them at scale required lots of manual time to recover from instance failure and to script and monitor mundane but critical jobs like backup and recovery. As we migrated from our own private cloud to AWS in 2015, one of the main objectives was to reduce the operational burden on our engineers by embracing the many managed services AWS offered.

We’re now using Amazon DynamoDB and Amazon ElastiCache (Memcached and Redis) widely in place of their self-managed equivalents. Now, engineers are able to focus on application code instead of managing database tiers, and we’ve improved our recovery times from instance failure (spoiler alert: machines are better at this than humans). But the one component missing here was MySQL. We loved the automation Amazon RDS for MySQL offers, but it relies on general-purpose Amazon Elastic Block Store (EBS) volumes for storage. Being able to dynamically allocate durable storage is great, but you trade off having to send traffic over the network, and traditional databases suffer from this additional latency. Our testing showed that the performance of RDS for MySQL just couldn’t compare to what we could obtain with i2 instances and their local (though ephemeral) SSDs. Provisioned IOPS narrow the gap, but they cost more. For these reasons, we used self-managed i2 instances wherever we had really strict performance requirements.

However, for one new service we were developing during our migration, we decided to take a measured bet on Amazon Aurora. Aurora is a MySQL-compatible relational database offered through Amazon RDS. Aurora was only in preview when we started writing the service, but it was expected to become generally available before production, and we knew we could always fall back to running MySQL on our own i2 instances. We were naturally apprehensive of any new technology, but we had to see for ourselves if Aurora could deliver on its claims of exceeding the performance of MySQL on local SSDs, while still using network storage and providing all the automation of a managed service like RDS. And after 8 months of production, Aurora has been nothing short of impressive. While our workload is fairly modest – the busiest instance is an r3.2xl handling ~9k selects/second during peak for a 150 GB data set – we love that so far Aurora has delivered the necessary performance without any of the operational overhead of running MySQL.

An example of what this kind of automation has enabled for us was an ops incident where a change in traffic patterns resulted in a huge load spike on one of our Aurora instances. Thankfully, the instance was able to keep serving traffic despite 100% CPU usage, but we needed even more throughput. With Aurora we were able to scale up the reader to an instance that was 4x larger, failover to it, and then watch it handle 4x the traffic, all with just a few clicks in the RDS console. And days later after we released a patch to prevent the incident from recurring, we were able to scale back down to smaller instances using the same procedure. Before Aurora we would have had to either get a DBA online to manually provision, replicate, and failover to a larger instance, or try to ship a code hotfix to reduce the load on the database. Manual changes are always slower and riskier, so Aurora’s automation is a great addition to our ops toolbox, and in this case it led to a resolution measured in minutes rather than hours.

Most of the automation we’re enjoying has long been standard for RDS, but using Aurora has delivered the automation of RDS along with the performance of self-managed i2 instances. Aurora is now our first choice for new services using relational databases.

Chris Broglie, Architect (Zynga)

 

JOIN Amazon Redshift AND Amazon RDS PostgreSQL WITH dblink

Post Syndicated from Tony Gibbs original https://blogs.aws.amazon.com/bigdata/post/Tx1GQ6WLEWVJ1OX/JOIN-Amazon-Redshift-AND-Amazon-RDS-PostgreSQL-WITH-dblink

Tony Gibbs is a Solutions Architect with AWS

When it comes to choosing a SQL-based database in AWS, there are many options. Sometimes it can be difficult to know which one to choose. For example, when would you use Amazon Aurora instead of Amazon RDS PostgreSQL or Amazon Redshift? To answer this question, you must first understand the nature of the data workload and then evaluate other factors such as the quantity of data and query access patterns.

The design and capabilities of the different AWS services mean that each service has different strengths and excels at different workloads. This leads to trying to pick the right tool for the job, which can result in tradeoffs. But sometimes you don’t want to compromise.

This post explains how to use two services together—Amazon Redshift and Amazon RDS PostgreSQL—to avoid tradeoffs when choosing between a columnar data store and a row-based data store.

Amazon Redshift

Amazon Redshift is a high-performance, petabyte-scale data warehouse service that excels at online analytical processing (OLAP) workloads. Databases such as RDS PostgreSQL or Amazon Aurora typically store terabytes of data, and they excel at online transaction processing (OLTP) workloads.

Amazon Redshift uses a columnar architecture, which means the data is organized by columns on disk instead of row-by-row as in the OLTP approach. Columnar architecture offers advantages when querying a subset of the columns in a table by greatly reducing I/O. And because the data is stored by column, it can be highly compressed which further reduces I/O and allows more data to be stored and quickly queried.

RDS PostgreSQL uses a row-based architecture, which offers advantages when the workload is selecting, inserting, updating or deleting a small number of rows at a time, which is typical for OLTP.

Amazon Redshift also uses a massively parallel processing (MPP), shared-nothing architecture. This means that Amazon Redshift is designed to use all of the computing resources across many machines (called nodes) even when executing a single query. This provides excellent performance for analytical queries across a large number of rows. In contrast, most OLTP databases only use a subset of resources on one machine to process each query. This difference in architecture means that most OLTP databases can handle more concurrent queries because each query is typically less resource-intensive than those in Amazon Redshift.

Linking the high-performance power of Amazon Redshift with the feature-richness of RDS PostgreSQL is an attractive proposition because the two systems complement each other so well. How is it possible to link these two systems? An RDS PostgreSQL database is not an MPP database, but it does have features that enable multiple instances to be linked to one another.

Interestingly, Amazon Redshift was originally forked from PostgreSQL, which is why PostgreSQL drivers and API libpq work with Amazon Redshift. The combination of this PostgreSQL feature and Amazon Redshift compatibility lets the two systems be connected. This connection enables PostgreSQL to issue queries, and Amazon Redshift to return the results for processing to PostgreSQL.

Combining Amazon Redshift and RDS PostgreSQL provides the following benefits: 

  • Materialized views for cached copies of data that work well for high-concurrency dashboards
  • High-concurrency, partitioned aggregate tables with block range indexes (BRIN).
  • Procedural Language/PostgreSQL (PL/pgSQL) user-defined functions that can query Amazon Redshift by using dynamic SQL.
  • Post-Amazon Redshift transformation, such as returning result sets as JSON.

The diagram above shows how the connections flow between the end users and the databases. Optionally, you can connect directly to Amazon Redshift if needed. If that is the case, consider configuring pgbouncer-rr on an Amazon EC2 instance to simplify management of the two connections. The diagram below illustrates this solution:

For further reading, check out Bob Strahan’s blog post Query Routing and Rewrite: Introducing pgbouncer-rr for Amazon Redshift and PostgreSQL post.

RDS PostgreSQL includes two extensions to execute queries remotely. The first extension is the PostgreSQL foreign-data wrapper, postgres_fdw. The postgres_fdw module enables the creation of external tables. External tables can be queried in the same way as a local native table, However, the query is not currently executed entirely on the remote side because postgres_fdw doesn’t push down aggregate functions and limit clauses. When you perform an aggregate query through an external table, all the data is pulled into PostgreSQL for an aggregation step. This is unacceptably slow for any meaningful number of rows.

The second extension is dblink, which includes a function also called dblink. The dblink function allows the entire query to be pushed to Amazon Redshift. This lets Amazon Redshift do what it does best—query large quantities of data efficiently and return the results to PostgreSQL for further processing.

Installation and setup

To set up this solution:

  1. Launch an Amazon Redshift cluster.
  2. Launch an RDS PostgreSQL (9.5+) instance in the same Availability Zone as the cluster in Step 1.
  3. Configure the VPC security group for the Amazon Redshift cluster to allow an incoming connection from the RDS PostgreSQL endpoint.
  4. Optional: load the Amazon Redshift sample data to run the queries included in this post.
  5. Connect to the RDS PostgreSQL instance, and then run the following SQL code, replacing the <placeholders> with the values from your own instances:
CREATE EXTENSION postgres_fdw;
CREATE EXTENSION dblink;
CREATE SERVER foreign_server
        FOREIGN DATA WRAPPER postgres_fdw
        OPTIONS (host '<amazon_redshift_hostname>', port '<port>', dbname '<database_name>', sslmode 'require');
CREATE USER MAPPING FOR <rds_postgresql_username>
        SERVER foreign_server
        OPTIONS (user '<amazon_redshift_username>', password '<password>');

For more information, see dblink in the PostgreSQL documentation.

Basic querying

The dblink function requires you to pass in the SQL statement as a string and define the result set, as shown in the following query:

SELECT *
FROM dblink('foreign_server',$REDSHIFT$
    SELECT sellerid, sum(pricepaid) sales
    FROM sales 
    WHERE saletime >= '2008-01-01' 
    AND saletime < '2008-02-01' 
    GROUP BY sellerid 
    ORDER BY sales DESC
$REDSHIFT$) AS t1 (sellerid int, sales decimal);

In this example:

  • The dblink function accepts the server connection (‘foreign_server’) that was created in the previous step.
  • The SQL query is passed in as a string between double dollar quotes ($REDSHIFT$). Using dollar quotes makes reading and writing the query easier.
  • The double dollar quotes are labeled REDSHIFT to help highlight the SQL that will be sent to Amazon Redshift.
  • The query results are a recordset that you must name and for which you must specify the datatypes (AS t1(sellerid int, sales decimal). This enables further joining and processing.

The partial result set from this query is:

After the results from the query are returned, PostgreSQL can do further processing such as transforming the results to JSON, as in the following example:

SELECT array_to_json(array_agg(t1)) FROM (
    SELECT *
    FROM
        dblink('foreign_server',$REDSHIFT$
             SELECT sellerid, sum(pricepaid) sales
             FROM sales 
             WHERE saletime >= '2008-01-01' 
             AND saletime < '2008-02-01' 
             GROUP BY sellerid 
             ORDER BY sales DESC    
$REDSHIFT$) AS t1 (sellerid int, sales decimal)
) t1;

Querying with views

To make the SQL less cumbersome to use, these queries can also be expressed as views. Below is a view of the first basic query:

CREATE OR REPLACE VIEW v_sales AS
SELECT *
FROM dblink ('foreign_server',$REDSHIFT$ 
    SELECT sellerid, sum(pricepaid) sales
    FROM sales 
    WHERE saletime >= '2008-01-01' 
    AND saletime < '2008-02-01' 
    GROUP BY sellerid 
    ORDER BY sales DESC
$REDSHIFT$) AS t1 (sellerid int, sales decimal);

Querying the view returns the same results as before:

SELECT * from v_sales;

Querying with user-defined functions

Another approach is to query the data using user-defined functions (UDF). In contrast to views, UDFs enable you to specify parameters when you run the UDF.  In the previous example, the date range was hard-coded in the view. With a UDF, you can query on an arbitrary datetime range. The following PL/pgSQL code creates the UDF in PostgreSQL:

CREATE OR REPLACE FUNCTION get_sales(_startdate timestamp, _enddate timestamp)
RETURNS TABLE (sellerid int, sales decimal) AS
$BODY$
DECLARE
remote_sql TEXT;
BEGIN
remote_sql = FORMAT( '
    SELECT sellerid, sum(pricepaid) sales
    FROM sales 
    WHERE saletime >= %L AND saletime < %L 
    GROUP BY sellerid 
    ORDER BY sales DESC
', _startdate, _enddate);
RETURN QUERY 
SELECT *
FROM dblink('foreign_server', remote_sql) 
AS t1  (sellerid int, sales decimal);
END;
$BODY$
LANGUAGE plpgsql VOLATILE;

In PostgreSQL, the function can be executed in a query:

SELECT * FROM get_sales ('2008-01-01', '2008-02-01');

The query returns the expected result set.

Caching data with materialized views

In the case of frequently accessed data, it may be better to use a materialized view. Materialized views cache the results so that the query can skip re-computing results. This makes them ideal for caching a small amount of frequently requested data, as in dashboards.

The following materialized view counts all the number of users who like sports and groups them by state. The DDL to create this materialized view is as follows:

CREATE MATERIALIZED VIEW v_users_likes_by_state AS
SELECT *
FROM dblink('foreign_server',$REDSHIFT$
        SELECT state, sum(likesports::int) sports_like_count
        FROM users 
        GROUP BY state
$REDSHIFT$) AS t1 (state text, sports_like_count int);

When the materialized view is created, the query is issued against Amazon Redshift. When a query is issued against the materialized view, there is no query issued against Amazon Redshift and instead the results are returned directly from PostgreSQL. Querying a materialized view is the same as querying a regular view, as shown in the following example:

SELECT * FROM v_users_likes_by_state;

Materialized views hold a cache of data that can become stale. The following SQL statement refreshes the data and re-issues the original query that created the materialized view:

REFRESH MATERIALIZED VIEW v_users_likes_by_state;

For more information, see Materialized Views in the PostgreSQL documentation.

To refresh the materialized view at regular intervals, you can use AWS Lambda. The Node.js code to refresh the materialized view is as follows:

var pg = require("pg");

exports.handler = function(event, context) {   
    var conn = "pg://username:[email protected]:port/dbname";
    var client = new pg.Client(conn);
    client.connect(function(err) {
        if (err) {
            context.fail("Failed" + err);
        }
        client.query('REFRESH MATERIALIZED VIEW v_users_likes_by_state', function (err, result) {
            if (err) {
                context.fail("Failed to run query" + err);
            }
            client.end();
            context.succeed("Successfully Refreshed.");
        });
    });
};

Lambda requires the pg module, which can be installed using the following command:

npm install pg

For more information about creating the Lambda deployment file, see Creating a Deployment Package (Node.js).

Copying data from Amazon Redshift to RDS PostgreSQL

When there is a larger quantity of data, it might be better to copy the data using the dblink function into PostgreSQL tables instead of using materialized views. This is useful where there are large quantities of new data and only the latest data needs to be copied. The disadvantage of the materialized view is that refreshes copy all of the data from the beginning.

The SQL to create the table:

CREATE TABLE sales_summary (
   saletime timestamp,
   sellerid int,
   sales decimal
);

PostgreSQL uses indexes to optimize reads and the new BRIN is an appropriate index for ordered timestamps. The SQL to create the index is:

CREATE INDEX idx_sales_summary_brin
   ON sales_summary
   USING BRIN (saletime);

The following query shows how to insert data into the tables using the dblink function:

INSERT INTO sales_summary

INSERT INTO sales_summary
SELECT *
FROM dblink('foreign_server',$REDSHIFT$
    SELECT date_trunc('hours', saletime) AS ts, sellerid, sum(pricepaid) sales
    FROM sales 
    WHERE saletime >= '2008-01-01' 
    AND saletime < '2008-02-01' 
    GROUP BY ts, sellerid 
    ORDER BY sales 
$REDSHIFT$) AS t1 (saletime timestamp, sellerid int, sales decimal);

Conclusion

You can use the dblink extension to connect to Amazon Redshift and leverage PostgreSQL functionality. This allows you to cache frequently queried small data sets, with two choices: materialized views with refreshes, or copying data into tables. For querying non-cached data, there are also two choices: regular views or UDFs that take parameters. There are likely many other uses for the dblink extension with Amazon Redshift, such as PostGIS or LDAP support in PostgreSQL (Amazon EC2 only), but they are beyond the scope of this post.

If you have questions or suggestions, please comment below.

————————————-

Related

Real-time in-memory OLTP and Analytics with Apache Ignite on AWS

Want to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.

New – Cross-Region Read Replicas for Amazon Aurora

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-cross-region-read-replicas-for-amazon-aurora/

You already have the power to scale the read capacity of your Amazon Aurora instances by adding additional read replicas to an existing cluster. Today we are giving you the power to create a read replica in another region. This new feature will allow you to support cross-region disaster recovery and to scale out reads. You can also use it to migrate from one region to another or to create a new database environment in a different region.

Creating a read replica in another region also creates an Aurora cluster in the region. This cluster can contain up to 15 more read replicas, with very low replication lag (typically less than 20 ms) within the region (between regions, latency will vary based on the distance between the source and target). You can use this model to duplicate your cluster and read replica setup across regions for disaster recovery. In the event of a regional disruption, you can promote the cross-region replica to be the master. This will allow you to minimize downtime for your cross-region application. This feature applies to unencrypted Aurora clusters.

Before you get actually create the read replica, you need to take care of a pair of prerequisites. You need to make sure that a VPC and the Database Subnet Groups exist in the target region, and you need to enable binary logging on the existing cluster.

Setting up the VPC
Because Aurora always runs within a VPC, ensure that the VPC and the desired Database Subnet Groups exist in the target region. Here are mine:

Enabling Binary Logging
Before you can create a cross region read replica, you need to enable binary logging for your existing cluster. Create a new DB Cluster Parameter Group (if you are not already using a non-default one):

Enable binary logging (choose MIXED) and then click on Save Changes:

Next, Modify the DB Instance, select the new DB Cluster Parameter Group, check Apply Immediately, and click on Continue. Confirm your modifications, and then click on Modify DB Instance to proceed:

Select the instance and reboot it, then wait until it is ready.

Create Read Replica
With the prerequisites out of the way it is time to create the read replica! From within the AWS Management Console, select the source cluster and choose Create Cross Region Read Replica from the Instance Actions menu:

Name the new cluster and the new instance, and then pick the target region. Choose the DB Subnet Group and set the other options as desired, then click Create:

Aurora will create the cluster and the instance. The state of both items will remain at creating until the items have been created and the data has been replicated (this could take some time, depending on amount of data stored in the existing cluster.

This feature is available now and you can start using it today!


Jeff;

 

New – Cross-Account Snapshot Sharing for Amazon Aurora

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-cross-account-snapshot-sharing-for-amazon-aurora/

Amazon Aurora is a high-performance, MySQL-compatible database engine. Aurora combines the speed and availability of high-end commercial databases with the simplicity and cost-effective of open source databases (see my post, Amazon Aurora – New Cost-Effective MySQL-Compatible Database Engine for Amazon RDS, to learn more). Aurora shares some important attributes with the other database engines that are available for Amazon RDS including easy administration, push-button scalability, speed, security, and cost-effectiveness.

You can create a snapshot backup of an Aurora cluster with just a couple of clicks. After you have created a snapshot, you can use it to restore your database, once again with a couple of clicks.

Share Snapshots
Today we are giving you the ability to share your Aurora snapshots. You can share them with other AWS accounts and you can also make them public. These snapshots can be used to restore the database to an Aurora instance running in a separate AWS account in the same Region as the snapshot.

There are several primary use cases for snapshot sharing:

Separation of Environments – Many AWS customers use separate AWS accounts for their development, test, staging, and production environments. You can share snapshots between these accounts as needed. For example, you can generate the initial database in your staging environment, snapshot it, share the snapshot with your production account, and then use it to create your production database. Or, should you encounter an issue with your production code or queries, you can create a snapshot of your production database and then share it with your test account for debugging and remediation.

Partnering – You can share database snapshots with selected partners on an as-needed basis.

Data Dissemination -If you are running a research project, you can generate snapshots and then share them publicly. Interested parties can then create their own Aurora databases using the snapshots, using your work and your data as a starting point.

To share a snapshot, simply select it in the RDS Console and click on Share Snapshot. Then enter the target AWS account (or click on Public to share the snapshot publicly) and click on Add:

You can share manually generated, unencrypted snapshots with other AWS accounts or publicly. You cannot share automatic snapshots or encrypted snapshots.

The shared snapshot becomes visible in the other account right away:

Public snapshots are also visible (select All Public Snapshots as the Filter):

Available Now
This feature is available now and you can start using it today.


Jeff;

GE Oil & Gas – Digital Transformation in the Cloud

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/ge-oil-gas-digital-transformation-in-the-cloud/

GE Oil & Gas is a relatively young division of General Electric, the product of a series of acquisitions made by parent company General Electric starting in the late 1980s. Today GE Oil &Gas is pioneering the digital transformation of the company. In the guest post below, Ben Cabanas, the CTO of GE Transportation and formerly the cloud architect for GE Oil & Gas, talks about some of the key steps involved in a major enterprise cloud migration, the theme of his recent presentation at the 2016 AWS Summit in Sydney, Australia.

You may also want to learn more about Enterprise Cloud Computing with AWS.


Jeff;


Challenges and Transformation
GE Oil & Gas is at the forefront of GE’s digital transformation, a key strategy for the company going forward. The division is also operating at a time when the industry is facing enormous competitive and cost challenges, so embracing technological innovation is essential. As GE CIO Jim Fowler has noted, today’s industrial companies have to become digital innovators to thrive.

Moving to the cloud is a central part of this transformation for GE. Of course, that’s easier said than done for a large enterprise division of our size, global reach, and role in the industry. GE Oil & Gas has more than 45,000 employees working across 11 different regions and seven research centers. About 85 percent of the world’s offshore oil rigs use our drilling systems, and we spend $5 billion annually on energy-related research and development—work that benefits the entire industry. To support all of that work, GE Oil & Gas has about 900 applications, part of a far larger portfolio of about 9,000 apps used across GE. A lot of those apps may have 100 users or fewer, but are still vital to the business, so it’s a huge undertaking to move them to the cloud.

Our cloud journey started in late 2013 with a couple of goals. We wanted to improve productivity in our shop floors and manufacturing operations. We sought to build applications and solutions that could reduce downtime and improve operations. Most importantly, we wanted to cut costs while improving the speed and agility of our IT processes and infrastructure.

Iterative Steps
Working with AWS Professional Services and Sogeti, we launched the cloud initiative in 2013 with a highly iterative approach. In the beginning, we didn’t know what we didn’t know, and had to learn agile as well as how to move apps to the cloud. We took steps that, in retrospect, were crucial in supporting later success and accelerated cloud adoption. For example, we sent more than 50 employees to Seattle for training and immersion in AWS technologies so we could keep critical technical IP in-house. We built foundational services on AWS, such as monitoring, backup, DNS, and SSO automation that, after a year or so, fostered the operational maturity to speed the cloud journey. In the process, we discovered that by using AWS, we can build things at a much faster pace than what we could ever accomplish doing it internally.

Moving to AWS has delivered both cost and operational benefits to GE Oil & Gas.

We architected for resilience, and strove to automate as much as possible to reduce touch times. Because automation was an overriding consideration, we created a “bot army” that is aligned with loosely coupled microservices to support continuous development without sacrificing corporate governance and security practices. We built in security at every layer with smart designs that could insulate and protect GE in the cloud, and set out to measure as much as we could—TCO, benchmarks, KPIs, and business outcomes. We also tagged everything for greater accountability and to understand the architecture and business value of the applications in the portfolio.

Moving Forward
All of these efforts are now starting to pay off. To date, we’ve realized a 52 percent reduction in TCO. That stems from a number of factors, including the bot-enabled automation, a push for self-service, dynamic storage allocation, using lower-cost VMs when possible, shutting off compute instances when they’re not needed, and moving from Oracle to Amazon Aurora. Ultimately, these savings are a byproduct of doing the right thing.

The other big return we’ve seen so far is an increase in productivity. With more resilient, cloud-enabled applications and a focus on self-service capability, we’re getting close to a “NoOps” environment, one where we can move away from “DevOps” and “ArchOps,” and all the other “ops,” using automation and orchestration to scale effectively without needing an army of people. We’ve also seen a 50 percent reduction in “tickets” and a 98 percent reduction in impactful business outages and incidents—an unexpected benefit that is as valuable as the cost savings.

For large organizations, the cloud journey is an extended process. But we’re seeing clear benefits and, from the emerging metrics, can draw a few conclusions. NoOps is our future, and automation is essential for speed and agility—although robust monitoring and automation require investments of skill, time, and money. People with the right skills sets and passion are a must, and it’s important to have plenty of good talent in-house. It’s essential to partner with business leaders and application owners in the organization to minimize friction and resistance to what is a major business transition. And we’ve found AWS to be a valuable service provider. AWS has helped move a business that was grounded in legacy IT to an organization that is far more agile and cost efficient in a transformation that is adding value to our business and to our people.

— Ben Cabanas, Chief Technology Officer, GE Transportation

 

AWS Week in Review – April 18, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-april-18-2016/

Let’s take a quick look at what happened in AWS-land last week:

Monday

April 18

Tuesday

April 19

Wednesday

April 20

Thursday

April 21

Friday

April 22

Saturday

April 23

Sunday

April 24

New & Notable Open Source

New SlideShare Presentations

New Customer Success Stories

New YouTube Videos

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

AWS Webinars – April 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-webinars-april-2016/

Our 2016 series of webinars continues with a strong set of 200-level topics in April. The webinars are free, but space is limited and you should sign up ahead of time if you would like to attend. Here’s what we have on the calendar for the last week of April (all times are Pacific):

Tuesday, April 26
Are you ready to launch and connect to your first EC2 instance? Do you want to learn how to use Amazon Simple Storage Service (S3) to store and share files? Our getting started webinar will show you how to do both.

Webinar: Getting Started with AWS (9 – 10 AM).

Do you want to learn how to use Apache Spark to analyze real-time streams of data on an Amazon EMR cluster? Do you want to know how to use Spark as part of a system that includes Amazon DynamoDB, Amazon Redshift, Amazon Kinesis, and other big data tools? This webinar will show you how to use Spark to address common big data use cases.

Webinar: Best Practices for Apache Spark on AWS (10:30 – 11:30 AM).

Are you interested in running a commercial relational database in the cloud? Do you want to know more about best practices for running single and multiple database instances, or do you have questions about costs and licensing? Attend this webinar to learn more about Amazon RDS running Oracle.

Webinar: RDS for Oracle: Quick Provision, Easy to Manage, Reduced Cost (Noon – 1 PM).

Wednesday, April 27
In today’s real-time world, going from raw data to insights as quickly as possible has become a must. Fortunately, a number of AWS tools can help you to capture, store, and analyze real-time streaming data. Attend this webinar to learn about Amazon Kinesis Streams, Lambda, and Spark Streaming on Amazon EMR.

Webinar: Getting Started with Real-Time Data Analytics on AWS (9 – 10 AM).

As you move your business and your applications to the cloud, you should also look at modernizing your development and deployment practices. For example, many AWS customers use tools like AWS CodePipeline and AWS CodeDeploy to implement continuous delivery. Attend this webinar to learn more about what this means and how to put it in to practice in your own organization.

Webinar: Getting Started with Continuous Delivery on AWS (10:30 – 11:30 AM).

Now that Amazon S3 is a decade old, we have a wealth of experience to share about the best ways to use it for backup, compliance, archiving, and many other purposes. This webinar will share best practices for keeping your data safe, and will also provide an overview of several different transfer services.

Webinar: S3 Best Practices: A Decade of Field Experience (Noon – 1 PM).

Thursday, April 28
AWS Lambda brings some new flexibility to the development and deployment process. When used in conjunction with AWS Storage Gateway, it can be used as the basis for an automated development workflow that easily supports distinct development, staging, and production environments. Attend this webinar to learn more.

Webinar: Continuous Delivery to AWS Lambda (9 AM – 10 AM).

Amazon Aurora is a MySQL-compatible database engine that can boost performance, reliability, and availability while reducing the total cost of ownership. Join this webinar to learn more about Aurora and to better understand how to migrate your existing on-premises or cloud-based databases to it.

Webinar: Migrating Your Databases to Amazon Aurora (10:30 – 11:30 AM).

Containers and microservices are both a natural fit for the cloud. Attend this webinar to learn more about the challenges that might arise and the best practices to address them.

Webinar: Running Microservices on Amazon ECS (Noon – 1 PM).


Jeff;

 

AWS Week in Review – April 11, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-april-11-2016/

Let’s take a quick look at what happened in AWS-land last week:

Monday

April 11

Tuesday

April 12

Wednesday

April 13

Thursday

April 14

Friday

April 15

Saturday

April 16

Sunday

April 17

New & Notable Open Source

  • cfn-include implements a Fn::Include for CloudFormation templates.
  • TumblessTemplates is a set of CloudFormation templates for quick setup of the Tumbless blogging platform.
  • s3git is Git for cloud storage.
  • s3_uploader is an S3 file uploader GUI written in Python.
  • SSH2EC2 lets you connect to EC2 instances via tags and metadata.
  • lambada is AWS Lambda for silly people.
  • aws-iam-proxy is a proxy that signs requests with IAM credentials.
  • hyperion is a Scala library and a set of abstractions for AWS Data Pipeline.
  • dynq is a DynamoDB query library.
  • cloud-custodian is a policy rules engine for AWS management.

New SlideShare Presentations

New Customer Success Stories

New YouTube Videos

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

AWS Week in Review – March 28, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-march-28-2016/

Let’s take a quick look at what happened in AWS-land last week:

Monday

March 28

Tuesday

March 29

Wednesday

March  30

Thursday

March 31

Friday

April 1

Sunday

April 3

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

AWS Week in Review – March 14, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-march-14-2016/

Let’s take a quick look at what happened in AWS-land last week:

Monday
March 14

We announced that the Developer Preview of AWS SDK for C++ is Now Available.
We celebrated Ten Years in the AWS Cloud.
We launched Amazon EMR 4.4.0 with Sqoop, HCatalog, Java 8, and More.
The AWS Compute Blog announced the Launch of AWS Lambda and Amazon API Gateway in the EU (Frankfurt) Region.
The Amazon Simple Email Service Blog annouced that Amazon SES Now Supports Custom Email From Domains.
The AWS Java Blog talked about Using Amazon SQS with Spring Boot and Spring JMS.
The AWS Partner Network Blog urged you to Take Advantage of AWS Self-Paced Labs.
The AWS Windows and .NET Developer Blog showed you how to Retrieve Request Metrics from the AWS SDK for .NET.
The AWS Government, Education, & Nonprofits Blog announced the New Amazon-Busan Cloud Innovation and Technology Center.
We announced Lumberyard Beta 1.1 is Now Available.
Bometric shared AWS Security Best Practices: Network Security.
CloudCheckr listed 5 AWS Security Traps You Might be Missing.
Serverless Code announced that ServerlessConf is Here!
Cloud Academy launched 2 New AWS Courses – (Advanced Techniques for AWS Monitoring, Metrics and Logging and Advanced Deployment Techniques on AWS).
Cloudonaut reminded you to Avoid Sharing Key Pairs for EC2.
8KMiles talked about How Cloud Computing Can Address Healthcare Industry Challenges.
Evident discussed the CIS Foundations Benchmark for AWS Security.
Talkin’ Cloud shared 10 Facts About AWS as it Celebrates 10 Years.
The Next Platform reviewed Ten Years of AWS And a Status Check for HPC Clouds.
ZephyCloud is AWS-powered Wind Farm Design Software.

Tuesday
March 15

We announced the AWS Database Migration Service.
We announced that AWS CloudFormation Now Supports Amazon GameLift.
The AWS Partner Network Blog reminded everyone that Friends Don’t Let Friends Build Data Centers.
The Amazon GameDev Blog talked about Using Autoscaling to Control Costs While Delivering Great Player Experiences.
We updated the AWS SDK for JavaScript, the AWS SDK for Ruby, and the AWS SDK for Go.
Calorious talked about Uploading Images into Amazon S3.
Serverless Code showed you How to Use LXML in Lambda.
The Acquia Developer Center talked about Open-Sourcing Moonshot.
Concurrency Labs encouraged you to Hatch a Swarm of AWS IoT Things Using Locust, EC2 and Get Your IoT Application Ready for Prime Time.

Wednesday
March 16

We announced an S3 Lifecycle Management Update with Support for Multipart Upload and Delete Markers.
We announced that the EC2 Container Service is Now Available in the US West (Oregon) Region.
We announced that Amazon ElastiCache now supports the R3 node family in AWS China (Beijing) and AWS South America (Sao Paulo) Regions.
We announced that AWS IoT Now Integrates with Amazon Elasticsearch Service and CloudWatch.
We published the Puppet on the AWS Cloud: Quick Start Reference Deployment.
We announced that Amazon RDS Enhanced Monitoring is now available in the Asia Pacific (Seoul) Region.
I wrote about Additional Failover Control for Amazon Aurora (this feature was launched earlier in the year).
The AWS Security Blog showed you How to Set Up Uninterrupted, Federated User Access to AWS Using AD FS.
The AWS Java Blog talked about Migrating Your Databases Using AWS Database Migration Service.
We updated the AWS SDK for Java and the AWS CLI.
CloudWedge asked Cloud Computing: Cost Saver or Additional Expense?
Gathering Clouds reviewed New 2016 AWS Services: Certificate Manager, Lambda, Dev SecOps.

Thursday
March 17

We announced the new Marketplace Metering Service for 3rd Party Sellers.
We announced Amazon VPC Endpoints for Amazon S3 in South America (Sao Paulo) and Asia Pacific (Seoul).
We announced AWS CloudTrail Support for Kinesis Firehose.
The AWS Big Data Blog showed you How to Analyze a Time Series in Real Time with AWS Lambda, Amazon Kinesis and Amazon DynamoDB Streams.
The AWS Enterprise Blog showed you How to Create a Cloud Center of Excellence in your Enterprise, and then talked about Staffing Your Enterprise’s Cloud Center of Excellence.
The AWS Mobile Development Blog showed you How to Analyze Device-Generated Data with AWS IoT and Amazon Elasticsearch Service.
Stelligent initiated a series on Serverless Delivery.
CloudHealth Academy talked about Modeling RDS Reservations.
N2W Software talked about How to Pre-Warm Your EBS Volumes on AWS.
ParkMyCloud explained How to Save Money on AWS With ParkMyCloud.

Friday
March 18

The AWS Government, Education, & Nonprofits Blog told you how AWS GovCloud (US) Helps ASD Cut Costs by 50% While Dramatically Improving Security.
The Amazon GameDev Blog discussed Code Archeology: Crafting Lumberyard.
Calorious talked about Importing JSON into DynamoDB.
DZone Cloud Zone talked about Graceful Shutdown Using AWS AutoScaling Groups and Terraform.

Saturday
March 19

DZone Cloud Zone wants to honor some Trailblazing Women in the Cloud.

Sunday
March 20

 Cloudability talked about How Atlassian Nailed the Reserved Instance Buying Process.
DZone Cloud Zone talked about Serverless Delivery Architectures.
Gorillastack explained Why the Cloud is THE Key Technology Enabler for Digital Transformation.

New & Notable Open Source

Tumbless is a blogging platform based only on S3 and your browser.
aws-amicleaner cleans up old, unused AMIs and related snapshots.
alexa-aws-administration helps you to do various administration tasks in your AWS account using an Amazon Echo.
aws-s3-zipper takes an S3 bucket folder and zips it for streaming.
aws-lambda-helper is a collection of helper methods for Lambda.
CloudSeed lets you describe a list of AWS stack components, then configure and build a custom stack.
aws-ses-sns-dashboard is a Go-based dashboard with SES and SNS notifications.
snowplow-scala-analytics-sdk is a Scala SDK for working with Snowplow-enriched events in Spark using Lambda.
StackFormation is a lightweight CloudFormation stack manager.
aws-keychain-util is a command-line utility to manage AWS credentials in the OS X keychain.

New SlideShare Presentations

Account Separation and Mandatory Access Control on AWS.
Crypto Options in AWS.
Security Day IAM Recommended Practices.
What’s Nearly New.

New Customer Success Stories

AdiMap measures online advertising spend, app financials, and salary data. Using AWS, AdiMap builds predictive financial models without spending millions on compute resources and hardware, providing scalable financial intelligence and reducing time to market for new products.
Change.org is the world’s largest and fastest growing social change platform, with more than 125 million users in 196 countries starting campaigns and mobilizing support for local causes and global issues. The organization runs its website and business intelligence cluster on AWS, and runs its continuous integration and testing on Solano CI from APN member Solano Labs.
Flatiron Health has been able to reach 230 cancer clinics and 2,200 clinicians across the United States with a solution that captures and organizes oncology data, helping to support cancer treatments. Flatiron moved its solution to AWS to improve speed to market and to minimize the time and expense that the startup company needs to devote to its IT infrastructure.
Global Red specializes in lifecycle marketing, including strategy, data, analytics, and execution across all digital channels. By re-architecting and migrating its data platform and related applications to AWS, Global Red reduced the time to onboard new customers for its advertising trading desk and marketing automation platforms by 50 percent.
GMobi primarily sells its products and services to Original Design Manufacturers and Original Equipment Manufacturers in emerging markets. By running its “over the air” firmware updates, mobile billing, and advertising software development kits in an AWS infrastructure, GMobi has grown to support 120 million users while maintaining more than 99.9 percent availability
Time Inc.’s new chief technology officer joined the renowned media organization in early 2014, and promised big changes. With AWS, Time Inc. can leverage security features and functionality that mirror the benefits of cloud computing, including rich tools, best-in-class industry standards and protocols and lower costs.
Seaco Global is one of the world’s largest shipping companies. By using AWS to run SAP applications, it also reduced the time needed to complete monthly business processes to just one day, down from four days in the past.

New YouTube Videos

AWS Database Migration Service.
Introduction to Amazon WorkSpaces.
AWS Pop-up Loft.
Save the Date – AWS re:Invent 2016.

Upcoming Events

March 22nd – Live Event (Seattle, Washington) – AWS Big Data Meetup – Intro to SparkR.
March 22nd – Live Broadcast – VoiceOps: Commanding and Controlling Your AWS environments using Amazon Echo and Lambda.
March 23rd – Live Event (Atlanta, Georgia) – AWS Key Management Service & AWS Storage Services for a Hybrid Cloud (Atlanta AWS Community).
April 6th – Live Event (Boston, Massachusetts) AWS at Bio-IT World.
April 18th & 19th – Live Event (Chicago, Illinois) – AWS Summit – Chicago.
April 20th – Live Event (Melbourne, Australia) – Inaugural Melbourne Serverless Meetup.
April 26th – Live Event (Sydney, Australia) – AWS Partner Summit.
April 26th – Live Event (Sydney, Australia) – Inaugural Sydney Serverless Meetup.
ParkMyCloud 2016 AWS Cost-Reduction Roadshow.
AWS Loft – San Francisco.
AWS Loft – New York.
AWS Loft – Tel Aviv.
AWS Zombie Microservices Roadshow.
AWS Public Sector Events.
AWS Global Summit Series.

Help Wanted

AWS Careers.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.
Jeff;

AWS Week in Review – March 14, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-march-14-2016/

Let’s take a quick look at what happened in AWS-land last week:

Monday
March 14

We announced that the Developer Preview of AWS SDK for C++ is Now Available.
We celebrated Ten Years in the AWS Cloud.
We launched Amazon EMR 4.4.0 with Sqoop, HCatalog, Java 8, and More.
The AWS Compute Blog announced the Launch of AWS Lambda and Amazon API Gateway in the EU (Frankfurt) Region.
The Amazon Simple Email Service Blog annouced that Amazon SES Now Supports Custom Email From Domains.
The AWS Java Blog talked about Using Amazon SQS with Spring Boot and Spring JMS.
The AWS Partner Network Blog urged you to Take Advantage of AWS Self-Paced Labs.
The AWS Windows and .NET Developer Blog showed you how to Retrieve Request Metrics from the AWS SDK for .NET.
The AWS Government, Education, & Nonprofits Blog announced the New Amazon-Busan Cloud Innovation and Technology Center.
We announced Lumberyard Beta 1.1 is Now Available.
Bometric shared AWS Security Best Practices: Network Security.
CloudCheckr listed 5 AWS Security Traps You Might be Missing.
Serverless Code announced that ServerlessConf is Here!
Cloud Academy launched 2 New AWS Courses – (Advanced Techniques for AWS Monitoring, Metrics and Logging and Advanced Deployment Techniques on AWS).
Cloudonaut reminded you to Avoid Sharing Key Pairs for EC2.
8KMiles talked about How Cloud Computing Can Address Healthcare Industry Challenges.
Evident discussed the CIS Foundations Benchmark for AWS Security.
Talkin’ Cloud shared 10 Facts About AWS as it Celebrates 10 Years.
The Next Platform reviewed Ten Years of AWS And a Status Check for HPC Clouds.
ZephyCloud is AWS-powered Wind Farm Design Software.

Tuesday
March 15

We announced the AWS Database Migration Service.
We announced that AWS CloudFormation Now Supports Amazon GameLift.
The AWS Partner Network Blog reminded everyone that Friends Don’t Let Friends Build Data Centers.
The Amazon GameDev Blog talked about Using Autoscaling to Control Costs While Delivering Great Player Experiences.
We updated the AWS SDK for JavaScript, the AWS SDK for Ruby, and the AWS SDK for Go.
Calorious talked about Uploading Images into Amazon S3.
Serverless Code showed you How to Use LXML in Lambda.
The Acquia Developer Center talked about Open-Sourcing Moonshot.
Concurrency Labs encouraged you to Hatch a Swarm of AWS IoT Things Using Locust, EC2 and Get Your IoT Application Ready for Prime Time.

Wednesday
March 16

We announced an S3 Lifecycle Management Update with Support for Multipart Upload and Delete Markers.
We announced that the EC2 Container Service is Now Available in the US West (Oregon) Region.
We announced that Amazon ElastiCache now supports the R3 node family in AWS China (Beijing) and AWS South America (Sao Paulo) Regions.
We announced that AWS IoT Now Integrates with Amazon Elasticsearch Service and CloudWatch.
We published the Puppet on the AWS Cloud: Quick Start Reference Deployment.
We announced that Amazon RDS Enhanced Monitoring is now available in the Asia Pacific (Seoul) Region.
I wrote about Additional Failover Control for Amazon Aurora (this feature was launched earlier in the year).
The AWS Security Blog showed you How to Set Up Uninterrupted, Federated User Access to AWS Using AD FS.
The AWS Java Blog talked about Migrating Your Databases Using AWS Database Migration Service.
We updated the AWS SDK for Java and the AWS CLI.
CloudWedge asked Cloud Computing: Cost Saver or Additional Expense?
Gathering Clouds reviewed New 2016 AWS Services: Certificate Manager, Lambda, Dev SecOps.

Thursday
March 17

We announced the new Marketplace Metering Service for 3rd Party Sellers.
We announced Amazon VPC Endpoints for Amazon S3 in South America (Sao Paulo) and Asia Pacific (Seoul).
We announced AWS CloudTrail Support for Kinesis Firehose.
The AWS Big Data Blog showed you How to Analyze a Time Series in Real Time with AWS Lambda, Amazon Kinesis and Amazon DynamoDB Streams.
The AWS Enterprise Blog showed you How to Create a Cloud Center of Excellence in your Enterprise, and then talked about Staffing Your Enterprise’s Cloud Center of Excellence.
The AWS Mobile Development Blog showed you How to Analyze Device-Generated Data with AWS IoT and Amazon Elasticsearch Service.
Stelligent initiated a series on Serverless Delivery.
CloudHealth Academy talked about Modeling RDS Reservations.
N2W Software talked about How to Pre-Warm Your EBS Volumes on AWS.
ParkMyCloud explained How to Save Money on AWS With ParkMyCloud.

Friday
March 18

The AWS Government, Education, & Nonprofits Blog told you how AWS GovCloud (US) Helps ASD Cut Costs by 50% While Dramatically Improving Security.
The Amazon GameDev Blog discussed Code Archeology: Crafting Lumberyard.
Calorious talked about Importing JSON into DynamoDB.
DZone Cloud Zone talked about Graceful Shutdown Using AWS AutoScaling Groups and Terraform.

Saturday
March 19

DZone Cloud Zone wants to honor some Trailblazing Women in the Cloud.

Sunday
March 20

 Cloudability talked about How Atlassian Nailed the Reserved Instance Buying Process.
DZone Cloud Zone talked about Serverless Delivery Architectures.
Gorillastack explained Why the Cloud is THE Key Technology Enabler for Digital Transformation.

New & Notable Open Source

Tumbless is a blogging platform based only on S3 and your browser.
aws-amicleaner cleans up old, unused AMIs and related snapshots.
alexa-aws-administration helps you to do various administration tasks in your AWS account using an Amazon Echo.
aws-s3-zipper takes an S3 bucket folder and zips it for streaming.
aws-lambda-helper is a collection of helper methods for Lambda.
CloudSeed lets you describe a list of AWS stack components, then configure and build a custom stack.
aws-ses-sns-dashboard is a Go-based dashboard with SES and SNS notifications.
snowplow-scala-analytics-sdk is a Scala SDK for working with Snowplow-enriched events in Spark using Lambda.
StackFormation is a lightweight CloudFormation stack manager.
aws-keychain-util is a command-line utility to manage AWS credentials in the OS X keychain.

New SlideShare Presentations

Account Separation and Mandatory Access Control on AWS.
Crypto Options in AWS.
Security Day IAM Recommended Practices.
What’s Nearly New.

New Customer Success Stories

AdiMap measures online advertising spend, app financials, and salary data. Using AWS, AdiMap builds predictive financial models without spending millions on compute resources and hardware, providing scalable financial intelligence and reducing time to market for new products.
Change.org is the world’s largest and fastest growing social change platform, with more than 125 million users in 196 countries starting campaigns and mobilizing support for local causes and global issues. The organization runs its website and business intelligence cluster on AWS, and runs its continuous integration and testing on Solano CI from APN member Solano Labs.
Flatiron Health has been able to reach 230 cancer clinics and 2,200 clinicians across the United States with a solution that captures and organizes oncology data, helping to support cancer treatments. Flatiron moved its solution to AWS to improve speed to market and to minimize the time and expense that the startup company needs to devote to its IT infrastructure.
Global Red specializes in lifecycle marketing, including strategy, data, analytics, and execution across all digital channels. By re-architecting and migrating its data platform and related applications to AWS, Global Red reduced the time to onboard new customers for its advertising trading desk and marketing automation platforms by 50 percent.
GMobi primarily sells its products and services to Original Design Manufacturers and Original Equipment Manufacturers in emerging markets. By running its “over the air” firmware updates, mobile billing, and advertising software development kits in an AWS infrastructure, GMobi has grown to support 120 million users while maintaining more than 99.9 percent availability
Time Inc.’s new chief technology officer joined the renowned media organization in early 2014, and promised big changes. With AWS, Time Inc. can leverage security features and functionality that mirror the benefits of cloud computing, including rich tools, best-in-class industry standards and protocols and lower costs.
Seaco Global is one of the world’s largest shipping companies. By using AWS to run SAP applications, it also reduced the time needed to complete monthly business processes to just one day, down from four days in the past.

New YouTube Videos

AWS Database Migration Service.
Introduction to Amazon WorkSpaces.
AWS Pop-up Loft.
Save the Date – AWS re:Invent 2016.

Upcoming Events

March 22nd – Live Event (Seattle, Washington) – AWS Big Data Meetup – Intro to SparkR.
March 22nd – Live Broadcast – VoiceOps: Commanding and Controlling Your AWS environments using Amazon Echo and Lambda.
March 23rd – Live Event (Atlanta, Georgia) – AWS Key Management Service & AWS Storage Services for a Hybrid Cloud (Atlanta AWS Community).
April 6th – Live Event (Boston, Massachusetts) AWS at Bio-IT World.
April 18th & 19th – Live Event (Chicago, Illinois) – AWS Summit – Chicago.
April 20th – Live Event (Melbourne, Australia) – Inaugural Melbourne Serverless Meetup.
April 26th – Live Event (Sydney, Australia) – AWS Partner Summit.
April 26th – Live Event (Sydney, Australia) – Inaugural Sydney Serverless Meetup.
ParkMyCloud 2016 AWS Cost-Reduction Roadshow.
AWS Loft – San Francisco.
AWS Loft – New York.
AWS Loft – Tel Aviv.
AWS Zombie Microservices Roadshow.
AWS Public Sector Events.
AWS Global Summit Series.

Help Wanted

AWS Careers.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.
Jeff;

Additional Failover Control for Amazon Aurora

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/additional-failover-control-for-amazon-aurora/

Amazon Aurora is a fully-managed, MySQL-compatible, relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source database (read my post, Amazon Aurora – New Cost-Effective MySQL-Compatible Database Engine for Amazon RDS, to learn more).
Aurora allows you create up to 15 read replicas to increase read throughput and for use as failover targets. The replicas share storage with the primary instance and provide lightweight, fine-grained replication that is almost synchronous, with a replication delay on the order of 10 to 20 milliseconds.
Additional Failover Control Today we are making Aurora even more flexible by giving you control over the failover priority of each read replica. Each read replica is now associated with a priority tier (0-15).  In the event of a failover, Amazon RDS will promote the read replica that has the highest priority (the lowest numbered tier). If two or more replicas have the same priority, RDS will promote the one that is the same size as the previous primary instance.
You can set the priority when you create the Aurora DB instance:

This feature is available now and you can start using it today. To learn more, read about Fault Tolerance for an Aurora DB Cluster. —
Jeff;

Additional Failover Control for Amazon Aurora

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/additional-failover-control-for-amazon-aurora/

Amazon Aurora is a fully-managed, MySQL-compatible, relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source database (read my post, Amazon Aurora – New Cost-Effective MySQL-Compatible Database Engine for Amazon RDS, to learn more).
Aurora allows you create up to 15 read replicas to increase read throughput and for use as failover targets. The replicas share storage with the primary instance and provide lightweight, fine-grained replication that is almost synchronous, with a replication delay on the order of 10 to 20 milliseconds.
Additional Failover Control Today we are making Aurora even more flexible by giving you control over the failover priority of each read replica. Each read replica is now associated with a priority tier (0-15).  In the event of a failover, Amazon RDS will promote the read replica that has the highest priority (the lowest numbered tier). If two or more replicas have the same priority, RDS will promote the one that is the same size as the previous primary instance.
You can set the priority when you create the Aurora DB instance:

This feature is available now and you can start using it today. To learn more, read about Fault Tolerance for an Aurora DB Cluster. —
Jeff;