Amazon Aurora MySQL (Aurora MySQL) is a managed relational database engine, wire-compatible with MySQL 5.6. Most of the drivers, connectors, and tools that you currently use with MySQL can be used with Aurora MySQL with little or no change.
Aurora MySQL database (DB) clusters provide advanced features such as:
One primary instance that supports read and write operations and up to 15 Aurora Replicas that support read-only operations, and can be automatically promoted to the primary role if the current primary instance fails
A cluster endpoint that automatically follows the primary instance in case of failover
A reader endpoint that includes all Aurora Replicas and is automatically updated when Aurora Replicas are added or removed
Internal server connection pooling and thread multiplexing for improved scalability
Near-instantaneous database restarts and crash recovery
Access to near-real-time cluster metadata that allows application developers to build “smart drivers,” connecting directly to individual instances based on their read-write or read-only role
Client-side components that are not well configured (applications, drivers, connectors, proxies) may be unable to react to recovery actions and DB cluster topology changes, or the reaction may be delayed, contributing to unexpected downtime and performance issues. In order to prevent that and make the most of Aurora MySQL features, database administrators (DBAs) and application developers are encouraged to implement best practices.
You can read more about these best practices in the recently-published Amazon Aurora MySQL DBA Handbook – Connection Management whitepaper.
About the Author
Szymon Komendera is a database engineer on the Amazon Aurora team.
As ransomware attacks have grown in number in recent months, the tactics and attack vectors also have evolved. While the primary method of attack used to be to target individual computer users within organizations with phishing emails and infected attachments, we’re increasingly seeing attacks that target weaknesses in businesses’ IT infrastructure.
How Ransomware Attacks Typically Work
In our previous posts on ransomware, we described the common vehicles used by hackers to infect organizations with ransomware viruses. Most often, downloaders distribute trojan horses through malicious downloads and spam emails. The emails contain a variety of file attachments, which if opened, will download and run one of the many ransomware variants. Once a user’s computer is infected with a malicious downloader, it will retrieve additional malware, which frequently includes crypto-ransomware. After the files have been encrypted, a ransom payment is demanded of the victim in order to decrypt the files.
What’s Changed With the Latest Ransomware Attacks?
In 2016, a customized ransomware strain called SamSam began attacking the servers in primarily health care institutions. SamSam, unlike more conventional ransomware, is not delivered through downloads or phishing emails. Instead, the attackers behind SamSam use tools to identify unpatched servers running Red Hat’s JBoss enterprise products. Once the attackers have successfully gained entry into one of these servers by exploiting vulnerabilities in JBoss, they use other freely available tools and scripts to collect credentials and gather information on networked computers. Then they deploy their ransomware to encrypt files on these systems before demanding a ransom. Gaining entry to an organization through its IT center rather than its endpoints makes this approach scalable and especially unsettling.
SamSam’s methodology is to scour the Internet searching for accessible and vulnerable JBoss application servers, especially ones used by hospitals. It’s not unlike a burglar rattling doorknobs in a neighborhood to find unlocked homes. When SamSam finds an unlocked home (unpatched server), the software infiltrates the system. It is then free to spread across the company’s network by stealing passwords. As it transverses the network and systems, it encrypts files, preventing access until the victims pay the hackers a ransom, typically between $10,000 and $15,000. The low ransom amount has encouraged some victimized organizations to pay the ransom rather than incur the downtime required to wipe and reinitialize their IT systems.
The success of SamSam is due to its effectiveness rather than its sophistication. SamSam can enter and transverse a network without human intervention. Some organizations are learning too late that securing internet-facing services in their data center from attack is just as important as securing endpoints.
The typical steps in a SamSam ransomware attack are:
1 Attackers gain access to vulnerable server
Attackers exploit vulnerable software or weak/stolen credentials.
2 Attack spreads via remote access tools
Attackers harvest credentials, create SOCKS proxies to tunnel traffic, and abuse RDP to install SamSam on more computers in the network.
3 Ransomware payload deployed
Attackers run batch scripts to execute ransomware on compromised machines.
4 Ransomware demand delivered requiring payment to decrypt files
Demand amounts vary from victim to victim. Relatively low ransom amounts appear to be designed to encourage quick payment decisions.
What all the organizations successfully exploited by SamSam have in common is that they were running unpatched servers that made them vulnerable to SamSam. Some organizations had their endpoints and servers backed up, while others did not. Some of those without backups they could use to recover their systems chose to pay the ransom money.
Timeline of SamSam History and Exploits
Since its appearance in 2016, SamSam has been in the news with many successful incursions into healthcare, business, and government institutions.
March 2016 SamSam appears
SamSam campaign targets vulnerable JBoss servers Attackers hone in on healthcare organizations specifically, as they’re more likely to have unpatched JBoss machines.
April 2016 SamSam finds new targets
SamSam begins targeting schools and government. After initial success targeting healthcare, attackers branch out to other sectors.
April 2017 New tactics include RDP
Attackers shift to targeting organizations with exposed RDP connections, and maintain focus on healthcare. An attack on Erie County Medical Center costs the hospital $10 million over three months of recovery.
January 2018 Municipalities attacked
• Attack on Municipality of Farmington, NM. • Attack on Hancock Health. • Attack on Adams Memorial Hospital • Attack on Allscripts (Electronic Health Records), which includes 180,000 physicians, 2,500 hospitals, and 7.2 million patients’ health records.
February 2018 Attack volume increases
• Attack on Davidson County, NC. • Attack on Colorado Department of Transportation.
March 2018 SamSam shuts down Atlanta
• Second attack on Colorado Department of Transportation. • City of Atlanta suffers a devastating attack by SamSam. The attack has far-reaching impacts — crippling the court system, keeping residents from paying their water bills, limiting vital communications like sewer infrastructure requests, and pushing the Atlanta Police Department to file paper reports. • SamSam campaign nets $325,000 in 4 weeks. Infections spike as attackers launch new campaigns. Healthcare and government organizations are once again the primary targets.
How to Defend Against SamSam and Other Ransomware Attacks
The best way to respond to a ransomware attack is to avoid having one in the first place. If you are attacked, making sure your valuable data is backed up and unreachable by ransomware infection will ensure that your downtime and data loss will be minimal or none if you ever suffer an attack.
In our previous post, How to Recover From Ransomware, we listed the ten ways to protect your organization from ransomware.
Use anti-virus and anti-malware software or other security policies to block known payloads from launching.
Make frequent, comprehensive backups of all important files and isolate them from local and open networks. Cybersecurity professionals view data backup and recovery (74% in a recent survey) by far as the most effective solution to respond to a successful ransomware attack.
Keep offline backups of data stored in locations inaccessible from any potentially infected computer, such as disconnected external storage drives or the cloud, which prevents them from being accessed by the ransomware.
Install the latest security updates issued by software vendors of your OS and applications. Remember to patch early and patch often to close known vulnerabilities in operating systems, server software, browsers, and web plugins.
Consider deploying security software to protect endpoints, email servers, and network systems from infection.
Exercise cyber hygiene, such as using caution when opening email attachments and links.
Segment your networks to keep critical computers isolated and to prevent the spread of malware in case of attack. Turn off unneeded network shares.
Turn off admin rights for users who don’t require them. Give users the lowest system permissions they need to do their work.
Restrict write permissions on file servers as much as possible.
Educate yourself, your employees, and your family in best practices to keep malware out of your systems. Update everyone on the latest email phishing scams and human engineering aimed at turning victims into abettors.
Please Tell Us About Your Experiences with Ransomware
Have you endured a ransomware attack or have a strategy to avoid becoming a victim? Please tell us of your experiences in the comments.
Backblaze just returned from exhibiting at NAB in Las Vegas, April 9-12, where the response to our recent announcements was tremendous. In case you missed the news, Backblaze B2 Cloud Storage continues to extend its lead as the most affordable, high performance cloud on the planet.
Our booth was busy from start to finish with attendees interested in learning more about Backblaze and B2 Cloud Storage. Here are the questions we were asked most often in the booth.
Q. How long has Backblaze been in business? A. The company was founded in 2007. Today, we have over 500 petabytes of data from customers in over 150 countries.
Q. Where is your data stored? A. We have data centers in California and Arizona and expect to expand to Europe by the end of the year.
Q. How can your services be so inexpensive? A. Backblaze’s goal from the beginning was to offer cloud backup and storage that was easy to use and affordable. All the existing options were simply too expensive to be viable, so we created our own infrastructure. Our purpose-built storage system — the Backblaze’s Storage Pod — is recognized as one of the most cost efficient storage platforms available.
Q. Tell me about your hardware. A. Backblaze’s Storage Pods hold 60 HDDs each, containing as much as 720TB data per pod, stored using Reed-Solomon error correction. Storage Pods are arranged in Tomes with twenty Storage Pods making up a Vault.
Q. Where do you fit in the data workflow? A. People typically use B2 in for archiving completed projects. All data is readily available for download from B2, making it more convenient than off-line storage. In addition, DAM and MAM systems such as CatDV, axle ai, Cantemo, and others have integrated with B2 to store raw images behind the proxies.
Q. Who uses B2 in the M&E business? A. KLRU-TV, the PBS station in Austin Texas, uses B2 to archive their entire 43 year catalog of Austin City Limits episodes and related materials. WunderVu, the production house for Pixvana, uses B2 to back up and archive their local storage systems on which they build virtual reality experiences for their customers.
Q. You’re the company that publishes the hard drive stats, right? A. Yes, we are!
Were You at NAB?
If you were, we hope you stopped by the Backblaze booth to say hello. We’d like to hear what you saw at the show that was interesting or exciting. Please tell us in the comments.
Data that describe processes in a spatial context are everywhere in our day-to-day lives and they dominate big data problems. Map data, for instance, whether describing networks of roads or remote sensing data from satellites, get us where we need to go. Atmospheric data from simulations and sensors underlie our weather forecasts and climate models. Devices and sensors with GPS can provide a spatial context to nearly all mobile data.
In this post, we introduce the WIND toolkit, a huge (500 TB), open weather model dataset that’s available to the world on Amazon’s cloud services. We walk through how to access this data and some of the open-source software developed to make it easily accessible. Our solution considers a subset of geospatial data that exist on a grid (raster) and explores ways to provide access to large-scale raster data from weather models. The solution uses foundational AWS services and the Hierarchical Data Format (HDF), a well adopted format for scientific data.
The approach developed here can be extended to any data that fit in an HDF5 file, which can describe sparse and dense vectors and matrices of arbitrary dimensions. This format is already popular within the physical sciences for both experimental and simulation data. We discuss solutions to gridded data storage for a massive dataset of public weather model outputs called the Wind Integration National Dataset (WIND) toolkit. We also highlight strategies that are general to other large geospatial data management problems.
Wind Integration National Dataset
As variable renewable power penetration levels increase in power systems worldwide, the importance of renewable integration studies to ensure continued economic and reliable operation of the power grid is also increasing. The WIND toolkit is the largest freely available grid integration dataset to date.
The WIND toolkit was developed by 3TIER by Vaisala. They were under a subcontract to the National Renewable Energy Laboratory (NREL) to support studies on integration of wind energy into the existing US grid. NREL is a part of a network of national laboratories for the US Department of Energy and has a mission to advance the science and engineering of energy efficiency, sustainable transportation, and renewable power technologies.
The toolkit has been used by consultants, research groups, and universities worldwide to support grid integration studies. Less traditional uses also include resource assessments for wind plants (such as those powering Amazon data centers), and studying the effects of weather on California condor migrations in the Baja peninsula.
The diversity of applications highlights the value of accessible, open public data. Yet, there’s a catch: the dataset is huge. The WIND toolkit provides simulated atmospheric (weather) data at a two-km spatial resolution and five-minute temporal resolution at multiple heights for seven years. The entire dataset is half a petabyte (500 TB) in size and is stored in the NREL High Performance Computing data center in Golden, Colorado. Making this dataset publicly available easily and in a cost-effective manner is a major challenge.
As other laboratories and public institutions work to release their data to the world, they may face similar challenges to those that we experienced. Some prior, well-intentioned efforts to release huge datasets as-is have resulted in data resources that are technically available but fundamentally unusable. They may be stored in an unintuitive format or indexed and organized to support only a subset of potential uses. Downloading hundreds of terabytes of data is often impractical. Most users don’t have access to a big data cluster (or super computer) to slice and dice the data as they need after it’s downloaded.
We aim to provide a large amount of data (50 terabytes) to the public in a way that is efficient, scalable, and easy to use. In many cases, researchers can access these huge cloud-located datasets using the same software and algorithms they have developed for smaller datasets stored locally. Only the pieces of data they need for their individual analysis must be downloaded. To make this work in practice, we worked with the HDF Group and have built upon their forthcoming Highly Scalable Data Service.
In the rest of this post, we discuss how the HSDS software was developed to use Amazon EC2 and Amazon S3 resources to provide convenient and scalable access to these huge geospatial datasets. We describe how the HSDS service has been put to work for the WIND Toolkit dataset and demonstrate how to access it using the h5pyd Python library and the REST API. We conclude with information about our ongoing work to release more ‘open’ datasets to the public using AWS services, and ways to improve and extend the HSDS with newer Amazon services like Amazon ECS and AWS Lambda.
Developing a scalable service for big geospatial data
The HDF5 file format and API have been used for many years and is an effective means of storing large scientific datasets. For example, NASA’s Earth Observing System (EOS) satellites collect more than 16 TBs of data per day using HDF5.
With the rise of the cloud, there are new challenges and opportunities to rethink how HDF5 can be enhanced to work effectively as a component in a cloud-native architecture. For the HDF Group, working with NREL has been a great opportunity to put ideas into practice with a production-size dataset.
An HDF5 file consists of a directed graph of group and dataset objects. Datasets can be thought of as a multidimensional array with support for user-defined metadata tags and compression. Typical operations on datasets would be reading or writing data to a regular subregion (a hyperslab) or reading and writing individual elements (a point selection). Also, group and dataset objects may each contain an arbitrary number of the user-defined metadata elements known as attributes.
Many people have used the HDF library in applications developed or ported to run on EC2 instances, but there are a number of constraints that often prove problematic:
The HDF5 library can’t read directly from HDF5 files stored as S3 objects. The entire file (often many GB in size) would need to be copied to local storage before the first byte can be read. Also, the instance must be configured with the appropriately sized EBS volume)
The HDF library only has access to the computational resources of the instance itself (as opposed to a cluster of instances), so many operations are bottlenecked by the library.
Any modifications to the HDF5 file would somehow have to be synchronized with changes that other instances have made to same file before writing back to S3.
Using a pattern common to many offerings from AWS, the solution to these constraints is to develop a service framework around the HDF data model. Using this model, the HDF Group has created the Highly Scalable Data Service (HSDS) that provides all the functionality that traditionally was provided by the HDF5 library. By using the service, you don’t need to manage your own file volumes, but can just read and write whatever data that you need.
Because the service manages the actual data persistence to a durable medium (S3, in this case), you don’t need to worry about disk management. Simply stream the data you need from the service as you need it. Secondly, putting the functionality behind a service allows some tricks to increase performance (described in more detail later). And lastly, HSDS allows any number of clients to access the data at the same time, enabling HDF5 to be used as a coordination mechanism for multiple readers and writers.
In designing the HSDS architecture, we gave much thought to how to achieve scalability of the HSDS service. For accessing HDF5 data, there are two different types of scaling to consider:
Multiple clients making many requests to the service
Single requests that require a significant amount of data processing
To deal with the first scaling challenge, as with most services, we considered how the service responds as the request rate increases. AWS provides some great tools that help in this regard:
Auto Scaling groups
Elastic Load Balancing load balancers
The ability of S3 to handle large aggregate throughput rates
By using a cluster of EC2 instances behind a load balancer, you can handle different client loads in a cost-effective manner.
The second scaling challenge concerns single requests that would take significant processing time with just one compute node. One example of this from the WIND toolkit would be extracting all the values in the seven-year time span for a given geographic point and dataset.
In HDF5, large datasets are typically stored as “chunks”; that is, a regular partition of the array. In HSDS, each chunk is stored as a binary object in S3. The sequential approach to retrieving the time series values would be for the service to read each chunk needed from S3, extract the needed elements, and go on to the next chunk. In this case, that would involve processing 2557 chunks, and would be quite slow.
Fortunately, with HSDS, you can speed this up quite a bit by exploiting the compute and I/O capabilities of the cluster. Upon receiving the request, the receiving node can use other nodes in the cluster to read different portions of the selection. With multiple nodes reading from S3 in parallel, performance improves as the cluster size increases.
The diagram below illustrates how this works in simplified case of four chunks and four nodes.
This architecture has worked in well in practice. In testing with the WIND toolkit and time series extraction, we observed a request latency of ~60 seconds using four nodes vs. ~5 seconds with 40 nodes. Performance roughly scales with the size of the cluster.
A planned enhancement to this is to use AWS Lambda for the worker processing. This enables 1000-way parallel reads at a reasonable cost, as you only pay for the milliseconds of CPU time used with AWS Lambda.
Public access to atmospheric data using HSDS and AWS
An early challenge in releasing the WIND toolkit data was in deciding how to subset the data for different use cases. In general, few researchers need access to the entire 0.5 PB of data and a great deal of efficiency and cost reduction can be gained by making directed constituent datasets.
NREL grid integration researchers initially extracted a 2-TB subset by selecting 120,000 points where the wind resource seemed appropriate for development. They also chose only those data important for wind applications (100-m wind speed, converted to power), the most interesting locations for those performing grid studies. To support the remaining users who needed more data resolution, we down-sampled the data to a 60-minute temporal resolution, keeping all the other variables and spatial resolution intact. This reduced dataset is 50 TB of data describing 30+ atmospheric variables of data for 7 years at a 60-minute temporal resolution.
The WindViz browser-based Gridded Wind Toolkit Visualizer was created as an example implementation of the HSDS REST API in JavaScript. The visualizer is written in the style of ECMAScript 2016 using a modern development toolchain that includes webpack and Babel. The source code is available through our GitHub repository. The demo page is hosted via GitHub pages, and we use a cross-origin AJAX request to fetch data from the HSDS service running on the EC2 infrastructure. The visualizer can be used to explore the gridded wind toolkit data on a map. Achieve full spatial resolution by zooming in to a specific region.
Programmatic access is possible using the h5pyd Python library, a distributed analog to the widely used h5py library. Users interact with the datasets (variables) and slice the data from its (time x longitude x latitude) cube form as they see fit.
Examples and use cases are described in a set of Jupyter notebooks and available on GitHub:
Now you have a Jupyter notebook server running on your EC2 server.
From your laptop, create an SSH tunnel:
$ ssh –L 8888:localhost:8888 (IP address of the EC2 server)
Now, you can browse to localhost:8888 using the correct token, and interact with the notebooks as if they were local. Within the directory, there are examples for accessing the HSDS API and plotting wind and weather data using matplotlib.
Controlling access and defraying costs
A final concern is rate limiting and access control. Although the HSDS service is scalable and relatively robust, we had a few practical concerns:
How can we protect from malicious or accidental use that may lead to high egress fees (for example, someone who attempts to repeatedly download the entire dataset from S3)?
How can we keep track of who is using the data both to document the value of the data resource and to justify the costs?
If costs become too high, can we charge for some or all API use to help cover the costs?
To approach these problems, we investigated using Amazon API Gateway and its simplified integration with the AWS Marketplace for SaaS monetization as well as third-party API proxies.
In the end, we chose to use API Umbrella due to its close involvement with http://data.gov. While AWS Marketplace is a compelling option for future datasets, the decision was made to keep this dataset entirely open, at least for now. As community use and associated costs grow, we’ll likely revisit Marketplace. Meanwhile, API Umbrella provides controls for rate limiting and API key registration out of the box and was simple to implement as a front-end proxy to HSDS. Those applications that may want to charge for API use can accomplish a similar strategy using Amazon API Gateway and AWS Marketplace.
Ongoing work and other resources
As NREL and other government research labs, municipalities, and organizations try to share data with the public, we expect many of you will face similar challenges to those we have tried to approach with the architecture described in this post. Providing large datasets is one challenge. Doing so in a way that is affordable and convenient for users is an entirely more difficult goal. Using AWS cloud-native services and the existing foundation of the HDF file format has allowed us to tackle that challenge in a meaningful way.
Dr. Caleb Phillips is a senior scientist with the Data Analysis and Visualization Group within the Computational Sciences Center at the National Renewable Energy Laboratory. Caleb comes from a background in computer science systems, applied statistics, computational modeling, and optimization. His work at NREL spans the breadth of renewable energy technologies and focuses on applying modern data science techniques to data problems at scale.
Dr. Caroline Draxl is a senior scientist at NREL. She supports the research and modeling activities of the US Department of Energy from mesoscale to wind plant scale. Caroline uses mesoscale models to research wind resources in various countries, and participates in on- and offshore boundary layer research and in the coupling of the mesoscale flow features (kilometer scale) to the microscale (tens of meters). She holds a M.S. degree in Meteorology and Geophysics from the University of Innsbruck, Austria, and a PhD in Meteorology from the Technical University of Denmark.
John Readey has been a Senior Architect at The HDF Group since he joined in June 2014. His interests include web services related to HDF, applications that support the use of HDF and data visualization.Before joining The HDF Group, John worked at Amazon.com from 2006–2014 where he developed service-based systems for eCommerce and AWS.
Jordan Perr-Sauer is an RPP intern with the Data Analysis and Visualization Group within the Computational Sciences Center at the National Renewable Energy Laboratory. Jordan hopes to use his professional background in software engineering and his academic training in applied mathematics to solve the challenging problems facing America and the world.
AWS Fargate is a technology that allows you to focus on running your application without needing to provision, monitor, or manage the underlying compute infrastructure. You package your application into a Docker container that you can then launch using your container orchestration tool of choice.
Fargate allows you to use containers without being responsible for Amazon EC2 instances, similar to how EC2 allows you to run VMs without managing physical infrastructure. Currently, Fargate provides support for Amazon Elastic Container Service (Amazon ECS). Support for Amazon Elastic Container Service for Kubernetes (Amazon EKS) will be made available in the near future.
Despite offloading the responsibility for the underlying instances, Fargate still gives you deep control over configuration of network placement and policies. This includes the ability to use many networking fundamentals such as Amazon VPC and security groups.
This post covers how to take advantage of the different ways of networking your containers in Fargate when using ECS as your orchestration platform, with a focus on how to do networking securely.
The first step to running any application in Fargate is defining an ECS task for Fargate to launch. A task is a logical group of one or more Docker containers that are deployed with specified settings. When running a task in Fargate, there are two different forms of networking to consider:
Container (local) networking
External networking
Container Networking
Container networking is often used for tightly coupled application components. Perhaps your application has a web tier that is responsible for serving static content as well as generating some dynamic HTML pages. To generate these dynamic pages, it has to fetch information from another application component that has an HTTP API.
One potential architecture for such an application is to deploy the web tier and the API tier together as a pair and use local networking so the web tier can fetch information from the API tier.
If you are running these two components as two processes on a single EC2 instance, the web tier application process could communicate with the API process on the same machine by using the local loopback interface. The local loopback interface has a special IP address of 127.0.0.1 and hostname of localhost.
By making a networking request to this local interface, it bypasses the network interface hardware and instead the operating system just routes network calls from one process to the other directly. This gives the web tier a fast and efficient way to fetch information from the API tier with almost no networking latency.
In Fargate, when you launch multiple containers as part of a single task, they can also communicate with each other over the local loopback interface. Fargate uses a special container networking mode called awsvpc, which gives all the containers in a task a shared elastic network interface to use for communication.
If you specify a port mapping for each container in the task, then the containers can communicate with each other on that port. For example the following task definition could be used to deploy the web tier and the API tier:
ECS, with Fargate, is able to take this definition and launch two containers, each of which is bound to a specific static port on the elastic network interface for the task.
Because each Fargate task has its own isolated networking stack, there is no need for dynamic ports to avoid port conflicts between different tasks as in other networking modes. The static ports make it easy for containers to communicate with each other. For example, the web container makes a request to the API container using its well-known static port:
curl 127.0.0.1:8080/my-endpoint
This sends a local network request, which goes directly from one container to the other over the local loopback interface without traversing the network. This deployment strategy allows for fast and efficient communication between two tightly coupled containers. But most application architectures require more than just internal local networking.
External Networking
External networking is used for network communications that go outside the task to other servers that are not part of the task, or network communications that originate from other hosts on the internet and are directed to the task.
Configuring external networking for a task is done by modifying the settings of the VPC in which you launch your tasks. A VPC is a fundamental tool in AWS for controlling the networking capabilities of resources that you launch on your account.
When setting up a VPC, you create one or more subnets, which are logical groups that your resources can be placed into. Each subnet has an Availability Zone and its own route table, which defines rules about how network traffic operates for that subnet. There are two main types of subnets: public and private.
Public subnets
A public subnet is a subnet that has an associated internet gateway. Fargate tasks in that subnet are assigned both private and public IP addresses:
A browser or other client on the internet can send network traffic to the task via the internet gateway using its public IP address. The tasks can also send network traffic to other servers on the internet because the route table can route traffic out via the internet gateway.
If tasks want to communicate directly with each other, they can use each other’s private IP address to send traffic directly from one to the other so that it stays inside the subnet without going out to the internet gateway and back in.
Private subnets
A private subnet does not have direct internet access. The Fargate tasks inside the subnet don’t have public IP addresses, only private IP addresses. Instead of an internet gateway, a network address translation (NAT) gateway is attached to the subnet:
There is no way for another server or client on the internet to reach your tasks directly, because they don’t even have an address or a direct route to reach them. This is a great way to add another layer of protection for internal tasks that handle sensitive data. Those tasks are protected and can’t receive any inbound traffic at all.
In this configuration, the tasks can still communicate to other servers on the internet via the NAT gateway. They would appear to have the IP address of the NAT gateway to the recipient of the communication. If you run a Fargate task in a private subnet, you must add this NAT gateway. Otherwise, Fargate can’t make a network request to Amazon ECR to download the container image, or communicate with Amazon CloudWatch to store container metrics.
Load balancers
If you are running a container that is hosting internet content in a private subnet, you need a way for traffic from the public to reach the container. This is generally accomplished by using a load balancer such as an Application Load Balancer or a Network Load Balancer.
ECS integrates tightly with AWS load balancers by automatically configuring a service-linked load balancer to send network traffic to containers that are part of the service. When each task starts, the IP address of its elastic network interface is added to the load balancer’s configuration. When the task is being shut down, network traffic is safely drained from the task before removal from the load balancer.
To get internet traffic to containers using a load balancer, the load balancer is placed into a public subnet. ECS configures the load balancer to forward traffic to the container tasks in the private subnet:
This configuration allows your tasks in Fargate to be safely isolated from the rest of the internet. They can still initiate network communication with external resources via the NAT gateway, and still receive traffic from the public via the Application Load Balancer that is in the public subnet.
Another potential use case for a load balancer is for internal communication from one service to another service within the private subnet. This is typically used for a microservice deployment, in which one service such as an internet user account service needs to communicate with an internal service such as a password service. Obviously, it is undesirable for the password service to be directly accessible on the internet, so using an internet load balancer would be a major security vulnerability. Instead, this can be accomplished by hosting an internal load balancer within the private subnet:
With this approach, one container can distribute requests across an Auto Scaling group of other private containers via the internal load balancer, ensuring that the network traffic stays safely protected within the private subnet.
Best Practices for Fargate Networking
Determine whether you should use local task networking
Local task networking is ideal for communicating between containers that are tightly coupled and require maximum networking performance between them. However, when you deploy one or more containers as part of the same task they are always deployed together so it removes the ability to independently scale different types of workload up and down.
In the example of the application with a web tier and an API tier, it may be the case that powering the application requires only two web tier containers but 10 API tier containers. If local container networking is used between these two container types, then an extra eight unnecessary web tier containers would end up being run instead of allowing the two different services to scale independently.
A better approach would be to deploy the two containers as two different services, each with its own load balancer. This allows clients to communicate with the two web containers via the web service’s load balancer. The web service could distribute requests across the eight backend API containers via the API service’s load balancer.
Run internet tasks that require internet access in a public subnet
If you have tasks that require internet access and a lot of bandwidth for communication with other services, it is best to run them in a public subnet. Give them public IP addresses so that each task can communicate with other services directly.
If you run these tasks in a private subnet, then all their outbound traffic has to go through an NAT gateway. AWS NAT gateways support up to 10 Gbps of burst bandwidth. If your bandwidth requirements go over this, then all task networking starts to get throttled. To avoid this, you could distribute the tasks across multiple private subnets, each with their own NAT gateway. It can be easier to just place the tasks into a public subnet, if possible.
Avoid using a public subnet or public IP addresses for private, internal tasks
If you are running a service that handles private, internal information, you should not put it into a public subnet or use a public IP address. For example, imagine that you have one task, which is an API gateway for authentication and access control. You have another background worker task that handles sensitive information.
The intended access pattern is that requests from the public go to the API gateway, which then proxies request to the background task only if the request is from an authenticated user. If the background task is in a public subnet and has a public IP address, then it could be possible for an attacker to bypass the API gateway entirely. They could communicate directly to the background task using its public IP address, without being authenticated.
Conclusion
Fargate gives you a way to run containerized tasks directly without managing any EC2 instances, but you still have full control over how you want networking to work. You can set up containers to talk to each other over the local network interface for maximum speed and efficiency. For running workloads that require privacy and security, use a private subnet with public internet access locked down. Or, for simplicity with an internet workload, you can just use a public subnet and give your containers a public IP address.
To deploy one of these Fargate task networking approaches, check out some sample CloudFormation templates showing how to configure the VPC, subnets, and load balancers.
If you have questions or suggestions, please comment below.
Previously, applications running inside a VPC required internet access to connect to AWS KMS. This meant managing internet connectivity through internet gateways, Network Address Translation (NAT) devices, or firewall proxies. With support for Amazon VPC endpoints, you can now keep all traffic between your VPC and AWS KMS within the AWS network and avoid management of internet connectivity. In this blog post, I show you how to create and use an Amazon VPC endpoint for AWS KMS, audit the use of AWS KMS keys through the Amazon VPC endpoint, and build stricter access controls using key policies.
Create and use an Amazon VPC endpoint with AWS KMS
To get started, I will show you how to use the Amazon VPC console to create an endpoint in the US East (N. Virginia) Region, also known as us-east-1.
To create an endpoint in the US East (N. Virginia) Region:
Navigate to the Amazon VPC console. In the navigation pane, choose Endpoints, and then choose Create Endpoint.
Choose AWS services for Service category.
Choose the AWS KMS endpoint service, com.amazonaws.us-east-1.kms, from the Service Name list, as shown in the following screenshot.
Your VPC endpoint can span multiple Availability Zones, providing isolation and fault tolerance. Choose a subnet from each Availability Zone from which you want to connect. An elastic network interface for the VPC endpoint is created in each subnet that you choose, each with its own DNS hostname and private IP address.
If your VPC has DNS hostnames and DNS support enabled, choose Enable for this endpoint under Enable Private DNS Name to have applications use the VPC endpoint by default.
You use security groups to control access to your endpoint. Choose a security group from the list, or create a new one.
To finish creating the endpoint, choose Create endpoint. The console returns a VPC Endpoint ID. In our example, the VPC Endpoint ID is vpce-0c0052e3fbffdb450.
To connect to this endpoint, you need a DNS hostname that is generated for this endpoint. You can view these DNS hostnames by choosing the VPC Endpoint ID and then choosing the Details tab of the endpoint in the Amazon VPC console. One of the DNS hostnames for the endpoint that I created in the previous step is vpce-0c0052e3fbffdb450-afmosqu8.kms.us-east-1.vpce.amazonaws.com.
You can connect to AWS KMS through the VPC endpoint by using the AWS CLI or an AWS SDK. In this example, I use the following AWS CLI command to list all AWS KMS keys in the account in us-east-1.
If your VPC has DNS hostnames and DNS support enabled and you enabled private DNS names in the preceding steps, you can connect to your VPC endpoint by using the standard AWS KMS DNS hostname (https://kms.<region>.amazonaws.com), instead of manually configuring the endpoints in the AWS CLI or AWS SDKs. The AWS CLI and SDKs use this hostname by default to connect to KMS, so there’s nothing to change in your application to begin using the VPC endpoint.
You can monitor and audit AWS KMS usage through your VPC endpoint. Every request made to AWS KMS is logged by AWS CloudTrail. Now, when you use a VPC endpoint to make requests to AWS KMS, the endpoint ID appears in the CloudTrail log entries.
Restrict access using key policies
A good security practice to follow is least privilege: granting the fewest permissions required to complete a task. You can control access to your AWS KMS keys from a specific VPC endpoint by using AWS KMS key policies and AWS Identity and Access Management (IAM) policies. The aws:sourceVpce condition key lets you grant or restrict access to AWS KMS keys based on the VPC endpoint used. For example, the following example key policy allows a user to perform encryption operations with a key only when the request comes through the specified VPC endpoint (replace the placeholder AWS account ID with your own account ID, and the placeholder VPC endpoint ID with your own endpoint ID).
This policy works by including a Deny statement with a StringNotEquals condition. When a user makes a request to AWS KMS through a VPC endpoint, the endpoint’s ID is compared to the aws:sourceVpce value specified in the policy. If the two values are not the same, the request is denied. You can modify AWS KMS key policies in the AWS KMS console. For more information, see Modifying a Key Policy.
You also can control access to your AWS KMS keys from any endpoint running in one or more VPCs by using the aws:sourceVpc policy condition key. Suppose you have an application that is running in one VPC, but uses a second VPC for resource management functions. In the following example policy, AWS KMS key administrative actions can only be made from VPC vpc-12345678, and the key can only be used for cryptographic operations from VPC vpc-2b2b2b2b.
The previous examples show how you can limit access to AWS KMS API actions that are attached to a key policy. If you want to limit access to AWS KMS API actions that are not attached to a specific key, you have to use these VPC-related conditions in an IAM policy that refers to the desired AWS KMS API actions.
Summary
In this post, I have demonstrated how to create and use a VPC endpoint for AWS KMS, and how to use the aws:sourceVpc and aws:sourceVpce policy conditions to scope permissions to call various AWS KMS APIs. AWS KMS VPC endpoints provide you with more control over how your applications connect to AWS KMS and can save you from managing internet connectivity from your VPC.
To learn more about connecting to AWS KMS through a VPC endpoint, see the AWS KMS Developer Guide. For helpful guidance about your overall VPC network structure, see Practical VPC Design.
If you have questions about this feature or anything else related to AWS KMS, start a new thread in the AWS KMS forum.
Since VPC Endpoints launched in 2015, creating Endpoints has been a popular way to securely access S3 and DynamoDB from an Amazon Virtual Private Cloud (VPC) without the need for an Internet gateway, a NAT gateway, or firewall proxies. With VPC Endpoints, the routing between the VPC and the AWS service is handled by the AWS network, and IAM policies can be used to control access to service resources.
Today we are announcing AWS PrivateLink, the newest generation of VPC Endpoints which is designed for customers to access AWS services in a highly available and scalable manner, while keeping all the traffic within the AWS network. Kinesis, Service Catalog, Amazon EC2, EC2 Systems Manager (SSM), and Elastic Load Balancing (ELB) APIs are now available to use inside your VPC, with support for more services coming soon such as Key Management Service (KMS) and Amazon Cloudwatch.
With traditional endpoints, it’s very much like connecting a virtual cable between your VPC and the AWS service. Connectivity to the AWS service does not require an Internet or NAT gateway, but the endpoint remains outside of your VPC. With PrivateLink, endpoints are instead created directly inside of your VPC, using Elastic Network Interfaces (ENIs) and IP addresses in your VPC’s subnets. The service is now in your VPC, enabling connectivity to AWS services via private IP addresses. That means that VPC Security Groups can be used to manage access to the endpoints and that PrivateLink endpoints can also be accessed from your premises via AWS Direct Connect.
Using the services powered by PrivateLink, customers can now manage fleets of instances, create and manage catalogs of IT services as well as store and process data, without requiring the traffic to traverse the Internet.
Creating a PrivateLink Endpoint To create a PrivateLink endpoint, I navigate to the VPC Console, select Endpoints, and choose Create Endpoint.
I then choose which service I’d like to access. New PrivateLink endpoints have an “interface” type. In this case I’d like to use the Kinesis service directly from my VPC and I choose the kinesis-streams service.
At this point I can choose which of my VPCs I’d like to launch my new endpoint in, and select the subnets that the ENIs and IP addresses will be placed in. I can also associate the endpoint with a new or existing Security Group, allowing me to control which of my instances can access the Endpoint.
Because PrivateLink endpoints will use IP addresses from my VPC, I have the option to over-ride DNS for the AWS service DNS name by using VPC Private DNS. By leaving Enable Private DNS Name checked, lookups from within my VPC for “kinesis.us-east-1.amazonaws.com” will resolve to the IP addresses for the endpoint that I’m creating. This makes the transition to the endpoint seamless without requiring any changes to my applications. If I’d prefer to test or configure the endpoint before handling traffic by default, I can leave this disabled and then change it at any time by editing the endpoint.
Once I’m ready and happy with the VPC, subnets and DNS settings, I click Create Endpoint to complete the process.
Using a PrivateLink Endpoint
By default, with the Private DNS Name enabled, using a PrivateLink endpoint is as straight-forward as using the SDK, AWS CLI or other software that accesses the service API from within your VPC. There’s no need to change any code or configurations.
To support testing and advanced configurations, every endpoint also gets a set of DNS names that are unique and dedicated to your endpoint. There’s a primary name for the endpoint and zonal names.
The primary name is particularly useful for accessing your endpoint via Direct Connect, without having to use any DNS over-rides on-premises. Naturally, the primary name can also be used inside of your VPC. The primary name, and the main service name – since I chose to over-ride it – include zonal fault-tolerance and will balance traffic between the Availability Zones. If I had an architecture that uses zonal isolation techniques, either for fault containment and compartmentalization, low latency, or for minimizing regional data transfer I could also use the zonal names to explicitly control whether my traffic flows between or stays within zones.
Pricing & Availability AWS PrivateLink is available today in all AWS commercial regions except China (Beijing). For the region availability of individual services, please check our documentation.
Pricing starts at $0.01 / hour plus a data processing charge at $0.01 / GB. Data transferred between availability zones, or between your Endpoint and your premises via Direct Connect will also incur the usual EC2 Regional and Direct Connect data transfer charges. For more information, see VPC Pricing.
You spent almost all day Saturday archiving your latest video project onto two 8 TB external hard drives. You need to archive four months’ worth of work from a recently finished video project to external hard drives to make room on your local storage system for the next project. You diligently label each of the newly minted archive drives with the project name and stack the drives neatly in your closet. There must be 50 drives in there. The final step is to add the file list from each drive to the catalog you keep on a shared spreadsheet so your employees and contractors can find content from previous projects. In reality, this type of searching is rarely done as rummaging through the closet for the correct archive drive is time consuming and on more than one occasion the drive has failed.
Are you thinking that maybe it is time to upgrade your media archive solution?
Media Archiving Solutions
There is no shortage of media archiving solutions and you’ve looked at everything from tape drive systems to SAN and NAS systems. Some are expensive, some are complex, and some are both. Here are a few things you’ve decided that you would like your media archive solution to do:
Fit into your company’s workflow
Make the archive more accessible and useable
Protect your archive off site
Archiving Your Media Content with Archiware P5 Archive
One proven solution is Archiware P5 Archive, which is part of the P5 software suite that provides data management at every step of your data’s life cycle. The P5 suite works with all kinds of data, but it has become well known for how well it archives and backs up media files, e.g. video and photos.
P5 Archive lets you easily archive data from your primary storage system to less expensive storage such as external disk, tape, and the cloud. Once the data is archived, you can use P5 to quickly locate your data in the archive by searching with keywords or previews. For example:
Search assets by keyword — Besides the default search parameters, you can add custom metadata to individualize your data storage. You can include categories such as time of day, lens, or film location, and thus locate and re-use your data more quickly and effectively.
Search assets with previews — P5 Archive has a direct integration with FFmpeg and ImageMagick and can create low resolution previews and proxies for all common video and audio formats.
With these capabilities, P5 can function as a bare bones asset management solution managing video and image archives. When you are ready to move forward with a more robust media asset management (MAM) system, P5 has integrations to leading providers including axle Video, Cat DV, and Cantemo Portal.
P5 Archive also includes the ability to customize the end-user experience so that users can access data (archived or live) based on their user profile.
Your Media Archive can be an Asset
With Archiware P5 and Backblaze B2, your media archive is an asset, instead of a burden.
P5 Archive lets you move or migrate data to disk, tape and the cloud. With Archiware P5 version 5.5, you can backup and archive your media files to Backblaze B2 Cloud Storage. With B2, your archived files are readily available for retrieval via P5. When this is combined with the P5 preview and keyword search capabilities, you can locate and retrieve archived video clips, images, and files in minutes versus hours or days when using external disk or tape.
Getting Rid of Your Closet Full of Hard Drives
Even if you move forward with P5, you still have your current closet full of archived data. To help with that, Backblaze provides the B2 Fireball data transfer service, which allows you to transfer up to 40 TB of data per trip from your location to your B2 Cloud Storage account. In this case, you’ll have to transfer the data from each external drive in your closet to a server or NAS device. Once there, the collected data is transferred to the Fireball and the Fireball is returned to Backblaze, where the data is extracted and placed in your B2 account.
As noted, each Fireball holds up to 40 terabytes of data — that’s ten 4 TB external drives — so it will take three round trips to transfer the 100 TB of archived data in your closet. Of course you can decide to transfer some or all of the data residing in your closet. How much you transfer depends on how much you want to protect it offsite and how valuable ready access to it on B2 from anywhere is to you.
If your media archive is a pile of hard drives or an aging tape library, the combination of Archiware P5 and Backblaze B2 provides a practical, affordable way to move your media archive library to the cloud. This will let you improve access to your archived data, reduce the management of your local storage system, protect your valuable assets offsite, and best of all you’ll be able to use your closet to store old computer monitors and pristine user manuals like everyone else.
Reverse proxies are a powerful software architecture primitive for fetching resources from a server on behalf of a client. They serve a number of purposes, from protecting servers from unwanted traffic to offloading some of the heavy lifting of HTTP traffic processing.
This post explains the benefits of a reverse proxy, and explains how to use NGINX and Amazon EC2 Container Service (Amazon ECS) to easily implement and deploy a reverse proxy for your containerized application.
Components
NGINX is a high performance HTTP server that has achieved significant adoption because of its asynchronous event driven architecture. It can serve thousands of concurrent requests with a low memory footprint. This efficiency also makes it ideal as a reverse proxy.
Amazon ECS is a highly scalable, high performance container management service that supports Docker containers. It allows you to run applications easily on a managed cluster of Amazon EC2 instances. Amazon ECS helps you get your application components running on instances according to a specified configuration. It also helps scale out these components across an entire fleet of instances.
Sidecar containers are a common software pattern that has been embraced by engineering organizations. It’s a way to keep server side architecture easier to understand by building with smaller, modular containers that each serve a simple purpose. Just like an application can be powered by multiple microservices, each microservice can also be powered by multiple containers that work together. A sidecar container is simply a way to move part of the core responsibility of a service out into a containerized module that is deployed alongside a core application container.
The following diagram shows how an NGINX reverse proxy sidecar container operates alongside an application server container:
In this architecture, Amazon ECS has deployed two copies of an application stack that is made up of an NGINX reverse proxy side container and an application container. Web traffic from the public goes to an Application Load Balancer, which then distributes the traffic to one of the NGINX reverse proxy sidecars. The NGINX reverse proxy then forwards the request to the application server and returns its response to the client via the load balancer.
Reverse proxy for security
Security is one reason for using a reverse proxy in front of an application container. Any web server that serves resources to the public can expect to receive lots of unwanted traffic every day. Some of this traffic is relatively benign scans by researchers and tools, such as Shodan or nmap:
[18/May/2017:15:10:10 +0000] "GET /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann HTTP/1.1" 404 1389 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
[18/May/2017:18:19:51 +0000] "GET /clientaccesspolicy.xml HTTP/1.1" 404 322 - Cloud mapping experiment. Contact [email protected]
But other traffic is much more malicious. For example, here is what a web server sees while being scanned by the hacking tool ZmEu, which scans web servers trying to find PHPMyAdmin installations to exploit:
In addition, servers can also end up receiving unwanted web traffic that is intended for another server. In a cloud environment, an application may end up reusing an IP address that was formerly connected to another service. It’s common for misconfigured or misbehaving DNS servers to send traffic intended for a different host to an IP address now connected to your server.
It’s the responsibility of anyone running a web server to handle and reject potentially malicious traffic or unwanted traffic. Ideally, the web server can reject this traffic as early as possible, before it actually reaches the core application code. A reverse proxy is one way to provide this layer of protection for an application server. It can be configured to reject these requests before they reach the application server.
Reverse proxy for performance
Another advantage of using a reverse proxy such as NGINX is that it can be configured to offload some heavy lifting from your application container. For example, every HTTP server should support gzip. Whenever a client requests gzip encoding, the server compresses the response before sending it back to the client. This compression saves network bandwidth, which also improves speed for clients who now don’t have to wait as long for a response to fully download.
NGINX can be configured to accept a plaintext response from your application container and gzip encode it before sending it down to the client. This allows your application container to focus 100% of its CPU allotment on running business logic, while NGINX handles the encoding with its efficient gzip implementation.
An application may have security concerns that require SSL termination at the instance level instead of at the load balancer. NGINX can also be configured to terminate SSL before proxying the request to a local application container. Again, this also removes some CPU load from the application container, allowing it to focus on running business logic. It also gives you a cleaner way to patch any SSL vulnerabilities or update SSL certificates by updating the NGINX container without needing to change the application container.
NGINX configuration
Configuring NGINX for both traffic filtering and gzip encoding is shown below:
http {
# NGINX will handle gzip compression of responses from the app server
gzip on;
gzip_proxied any;
gzip_types text/plain application/json;
gzip_min_length 1000;
server {
listen 80;
# NGINX will reject anything not matching /api
location /api {
# Reject requests with unsupported HTTP method
if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
return 405;
}
# Only requests matching the whitelist expectations will
# get sent to the application server
proxy_pass http://app:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
}
}
}
The above configuration only accepts traffic that matches the expression /api and has a recognized HTTP method. If the traffic matches, it is forwarded to a local application container accessible at the local hostname app. If the client requested gzip encoding, the plaintext response from that application container is gzip-encoded.
Amazon ECS configuration
Configuring ECS to run this NGINX container as a sidecar is also simple. ECS uses a core primitive called the task definition. Each task definition can include one or more containers, which can be linked to each other:
This task definition causes ECS to start both an NGINX container and an application container on the same instance. Then, the NGINX container is linked to the application container. This allows the NGINX container to send traffic to the application container using the hostname app.
The NGINX container has a port mapping that exposes port 80 on a publically accessible port but the application container does not. This means that the application container is not directly addressable. The only way to send it traffic is to send traffic to the NGINX container, which filters that traffic down. It only forwards to the application container if the traffic passes the whitelisted rules.
Conclusion
Running a sidecar container such as NGINX can bring significant benefits by making it easier to provide protection for application containers. Sidecar containers also improve performance by freeing your application container from various CPU intensive tasks. Amazon ECS makes it easy to run sidecar containers, and automate their deployment across your cluster.
As a hosting provider, we speak with many businesses who need a fix for their slow site speeds. There are many contributing factors why hosting infrastructure may be constraining your site performance but typically; old infrastructure used by some hosting providers, contention issues and even the physical location of the servers. Having your site hosted in a high-speed environment with world class managed services (such as Anchor) provides the right foundations and utilising a Content Delivery Network (CDN) that can give you that extra boost in speed and performance you desire – and deserve. One of the more popular site performance applications is Cloudflare; global network designed to optimize security, performance and reliability, without the bloat of legacy technologies. Cloudflare has some robust CDN capabilities in addition to other security services like DDoS (Distributed Denial of Service) protection and reverse proxies.
A traditional CDN is a group of web servers distributed across multiple locations around the world, which delivers content more efficiently to users. The server selected for delivering content to a specific user is typically based on a measure of network proximity. For example, the server with the fewest network hops or the server with the quickest response time is chosen.
If you are looking to take advantage of a CDN, a great place to to start is Cloudflare’s free plan. This basic plan can be set up in less than 5 minutes and only requires a simple change to your domain’s DNS settings to get you up and running. There is no hardware or software to install or maintain and you do not need to change any of your site’s existing code. As a partner of Cloudflare, we can offer discounted pricing to our customers if you are looking to take advantage of some of Cloudflare’s advanced performance and security features such as image optimisations, firewalls and PCI compliance to name just a few.
CloudFlare utilises more than 40 data centres in almost as many countries, and use the size of their ‘quietly built cloud’ to process more than 5% of all web requests. It includes:
A Global CDN
DDoS Protection
Page Rules
DDoS Protection- Why do I need it and how to protect against attack?
In 2015 the internet saw the highest rate of DDoS attacks ever. Generally, the attackers will flood a network or service (usually with thousands of IP addresses) in order to overwhelm the server and make a network or website unavailable for its users. It is extremely important to make sure your site is protected from such an attack, especially if your site is eCommerce and down time will prevent customers completing their purchases.
What are Global CDN’s?
As mentioned above, Content Delivery Networks (CDNs) are important for a number of reasons. The primary feature that a CDN does, is provides alternative server nodes, or locations for the user to download resources (usually JavaScript or static content). This means that although the server may be located in the US, someone in Sydney can still experience fast load speed and response times due to this reduced latency. This is extremely important for sites that have users in other countries, especially those who are shopping online, as these sites generally have a large volumes of images, which can be timely to load. Overall, it improves your user’s experience in terms of speed.
Page Rules
Page Rules give you the ability to control how Cloudflare actually works on a URL or subdomain basis, which means it allows you to customise it’s functionality to match your domain’s unique needs. They give you the ability to take various actions based on the page’s URL, such as creating redirects, fine tuning caching behavior, or enabling and disabling our various services. This helps you to optimize speed, harden security, increase reliability, maximize bandwidth savings, and much more.
Other benefits include, the added scalability or capacity effects that a CDN like Cloudflare has, not only does it have higher availability but also lower packet loss. Further, Cloudflare provides website traffic insight and other analytics such as threat monitoring, so that you can improve your site even further.
As a partner of Cloudflare, Anchor receives discounted rates for the Pro and Business plans, as well as can help you install the free plan if you are a customer. The easiest part about Cloudflare however, is that it only requires a simple change to your domain’s DNS settings. There is no hardware or software to install or maintain and you do not need to change any of your site’s existing code.
If your site is running slow and want know how you can boost your site performance, contact us for a free, no obligation site hosting check up.
In case you missed any AWS Security Blog posts published so far in 2017, they are summarized and linked to below. The posts are shown in reverse chronological order (most recent first), and the subject matter ranges from protecting dynamic web applications against DDoS attacks to monitoring AWS account configuration changes and API calls to Amazon EC2 security groups.
March
March 22:How to Help Protect Dynamic Web Applications Against DDoS Attacks by Using Amazon CloudFront and Amazon Route 53 Using a content delivery network (CDN) such as Amazon CloudFront to cache and serve static text and images or downloadable objects such as media files and documents is a common strategy to improve webpage load times, reduce network bandwidth costs, lessen the load on web servers, and mitigate distributed denial of service (DDoS) attacks. AWS WAF is a web application firewall that can be deployed on CloudFront to help protect your application against DDoS attacks by giving you control over which traffic to allow or block by defining security rules. When users access your application, the Domain Name System (DNS) translates human-readable domain names (for example, www.example.com) to machine-readable IP addresses (for example, 192.0.2.44). A DNS service, such as Amazon Route 53, can effectively connect users’ requests to a CloudFront distribution that proxies requests for dynamic content to the infrastructure hosting your application’s endpoints. In this blog post, I show you how to deploy CloudFront with AWS WAF and Route 53 to help protect dynamic web applications (with dynamic content such as a response to user input) against DDoS attacks. The steps shown in this post are key to implementing the overall approach described in AWS Best Practices for DDoS Resiliency and enable the built-in, managed DDoS protection service, AWS Shield.
March 21:New AWS Encryption SDK for Python Simplifies Multiple Master Key Encryption The AWS Cryptography team is happy to announce a Python implementation of the AWS Encryption SDK. This new SDK helps manage data keys for you, and it simplifies the process of encrypting data under multiple master keys. As a result, this new SDK allows you to focus on the code that drives your business forward. It also provides a framework you can easily extend to ensure that you have a cryptographic library that is configured to match and enforce your standards. The SDK also includes ready-to-use examples. If you are a Java developer, you can refer to this blog post to see specific Java examples for the SDK. In this blog post, I show you how you can use the AWS Encryption SDK to simplify the process of encrypting data and how to protect your encryption keys in ways that help improve application availability by not tying you to a single region or key management solution.
March 21:Updated CJIS Workbook Now Available by Request The need for guidance when implementing Criminal Justice Information Services (CJIS)–compliant solutions has become of paramount importance as more law enforcement customers and technology partners move to store and process criminal justice data in the cloud. AWS services allow these customers to easily and securely architect a CJIS-compliant solution when handling criminal justice data, creating a durable, cost-effective, and secure IT infrastructure that better supports local, state, and federal law enforcement in carrying out their public safety missions. AWS has created several documents (collectively referred to as the CJIS Workbook) to assist you in aligning with the FBI’s CJIS Security Policy. You can use the workbook as a framework for developing CJIS-compliant architecture in the AWS Cloud. The workbook helps you define and test the controls you operate, and document the dependence on the controls that AWS operates (compute, storage, database, networking, regions, Availability Zones, and edge locations).
March 9:New Cloud Directory API Makes It Easier to Query Data Along Multiple Dimensions Today, we made available a new Cloud Directory API, ListObjectParentPaths, that enables you to retrieve all available parent paths for any directory object across multiple hierarchies. Use this API when you want to fetch all parent objects for a specific child object. The order of the paths and objects returned is consistent across iterative calls to the API, unless objects are moved or deleted. In case an object has multiple parents, the API allows you to control the number of paths returned by using a paginated call pattern. In this blog post, I use an example directory to demonstrate how this new API enables you to retrieve data across multiple dimensions to implement powerful applications quickly.
March 8:How to Access the AWS Management Console Using AWS Microsoft AD and Your On-Premises Credentials AWS Directory Service for Microsoft Active Directory, also known as AWS Microsoft AD, is a managed Microsoft Active Directory (AD) hosted in the AWS Cloud. Now, AWS Microsoft AD makes it easy for you to give your users permission to manage AWS resources by using on-premises AD administrative tools. With AWS Microsoft AD, you can grant your on-premises users permissions to resources such as the AWS Management Console instead of adding AWS Identity and Access Management (IAM) user accounts or configuring AD Federation Services (AD FS) with Security Assertion Markup Language (SAML). In this blog post, I show how to use AWS Microsoft AD to enable your on-premises AD users to sign in to the AWS Management Console with their on-premises AD user credentials to access and manage AWS resources through IAM roles.
March 7:How to Protect Your Web Application Against DDoS Attacks by Using Amazon Route 53 and an External Content Delivery Network Distributed Denial of Service (DDoS) attacks are attempts by a malicious actor to flood a network, system, or application with more traffic, connections, or requests than it is able to handle. To protect your web application against DDoS attacks, you can use AWS Shield, a DDoS protection service that AWS provides automatically to all AWS customers at no additional charge. You can use AWS Shield in conjunction with DDoS-resilient web services such as Amazon CloudFront and Amazon Route 53 to improve your ability to defend against DDoS attacks. Learn more about architecting for DDoS resiliency by reading the AWS Best Practices for DDoS Resiliency whitepaper. You also have the option of using Route 53 with an externally hosted content delivery network (CDN). In this blog post, I show how you can help protect the zone apex (also known as the root domain) of your web application by using Route 53 to perform a secure redirect to prevent discovery of your application origin.
February 23:s2n Is Now Handling 100 Percent of SSL Traffic for Amazon S3 Today, we’ve achieved another important milestone for securing customer data: we have replaced OpenSSL with s2n for all internal and external SSL traffic in Amazon Simple Storage Service (Amazon S3) commercial regions. This was implemented with minimal impact to customers, and multiple means of error checking were used to ensure a smooth transition, including client integration tests, catching potential interoperability conflicts, and identifying memory leaks through fuzz testing.
February 22:Easily Replace or Attach an IAM Role to an Existing EC2 Instance by Using the EC2 Console AWS Identity and Access Management (IAM) roles enable your applications running on Amazon EC2 to use temporary security credentials. IAM roles for EC2 make it easier for your applications to make API requests securely from an instance because they do not require you to manage AWS security credentials that the applications use. Recently, we enabled you to use temporary security credentials for your applications by attaching an IAM role to an existing EC2 instance by using the AWS CLI and SDK. To learn more, see New! Attach an AWS IAM Role to an Existing Amazon EC2 Instance by Using the AWS CLI. Starting today, you can attach an IAM role to an existing EC2 instance from the EC2 console. You can also use the EC2 console to replace an IAM role attached to an existing instance. In this blog post, I will show how to attach an IAM role to an existing EC2 instance from the EC2 console.
February 22:How to Audit Your AWS Resources for Security Compliance by Using Custom AWS Config Rules AWS Config Rules enables you to implement security policies as code for your organization and evaluate configuration changes to AWS resources against these policies. You can use Config rules to audit your use of AWS resources for compliance with external compliance frameworks such as CIS AWS Foundations Benchmark and with your internal security policies related to the US Health Insurance Portability and Accountability Act (HIPAA), the Federal Risk and Authorization Management Program (FedRAMP), and other regimes. AWS provides some predefined, managed Config rules. You also can create custom Config rules based on criteria you define within an AWS Lambda function. In this post, I show how to create a custom rule that audits AWS resources for security compliance by enabling VPC Flow Logs for an Amazon Virtual Private Cloud (VPC). The custom rule meets requirement 4.3 of the CIS AWS Foundations Benchmark: “Ensure VPC flow logging is enabled in all VPCs.”
February 13:How to Enable Multi-Factor Authentication for AWS Services by Using AWS Microsoft AD and On-Premises Credentials You can now enable multi-factor authentication (MFA) for users of AWS services such as Amazon WorkSpaces and Amazon QuickSight and their on-premises credentials by using your AWS Directory Service for Microsoft Active Directory (Enterprise Edition) directory, also known as AWS Microsoft AD. MFA adds an extra layer of protection to a user name and password (the first “factor”) by requiring users to enter an authentication code (the second factor), which has been provided by your virtual or hardware MFA solution. These factors together provide additional security by preventing access to AWS services, unless users supply a valid MFA code.
February 13:How to Create an Organizational Chart with Separate Hierarchies by Using Amazon Cloud Directory Amazon Cloud Directory enables you to create directories for a variety of use cases, such as organizational charts, course catalogs, and device registries. Cloud Directory offers you the flexibility to create directories with hierarchies that span multiple dimensions. For example, you can create an organizational chart that you can navigate through separate hierarchies for reporting structure, location, and cost center. In this blog post, I show how to use Cloud Directory APIs to create an organizational chart with two separate hierarchies in a single directory. I also show how to navigate the hierarchies and retrieve data. I use the Java SDK for all the sample code in this post, but you can use other language SDKs or the AWS CLI.
February 10:How to Easily Log On to AWS Services by Using Your On-Premises Active Directory AWS Directory Service for Microsoft Active Directory (Enterprise Edition), also known as Microsoft AD, now enables your users to log on with just their on-premises Active Directory (AD) user name—no domain name is required. This new domainlesslogon feature makes it easier to set up connections to your on-premises AD for use with applications such as Amazon WorkSpaces and Amazon QuickSight, and it keeps the user logon experience free from network naming. This new interforest trusts capability is now available when using Microsoft AD with Amazon WorkSpaces and Amazon QuickSight Enterprise Edition. In this blog post, I explain how Microsoft AD domainless logon works with AD interforest trusts, and I show an example of setting up Amazon WorkSpaces to use this capability.
February 9:New! Attach an AWS IAM Role to an Existing Amazon EC2 Instance by Using the AWS CLI AWS Identity and Access Management (IAM) roles enable your applications running on Amazon EC2 to use temporary security credentials that AWS creates, distributes, and rotates automatically. Using temporary credentials is an IAM best practice because you do not need to maintain long-term keys on your instance. Using IAM roles for EC2 also eliminates the need to use long-term AWS access keys that you have to manage manually or programmatically. Starting today, you can enable your applications to use temporary security credentials provided by AWS by attaching an IAM role to an existing EC2 instance. You can also replace the IAM role attached to an existing EC2 instance. In this blog post, I show how you can attach an IAM role to an existing EC2 instance by using the AWS CLI.
January 30:How to Protect Data at Rest with Amazon EC2 Instance Store Encryption Encrypting data at rest is vital for regulatory compliance to ensure that sensitive data saved on disks is not readable by any user or application without a valid key. Some compliance regulations such as PCI DSS and HIPAA require that data at rest be encrypted throughout the data lifecycle. To this end, AWS provides data-at-rest options and key management to support the encryption process. For example, you can encrypt Amazon EBS volumes and configure Amazon S3 buckets for server-side encryption (SSE) using AES-256 encryption. Additionally, Amazon RDS supports Transparent Data Encryption (TDE). Instance storage provides temporary block-level storage for Amazon EC2 instances. This storage is located on disks attached physically to a host computer. Instance storage is ideal for temporary storage of information that frequently changes, such as buffers, caches, and scratch data. By default, files stored on these disks are not encrypted. In this blog post, I show a method for encrypting data on Linux EC2 instance stores by using Linux built-in libraries. This method encrypts files transparently, which protects confidential data. As a result, applications that process the data are unaware of the disk-level encryption.
January 27:How to Detect and Automatically Remediate Unintended Permissions in Amazon S3 Object ACLs with CloudWatch Events Amazon S3Access Control Lists (ACLs) enable you to specify permissions that grant access to S3 buckets and objects. When S3 receives a request for an object, it verifies whether the requester has the necessary access permissions in the associated ACL. For example, you could set up an ACL for an object so that only the users in your account can access it, or you could make an object public so that it can be accessed by anyone. If the number of objects and users in your AWS account is large, ensuring that you have attached correctly configured ACLs to your objects can be a challenge. For example, what if a user were to call the PutObjectAcl API call on an object that is supposed to be private and make it public? Or, what if a user were to call the PutObject with the optional Acl parameter set to public-read, therefore uploading a confidential file as publicly readable? In this blog post, I show a solution that uses Amazon CloudWatch Events to detect PutObject and PutObjectAcl API calls in near-real time and helps ensure that the objects remain private by making automatic PutObjectAcl calls, when necessary.
January 24:New SOC 2 Report Available: Confidentiality As with everything at Amazon, the success of our security and compliance program is primarily measured by one thing: our customers’ success. Our customers drive our portfolio of compliance reports, attestations, and certifications that support their efforts in running a secure and compliant cloud environment. As a result of our engagement with key customers across the globe, we are happy to announce the publication of our new SOC 2 Confidentiality report. This report is available now through AWS Artifact in the AWS Management Console.
January 18:Compliance in the Cloud for New Financial Services Cybersecurity Regulations Financial regulatory agencies are focused more than ever on ensuring responsible innovation. Consequently, if you want to achieve compliance with financial services regulations, you must be increasingly agile and employ dynamic security capabilities. AWS enables you to achieve this by providing you with the tools you need to scale your security and compliance capabilities on AWS. The following breakdown of the most recent cybersecurity regulations, NY DFS Rule 23 NYCRR 500, demonstrates how AWS continues to focus on your regulatory needs in the financial services sector.
January 9:New Amazon GameDev Blog Post: Protect Multiplayer Game Servers from DDoS Attacks by Using Amazon GameLift In online gaming, distributed denial of service (DDoS) attacks target a game’s network layer, flooding servers with requests until performance degrades considerably. These attacks can limit a game’s availability to players and limit the player experience for those who can connect. Today’s new Amazon GameDev Blog post uses a typical game server architecture to highlight DDoS attack vulnerabilities and discusses how to stay protected by using built-in AWS Cloud security, AWS security best practices, and the security features of Amazon GameLift. Read the post to learn more.
January 6:The Top 10 Most Downloaded AWS Security and Compliance Documents in 2016 The following list includes the 10 most downloaded AWS security and compliance documents in 2016. Using this list, you can learn about what other people found most interesting about security and compliance last year.
January 6:FedRAMP Compliance Update: AWS GovCloud (US) Region Receives a JAB-Issued FedRAMP High Baseline P-ATO for Three New Services Three new services in the AWS GovCloud (US) region have received a Provisional Authority to Operate (P-ATO) from the Joint Authorization Board (JAB) under the Federal Risk and Authorization Management Program (FedRAMP). JAB issued the authorization at the High baseline, which enables US government agencies and their service providers the capability to use these services to process the government’s most sensitive unclassified data, including Personal Identifiable Information (PII), Protected Health Information (PHI), Controlled Unclassified Information (CUI), criminal justice information (CJI), and financial data.
January 4:The Top 20 Most Viewed AWS IAM Documentation Pages in 2016 The following 20 pages were the most viewed AWS Identity and Access Management (IAM) documentation pages in 2016. I have included a brief description with each link to give you a clearer idea of what each page covers. Use this list to see what other people have been viewing and perhaps to pique your own interest about a topic you’ve been meaning to research.
January 3:The Most Viewed AWS Security Blog Posts in 2016 The following 10 posts were the most viewed AWS Security Blog posts that we published during 2016. You can use this list as a guide to catch up on your blog reading or even read a post again that you found particularly useful.
January 3:How to Monitor AWS Account Configuration Changes and API Calls to Amazon EC2 Security Groups You can use AWS security controls to detect and mitigate risks to your AWS resources. The purpose of each security control is defined by its control objective. For example, the control objective of an Amazon VPC security group is to permit only designated traffic to enter or leave a network interface. Let’s say you have an Internet-facing e-commerce website, and your security administrator has determined that only HTTP (TCP port 80) and HTTPS (TCP 443) traffic should be allowed access to the public subnet. As a result, your administrator configures a security group to meet this control objective. What if, though, someone were to inadvertently change this security group’s rules and enable FTP or other protocols to access the public subnet from any location on the Internet? That expanded access could weaken the security posture of your assets. Consequently, your administrator might need to monitor the integrity of your company’s security controls so that the controls maintain their desired effectiveness. In this blog post, I explore two methods for detecting unintended changes to VPC security groups. The two methods address not only control objectives but also control failures.
If you have questions about or issues with implementing the solutions in any of these posts, please start a new thread on the forum identified near the end of each post.
Using a content delivery network (CDN) such as Amazon CloudFront to cache and serve static text and images or downloadable objects such as media files and documents is a common strategy to improve webpage load times, reduce network bandwidth costs, lessen the load on web servers, and mitigate distributed denial of service (DDoS) attacks. AWS WAF is a web application firewall that can be deployed on CloudFront to help protect your application against DDoS attacks by giving you control over which traffic to allow or block by defining security rules. When users access your application, the Domain Name System (DNS) translates human-readable domain names (for example, www.example.com) to machine-readable IP addresses (for example, 192.0.2.44). A DNS service, such as Amazon Route 53, can effectively connect users’ requests to a CloudFront distribution that proxies requests for dynamic content to the infrastructure hosting your application’s endpoints.
In this blog post, I show you how to deploy CloudFront with AWS WAF and Route 53 to help protect dynamic web applications (with dynamic content such as a response to user input) against DDoS attacks. The steps shown in this post are key to implementing the overall approach described in AWS Best Practices for DDoS Resiliency and enable the built-in, managed DDoS protection service, AWS Shield.
Background
AWS hosts CloudFront and Route 53 services on a distributed network of proxy servers in data centers throughout the world called edge locations. Using the global Amazon network of edge locations for application delivery and DNS service plays an important part in building a comprehensive defense against DDoS attacks for your dynamic web applications. These web applications can benefit from the increased security and availability provided by CloudFront and Route 53 as well as improving end users’ experience by reducing latency.
The following screenshot of an Amazon.com webpage shows how static and dynamic content can compose a dynamic web application that is delivered via HTTPS protocol for the encryption of user page requests as well as the pages that are returned by a web server.
The following map shows the global Amazon network of edge locations available to serve static content and proxy requests for dynamic content back to the origin as of the writing of this blog post. For the latest list of edge locations, see AWS Global Infrastructure.
How AWS Shield, CloudFront, and Route 53 work to help protect against DDoS attacks
To help keep your dynamic web applications available when they are under DDoS attack, the steps in this post enable AWS Shield Standard by configuring your applications behind CloudFront and Route 53. AWS Shield Standard protects your resources from common, frequently occurring network and transport layer DDoS attacks. Attack traffic can be geographically isolated and absorbed using the capacity in edge locations close to the source. Additionally, you can configure geographical restrictions to help block attacks originating from specific countries.
The request-routing technology in CloudFront connects each client to the nearest edge location, as determined by continuously updated latency measurements. HTTP and HTTPS requests sent to CloudFront can be monitored, and access to your application resources can be controlled at edge locations using AWS WAF. Based on conditions that you specify in AWS WAF, such as the IP addresses that requests originate from or the values of query strings, traffic can be allowed, blocked, or allowed and counted for further investigation or remediation. The following diagram shows how static and dynamic web application content can originate from endpoint resources within AWS or your corporate data center. For more details, see How CloudFront Delivers Content and How CloudFront Works with Regional Edge Caches.
Route 53 DNS requests and subsequent application traffic routed through CloudFront are inspected inline. Always-on monitoring, anomaly detection, and mitigation against common infrastructure DDoS attacks such as SYN/ACK floods, UDP floods, and reflection attacks are built into both Route 53 and CloudFront. For a review of common DDoS attack vectors, see How to Help Prepare for DDoS Attacks by Reducing Your Attack Surface. When the SYN flood attack threshold is exceeded, SYN cookies are activated to avoid dropping connections from legitimate clients. Deterministic packet filtering drops malformed TCP packets and invalid DNS requests, only allowing traffic to pass that is valid for the service. Heuristics-based anomaly detection evaluates attributes such as type, source, and composition of traffic. Traffic is scored across many dimensions, and only the most suspicious traffic is dropped. This method allows you to avoid false positives while protecting application availability.
Route 53 is also designed to withstand DNS query floods, which are real DNS requests that can continue for hours and attempt to exhaust DNS server resources. Route 53 uses shuffle sharding and anycast striping to spread DNS traffic across edge locations and help protect the availability of the service.
The next four sections provide guidance about how to deploy CloudFront, Route 53, AWS WAF, and, optionally, AWS Shield Advanced.
Deploy CloudFront
To take advantage of application delivery with DDoS mitigations at the edge, start by creating a CloudFront distribution and configuring origins:
Specify cache behavior settings for the distribution, as shown in the following screenshot. You can configure each URL path pattern with a set of associated cache behaviors. For dynamic web applications, set the Minimum TTL to 0 so that CloudFront will make a GET request with an If-Modified-Since header back to the origin. When CloudFront proxies traffic to the origin from edge locations and back, multiple concurrent requests for the same object are collapsed into a single request. The request is sent over a persistent connection from the edge location to the region over networks monitored by AWS. The use of a large initial TCP window size in CloudFront maximizes the available bandwidth, and TCP Fast Open (TFO) reduces latency.
To ensure that all traffic to CloudFront is encrypted and to enable SSL termination from clients at global edge locations, specify Redirect HTTP to HTTPS for Viewer Protocol Policy. Moving SSL termination to CloudFront offloads computationally expensive SSL negotiation, helps mitigate SSL abuse, and reduces latency with the use of OCSP stapling and session tickets. For more information about options for serving HTTPS requests, see Choosing How CloudFront Serves HTTPS Requests. For dynamic web applications, set Allowed HTTP Methods to include all methods, set Forward Headers to All, and for Query String Forwarding and Caching, choose Forward all, cache based on all.
Specify distribution settings for the distribution, as shown in the following screenshot. Enter your domain names in the Alternate Domain Names box and choose Custom SSL Certificate.
Choose Create Distribution. Note the x.cloudfront.net Domain Name of the distribution. In the next section, you will configure Route 53 to route traffic to this CloudFront distribution domain name.
Configure Route 53
When you created a web distribution in the previous section, CloudFront assigned a domain name to the distribution, such as d111111abcdef8.cloudfront.net. You can use this domain name in the URLs for your content, such as: http://d111111abcdef8.cloudfront.net/logo.jpg.
Alternatively, you might prefer to use your own domain name in URLs, such as: http://example.com/logo.jpg. You can accomplish this by creating a Route 53 alias resource record set that routes dynamic web application traffic to your CloudFront distribution by using your domain name. Alias resource record sets are virtual records specific to Route 53 that are used to map alias resource record sets for your domain to your CloudFront distribution. Alias resource record sets are similar to CNAME records except there is no charge for DNS queries to Route 53 alias resource record sets mapped to AWS services. Alias resource record sets are also not visible to resolvers, and they can be created for the root domain (zone apex) as well as subdomains.
A hosted zone, similar to a DNS zone file, is a collection of records that belongs to a single parent domain name. Each hosted zone has four nonoverlapping name servers in a delegation set. If a DNS query is dropped, the client automatically retries the next name server. If you have not already registered a domain name and have not configured a hosted zone for your domain, complete these two prerequisite steps before proceeding:
Choose the name of the hosted zone for the domain that you want to use to route traffic to your CloudFront distribution.
Choose Create Record Set.
Specify the following values:
Name – Type the domain name that you want to use to route traffic to your CloudFront distribution. The default value is the name of the hosted zone. For example, if the name of the hosted zone is example.com and you want to use acme.example.com to route traffic to your distribution, type acme.
Type – Choose A – IPv4 address. If IPv6 is enabled for the distribution and you are creating a second resource record set, choose AAAA – IPv6 address.
Alias – Choose Yes.
Alias Target – In the CloudFront distributions section, choose the name that CloudFront assigned to the distribution when you created it.
Routing Policy – Accept the default value of Simple.
Evaluate Target Health – Accept the default value of No.
Choose Create.
If IPv6 is enabled for the distribution, repeat Steps 4 through 6. Specify the same settings except for the Type field, as explained in Step 5.
The following screenshot of the Route 53 console shows a Route 53 alias resource record set that is configured to map a domain name to a CloudFront distribution.
If your dynamic web application requires geo redundancy, you can use latency-based routing in Route 53 to run origin servers in different AWS regions. Route 53 is integrated with CloudFront to collect latency measurements from each edge location. With Route 53 latency-based routing, each CloudFront edge location goes to the region with the lowest latency for the origin fetch.
Enable AWS WAF
AWS WAF is a web application firewall that helps detect and mitigate web application layer DDoS attacks by inspecting traffic inline. Application layer DDoS attacks use well-formed but malicious requests to evade mitigation and consume application resources. You can define custom security rules (also called web ACLs) that contain a set of conditions, rules, and actions to block attacking traffic. After you define web ACLs, you can apply them to CloudFront distributions, and web ACLs are evaluated in the priority order you specified when you configured them. Real-time metrics and sampled web requests are provided for each web ACL.
The following diagram shows how the (a) flow of CloudFront access logs files to an Amazon S3 bucket (b) provides the source data for the Lambda log parser function (c) to identify HTTP flood traffic and update AWS WAF web ACLs. As CloudFront receives requests on behalf of your dynamic web application, it sends access logs to an S3 bucket, triggering the Lambda log parser. The Lambda function parses CloudFront access logs to identify suspicious behavior, such as an unusual number of requests or errors, and it automatically updates your AWS WAF rules to block subsequent requests from the IP addresses in question for a predefined amount of time that you specify.
In addition to automated rate-based blacklisting to help protect against HTTP flood attacks, prebuilt AWS CloudFormation templates are available to simplify the configuration of AWS WAF for a proactive application-layer security defense. The following diagram provides an overview of CloudFormation template input into the creation of the CommonAttackProtection stack that includes AWS WAF web ACLs used to block, allow, or count requests that meet the criteria defined in each rule.
Choose the link under the ID column for your CloudFront distribution.
Choose Edit under the General
Choose your AWS WAF Web ACL from the drop-down
Choose Yes, Edit.
Activate AWS Shield Advanced (optional)
Deploying CloudFront, Route 53, and AWS WAF as described in this post enables the built-in DDoS protections for your dynamic web applications that are included with AWS Shield Standard. (There is no upfront cost or charge for AWS Shield Standard beyond the normal pricing for CloudFront, Route 53, and AWS WAF.) AWS Shield Standard is designed to meet the needs of many dynamic web applications.
For dynamic web applications that have a high risk or history of frequent, complex, or high volume DDoS attacks, AWS Shield Advanced provides additional DDoS mitigation capacity, attack visibility, cost protection, and access to the AWS DDoS Response Team (DRT). For more information about AWS Shield Advanced pricing, see AWS Shield Advanced pricing. To activate advanced protection services, follow these steps:
Sign in to the AWS Management Console and open the AWS WAF console.
If this is your first time signing in to the AWS WAF console, choose Get started with AWS Shield Advanced. Otherwise, choose Protected resources.
Choose Activate AWS Shield Advanced.
Choose the resource type and resource to protect.
For Name, enter a friendly name that will help you identify the AWS resources that are protected. For example, My CloudFront AWS Shield Advanced distributions.
(Optional) For Web DDoS attack, select Enable. You will be prompted to associate an existing web ACL with these resources, or create a new ACL if you don’t have any yet.
Choose Add DDoS protection.
Summary
In this blog post, I outline the steps to deploy CloudFront and configure Route 53 in front of your dynamic web application to leverage the global Amazon network of edge locations for DDoS resiliency. The post also provides guidance about enabling AWS WAF for application layer traffic monitoring and automated rules creation to block malicious traffic. I also cover the optional steps to activate AWS Shield Advanced, which helps build a more comprehensive defense against DDoS attacks for your dynamic web applications.
If you have comments about this post, submit them in the “Comments” section below. If you have questions about or issues implementing this solution, please open a new thread on the AWS WAF forum.
AWS Quick Starts help you to deploy popular solutions on AWS. Each Quick Start is designed by AWS solutions architects or partners, and makes use of AWS best practices for security and high availability. You can use them to spin up test or production environments that you can use right away.
The Quick Starts include comprehensive deployment guides and AWS CloudFormation templates that you can launch with a single click. The collection of Quick Starts is broken down in to seven categories, as follows:
DevOps
Databases & storage
Big Data & analytics
Security & compliance
Microsoft & SAP
Networking & access
Additional
Over the past two months we have added six new Quick Starts to our collection, bringing the total up to 42. Today I would like to give you an overview of the newest Quick Starts in each category.
Tableau Server (Big data & analytics) The Tableau Server on AWS Quick Start helps you to deploy a fully functional Tableau Server on the AWS Cloud. You can launch a single node deployment in your default VPC, or a multi-node cluster deployment in a new or existing VPC. Here’s the cluster architecture:
The CloudFormation template will prompt you for (among other things) your Tableau Activation Key.
Splunk Enterprise (Big data & analytics) The Splunk Enterprise on AWS Quick Start helps you to deploy a distributed Splunk Enterprise environment on the AWS Cloud. You can launch into an existing VPC with two or more Availability Zones or you can create a new VPC. Here’s the architecture:
The template will prompt you for the name of an S3 bucket and the path (within the bucket) to a Splunk license file.
UK OFFICIAL (Security & compliance) The UK-OFFICIAL on AWS Quick Start sets up a standardized AWS Cloud environment that supports workloads that are classified as United Kingdom (UK) OFFICIAL. The environment aligns with the in-scope guidelines found in the NCSC Cloud Security Principles and the CIS Critical Security Controls (take a look at the security controls matrix to learn more). Here’s the architecture:
You will need to have an Alfresco trial license in order to launch the cluster.
Symantec Protection Engine (Security & compliance) The Symantec Protection Engine on AWS Quick Start helps you to deploy Symantec Protection Engine (SPE) in less than an hour. Once deployed (into a new or existing VPC), you can use SPE’s APIs to incorporate malware and threat detection into your applications. You can also connect it to proxies and scan traffic for viruses, trojans, and other types of malware. Here’s the architecture:
You will need to purchase an SPE license or subscribe to the SPE AMI in order to use this Quick Start.
Thanks to Josh Triplett for sending us this chromium bug report about a dump of unitialized memory caused by Cloudflare’s reverse proxies. “A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I’ll explain later). My working theory was that this was related to their “ScrapeShield” feature which parses and obfuscates html – but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers. We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security. ”
NAT is a firewall. It’s the most common firewall. It’s the best firewall.
I thought I’d point this out because most security experts might disagree, pointing to some “textbook definition”. This is wrong. A “firewall” is anything that establishes a barrier between some internal (presumably trusted) network and the outside, public, and dangerous Internet where anybody can connect to you at any time. A NAT creates exactly that sort of barrier.
What other firewalls provide (the SPI packet filters) is the ability to block outbound connections, not just incoming connections. That’s nice, but that’s not a critical feature. Indeed, few organizations use firewalls that way, it just causes complaints when internal users cannot access Internet resources.
Another way of using firewalls is to specify connections between a DMZ and an internal network, such as a web server exposed to the Internet that needs a hole in the firewall to access an internal database. While not technically part of the NAT definition, it’s a feature of all modern NATs. It’s the only way to get some games to work, for example.
There’s already more than 10-billion devices on the Internet, including homes with many devices, as well as most mobile phones. This means that NAT is the most common firewall. The reason hackers find it difficult hacking into iPhones is partly because they connect to the Internet through carrier-grade NAT. When hackers used “alpine” as the backdoor in Cydia, they still had to exploit it over local WiFi rather than the carrier network.
Not only is NAT the most common firewall, it’s the best firewall. Simple SPI firewalls that don’t translate addresses have an inherent hole in that they are “fail open”. It’s easy to apply the wrong firewall ruleset, either permanently, or just for moment. You see this on internal IDS, where for no reason there’s suddenly a spike of attacks against internal machines because of a bad rule. Every large organization I’ve worked with can cite examples of this.
NAT, on the other hand, fails closed. Common mistakes shutdown access to the Internet rather than open up access from the Internet. The benefit is so compelling that organizations with lots of address space really need to give it up and move to private addressing instead.
The definition of firewall is malleable. At one time it included explicit and transparent proxies, for example, which were the most popular type. These days, many people think of only state packet inspection filters as the “true” firewall. I take the more expansive view of things.
The upshot is this: NAT is by definition a firewall. It’s the most popular firewall. It’s the best firewalling technology.
Note: Of course, no organization should use firewalls of any type. They break the “end-to-end” principle of the Internet, and thus should be banned by law.
Amazon API Gateway helps you quickly build highly scalable, secure, and robust APIs. Developers who want to consume your API to build web, mobile, or other types of apps need a site where they can learn about the API, acquire access, and manage their consumption. You can do this by hosting a developer portal: a web application that lists your APIs in catalog form, displays API documentation to help developers understand your API, and allows them to sign up for access. The application also needs to let developers test your APIs, and provide feedback to grow a successful developer ecosystem. Finally, you may also want to monetize your APIs and grow API product revenue.
Today, we published aws-api-gateway-developer-portal, an open source serverless web application that you can use to get started building your own developer portal. In just a few easy steps, you can generate a serverless web application that lists your APIs on API Gateway in catalog form, and allows for developer signups. The application is built using aws-serverless-express, an open source library that we published recently, that makes it incredibly easy to build serverless web applications using API Gateway and AWS Lambda. The application uses a SAM (Serverless Application Model) template to deploy its serverless resources, and is very easy to deploy and operate.
Walkthrough
In this post, I walk through the steps for creating a sample API and developer portal for third-party developer consumption:
Create an API or import a Swagger definition
Deploy the API for developer access
Document the API for easier developer consumption
Structure it into a usage plan to set access control policies, and set throttling limits.
Log in as a developer and verify the process of consuming an API.
You can also see how to create a AWS Marketplace listing and monetize your API.
Create an API
To get started, log in to the Amazon API Gateway console and import the example API (PetStore). You can also create your own APIs using Swagger, the console, or APIs.
Deploy the API
After creation, deploy the API to a stage to make it accessible to developers, and ready for testing. For this example, use “Marketplace” as the name for the API stage.
Document the API
API Gateway recently announced support for API documentation. You can now document various parts of your API by either importing Swagger or using the API Gateway console to edit API documentation directly.
Navigate to the Documentation tab to add missing documentation and make it easier for developers to understand your API. After you are satisfied with the new documentation, choose Publish Documentation and choose the Marketplace stage created earlier to update the previously deployed PetStore API with the modified documentation.
Package the API into usage plans
Usage plans in API Gateway help you structure your API for developer access. You can associate API keys to usage plans, and set throttling limits and quotas on a per-API key basis. Choose Usage Plans in the console, create a new usage plan, and set throttling limits and quotas as shown below.
When your customers subscribe to this usage plan, their requests are throttled at 200 RPS, and they can each make only 200,000 requests per month. You can change these limits at any time.
Choose Next to create the usage plan. (Skip the API Key screen, and add the Marketplace stage we created earlier)
Generate and configure a developer portal
Now, generate a developer portal in order to list the usage plan that you created earlier. To do this, clone the aws-api-gateway-developer-portal into a local folder. The README includes instructions on how to get set up, but let’s run through it step-by-step. First, ensure that you have the latest version of Node.js installed. For this walkthrough, you use the latest version of the AWS CLI, so make sure that you have your AWS credentials handy. and that you have configured the AWS CLI with the access-key and secret-key using the `aws –configure’ command.
Set up the developer portal
Then, open up a terminal window and run `npm run setup’. This step modifies several project files in-line with the information that you supply—choose a name for the S3 bucket that store webpages for the developer portal, as well as a name for the Lambda function that wraps the Express.js web application. The S3 bucket names that you specify in this step must be region-unique, so use a prefix (eg. ‘my-org-dev-portal-artifacts’).
After entering those details, it then creates the artifacts bucket on S3 (if the S3 bucket name you provided in the previous step doesn’t already exist). It also runs “package and deploy” on the SAM template using the new
command. It takes several minutes for CloudFormation to create the stack.
This step creates all the resources that you need for the developer portal, after setting up the IAM roles that are needed for the operation of this application. Navigate to the AWS CloudFormation console, and choose the Resources tab to see the full list of resources that have been created.
The core components that were created for your portal are listed below:
API Gateway API: Proxies HTTP requests to the back-end Lambda function (Express.js web server that serves website content)
Developer portal S3 website URL: Runs the developer portal site (publically accessible bucket)
Lambda function: Acts as the primary back end or API server
Cognito user and identity pools: Creates Cognito user pool and identity pool, and links them.
DynamoDB table – Stores a map of API key IDs to customer IDs, which lets you track the API key and customer association.
Lambda function Acts as a listener that subscribes to a particular SNS topic, which is useful when customers cancel or subscribe to an API.
Configure the developer portal post-setup
Additional files are then modified with values from your new resources, such as Cognito pool IDs, etc., and your web app is then uploaded to Amazon S3. After this step completes, it opens up your developer portal website in your default browser.
At this point, you have a complete serverless website ready. The application has been integrated with Cognito User Pools, which makes it easy for you to sign up developers and manage their access. For more information, see the Amazon Cognito Your User Pools – Now Generally Available post.
Edit the dev-portal/src/catalog.json file to list your APIs that appear in the developer portal. This can be a subset of all the APIs you have in API Gateway. Replace the contents of this file with your API definitions. For accuracy, export your APIs from API Gateway as Swagger, and copy them into the catalog.json file. Then, run `npm run upload-site’ to make these APIs available in the developer portal.
Verify the process
Test the end-user flow by signing up for access.
After you sign in as a developer, you can subscribe to an API, to receive an API key that you as a developer would use in your API requests.
You can also browse the API documentation that the API owner published (you, as of 5 minutes ago…), and learn more about the API.
You can even test an API with your API Key. Click the “Show API Key” button on the top right corner of the page, and copy your API Key. Next click the red alert icon, enter your API Key, and click Authorize. Finally, click the “Try it out!” button on any of your resources to make a request to your live API.
Your static content for the site is packaged into the folder ‘/dev-portal’. Modify the case studies, and getting started content in this folder; when you’re satisfied, run ‘npm run upload-site’ to push the changes to your portal.
Monetize your APIs
Amazon API Gateway now integrates with the AWS Marketplace to help API owners monetize their API and meter usage for their API products, without writing any code. Now, any API built using API Gateway can easily be published on the AWS Marketplace using the API Gateway console, AWS CLI, or AWS SDK.
API Gateway automatically and accurately meters API consumption for any such API published to the AWS Marketplace and send it to the AWS Marketplace Metering Service, enabling sellers to focus on adding value to their API products rather than building integrations with AWS Marketplace. For more information, see the Monetize your APIs in AWS Marketplace using API Gateway post.
To leverage this feature, you must offer a developer portal that accepts signups from AWS customers. You can use this developer portal implementation to either build your own from scratch, or use it to add functionality to your existing site.
Conclusion
To summarize, you started with an API on API Gateway and structured it for developer consumption. Then in just a few easy steps, you used aws-api-gateway-developer-portal, to spin up a developer portal on serverless architecture, ready to accept developer signups. This application needs no infrastructure management, scales out-of-the-box, and directly integrates with AWS Marketplace to help you monetize your APIs.
If you have any questions or suggestions, please comment below.
In the news is Obama’s visit to the Hiroshima atomic bomb memorial. The memorial is more complex than you think. It’s not a simple condemnation of the bomb. Instead, it’s a much more subtle presentation of the complexity of what happened.
I mention this because of articles like this one at Foreign Policy magazine, in which the author starts by claiming he frequently re-visits Hiroshima. He claims that the memorial has a clear meaning, a message, that he takes back from the site. It doesn’t.
The museum puts the bombing into context. It shows how Japan had been in a constant state of war since the 1890s, with militaristic roots going back further in to Samurai culture. It showed how Japan would probably have continued their militaristic ways had the United States not demanded complete surrender, a near abdication of the emperor, and imposed a pacifist constitution on the country.
In other words, Japan accepts partial responsibility for having been bombed.
It doesn’t shy away from the horror of the bomb. It makes it clear that such bombs should never again be used on humans. But even that has complexity. More people were killed in the Tokyo firebombing than Hiroshima and Nagasaki combined. Had Hiroshima not been nuked, it would’ve instead been flattened with conventional bombs, causing more devastation and killing nearly as many people. The Japanese were likely more afraid of the Russian invasion than American nukes when they surrendered unconditionally.
When I left the memorial, I was left with the profound sense that I just didn’t know the answers.
The truth is that few have “real” opinions about the Hiroshima atomic blast. Their opinions just proxies for how they feel about American’s military might today. The Foreign Policy article above, which claims to have gotten the “message” from the memorial, is a lie. Nobody gets clarity and focus on the issue coming from the memorial, just a new appreciation of the problem.
Private subnets allow you to limit access to deployed components, and to control security and routing of the system. You can also use a private subnet to connect an on-premises local network to AWS through a VPN or AWS Direct Connect. Amazon EMR allows customers to launch clusters in a private subnet in Amazon VPC. For more information, see Select an Amazon VPC Subnet for the Cluster in the Amazon EMR Developer Guide.
When you launch an EMR cluster in a public subnet, the master node of the cluster has a public DNS, allowing you to create a SSH tunnel and securely access web applications like Hue, Apache Zeppelin, and other Hadoop web-UIs running on that node. However, when you launch an EMR cluster in a private subnet, your master node doesn’t have a public DNS, which can make it more difficult to access the web applications running on it.
In this post, I outline two possible solutions to securely access web UIs on an EMR cluster running in a private subnet. These options cover scenarios such as a connecting through a local network to your VPC or connecting through the Internet if your private subnet is not directly accessible.
Networking requirements in a private subnet
EMR depends on resources on the Internet and regional endpoints within the AWS cloud. Many customers analyze data in Amazon S3, Amazon Kinesis, and Amazon DynamoDB using EMR. Communication route to these services differ when your EMR cluster is in a private subnet.
Network routes can be configured from your private subnet to provide communication to these resources. A common setup is to use an S3 VPC endpoint for S3 and a NAT instance to access the public endpoints of other AWS services (for optional EMR functionality). Amazon VPC also offers a managed NAT gateway that can be configured to allow access to these public endpoints. The new VPC Subnets page in the EMR console makes the setting up S3 endpoints and NAT instances very easy.
In the above screenshot of the Create Cluster – Advanced Options page, you can also see that the private subnet does not have a route to an S3 Endpoint or a NAT instance; choose Add S3 Endpoint and NAT Instance to add these components. This takes you to the Configure Subnet page (which you can also access from the VPC Subnets page), which gives you options to configure these components directly before creating your cluster.
Connecting to a local network
When you connect a private subnet to your local network through a VPN connection or AWS Direct Connect, routing is configured so that communication spans the two networks. After the private connection is established, data scientists and engineers can connect directly from their local network to the private and public subnets in the VPC.
The diagram below shows an example networking flow in green:
How can you access the EMR web interfaces through the Internet when it’s launched in a private subnet? In this section, I’ll focus on how to stand up a bastion host with a public IP and configure dynamic port forwarding to enable this communication.
A bastion host provides a secure gateway between the public and private subnets. The bastion host has a public IP address that clients connect to and provides a proxy to the web interfaces on the master node of your EMR cluster in a private subnet.
The diagram below shows network flow through the bastion host to EMR.
Setting up the bastion host and dynamic port forwarding
The following section focuses on the steps needed to set up the dynamic port forwarding to enable the web interfaces running on your EMR cluster. The steps assume that a VPC has already been created with a private and public subnet. For more information, see Getting Started with Amazon VPC.
Launch the bastion host
An easy way to launch a Linux bastion host is to use the Amazon Linux AMI, and follow the instructions on how to launch a Linux EC2 instance. It’s a good idea to assign an Elastic IP address to the instance. Elastic IP addresses are persistent public IP addresses for an instance. This allows the public IP address to stay the same if the instance is stopped or started, or if the instance itself ever needs to be changed. For more information, see Step 4: Assign an Elastic IP Address to Your Instance.
Configure VPC security groups to allow incoming traffic from the bastion host
EMR uses separate master and slave (core and task) security groups for the respective node types in your cluster. For more information about how security groups work as a stateful firewall, see Security Groups for EC2-VPC.
Create a security group within the VPC prior to launching your EMR cluster.
Associate this security group with the bastion host. The security group rules need to be set to allow ingress SSH traffic on port 22 from your client’s IP CIDR ranges.
Create another new security group that allows ingress TCP traffic from the bastion host’s security group specified as a source.
When creating your EMR cluster, add these security groups to both master and slave nodes under the Additional security groups as part of the launch configuration for the EC2 instances running the EMR cluster.
In the following screenshot, the “Internal TCP Traffic” security group is applied to all the nodes in the EMR cluster, which allows the TCP traffic between the bastion host and to all nodes in your cluster.
In the following screenshot, the master node shows both security groups.
If your EMR cluster has already launched, you can modify the existing security groups with a new rule that allows ingress TCP traffic from the bastion host’s security group.
Connect to the bastion host with dynamic port forwarding
In the EMR console, note the DNS entry for the master node, which is a private IP address because the cluster is running in a private subnet.
Use SSH to connect to the bastion host running in the public subnet (through the bastion host’s Elastic IP address):
The SSH tunnel may time out in the above statement as it’s both setting up the tunnels and starting a secure shell. If you’d like to create the tunnel’s setup, you can use the following command:
You can choose the links to open the web UIs in your browser, or enter the internal master node IP address with the relevant port for a specific web UI.
Other patterns for access
Other patterns also support accessing the web interfaces running on an EMR cluster in a private subnet. These patterns have their trade-offs, but are important to mention.
HTTP(S) proxy in the public subnet
One option is to run a component that accepts HTTP(S) traffic in the public VPC subnet and proxies requests to your EMR cluster in the private subnet. An easy setup for this would be to launch an Elastic Load Balancing load balancer in the public subnets that points to web UI endpoints on your cluster’s master node. Other flavors of this include running a HTTP proxy (such as NGINX or HAProxy) on an EC2 instance in the public subnet. This is very easy to set up, and because users do not need an SSH key, it can be easier for them to connect. However, this can make it more difficult for fine-grained user access and this method does not support all web UIs on your cluster.
Client/desktops within AWS
Generally, the closer you can be to the data and processing, the more you can do. This also applies to clients accessing the web interface. An excellent option is for customers to use Amazon Workspaces as a virtual desktop, and access the web interfaces on EMR from their Workspace, which can be configured to access a private subnet.
Conclusion
You can now create secure SSH tunnels to access the various web interfaces like Hue and the YARN Resource Manager UI running on your EMR cluster, even when that cluster is in a private subnet.
If you have questions or suggestions, please leave a comment below.
Private subnets allow you to limit access to deployed components, and to control security and routing of the system. You can also use a private subnet to connect an on-premises local network to AWS through a VPN or AWS Direct Connect. Amazon EMR allows customers to launch clusters in a private subnet in Amazon VPC. For more information, see Select an Amazon VPC Subnet for the Cluster in the Amazon EMR Developer Guide.
When you launch an EMR cluster in a public subnet, the master node of the cluster has a public DNS, allowing you to create a SSH tunnel and securely access web applications like Hue, Apache Zeppelin, and other Hadoop web-UIs running on that node. However, when you launch an EMR cluster in a private subnet, your master node doesn’t have a public DNS, which can make it more difficult to access the web applications running on it.
In this post, I outline two possible solutions to securely access web UIs on an EMR cluster running in a private subnet. These options cover scenarios such as a connecting through a local network to your VPC or connecting through the Internet if your private subnet is not directly accessible.
Networking requirements in a private subnet
EMR depends on resources on the Internet and regional endpoints within the AWS cloud. Many customers analyze data in Amazon S3, Amazon Kinesis, and Amazon DynamoDB using EMR. Communication route to these services differ when your EMR cluster is in a private subnet.
Network routes can be configured from your private subnet to provide communication to these resources. A common setup is to use an S3 VPC endpoint for S3 and a NAT instance to access the public endpoints of other AWS services (for optional EMR functionality). Amazon VPC also offers a managed NAT gateway that can be configured to allow access to these public endpoints. The new VPC Subnets page in the EMR console makes the setting up S3 endpoints and NAT instances very easy.
In the above screenshot of the Create Cluster – Advanced Options page, you can also see that the private subnet does not have a route to an S3 Endpoint or a NAT instance; choose Add S3 Endpoint and NAT Instance to add these components. This takes you to the Configure Subnet page (which you can also access from the VPC Subnets page), which gives you options to configure these components directly before creating your cluster.
Connecting to a local network
When you connect a private subnet to your local network through a VPN connection or AWS Direct Connect, routing is configured so that communication spans the two networks. After the private connection is established, data scientists and engineers can connect directly from their local network to the private and public subnets in the VPC.
The diagram below shows an example networking flow in green:
How can you access the EMR web interfaces through the Internet when it’s launched in a private subnet? In this section, I’ll focus on how to stand up a bastion host with a public IP and configure dynamic port forwarding to enable this communication.
A bastion host provides a secure gateway between the public and private subnets. The bastion host has a public IP address that clients connect to and provides a proxy to the web interfaces on the master node of your EMR cluster in a private subnet.
The diagram below shows network flow through the bastion host to EMR.
Setting up the bastion host and dynamic port forwarding
The following section focuses on the steps needed to set up the dynamic port forwarding to enable the web interfaces running on your EMR cluster. The steps assume that a VPC has already been created with a private and public subnet. For more information, see Getting Started with Amazon VPC.
Launch the bastion host
An easy way to launch a Linux bastion host is to use the Amazon Linux AMI, and follow the instructions on how to launch a Linux EC2 instance. It’s a good idea to assign an Elastic IP address to the instance. Elastic IP addresses are persistent public IP addresses for an instance. This allows the public IP address to stay the same if the instance is stopped or started, or if the instance itself ever needs to be changed. For more information, see Step 4: Assign an Elastic IP Address to Your Instance.
Configure VPC security groups to allow incoming traffic from the bastion host
EMR uses separate master and slave (core and task) security groups for the respective node types in your cluster. For more information about how security groups work as a stateful firewall, see Security Groups for EC2-VPC.
Create a security group within the VPC prior to launching your EMR cluster.
Associate this security group with the bastion host. The security group rules need to be set to allow ingress SSH traffic on port 22 from your client’s IP CIDR ranges.
Create another new security group that allows ingress TCP traffic from the bastion host’s security group specified as a source.
When creating your EMR cluster, add these security groups to both master and slave nodes under the Additional security groups as part of the launch configuration for the EC2 instances running the EMR cluster.
In the following screenshot, the “Internal TCP Traffic” security group is applied to all the nodes in the EMR cluster, which allows the TCP traffic between the bastion host and to all nodes in your cluster.
In the following screenshot, the master node shows both security groups.
If your EMR cluster has already launched, you can modify the existing security groups with a new rule that allows ingress TCP traffic from the bastion host’s security group.
Connect to the bastion host with dynamic port forwarding
In the EMR console, note the DNS entry for the master node, which is a private IP address because the cluster is running in a private subnet.
Use SSH to connect to the bastion host running in the public subnet (through the bastion host’s Elastic IP address):
The SSH tunnel may time out in the above statement as it’s both setting up the tunnels and starting a secure shell. If you’d like to create the tunnel’s setup, you can use the following command:
You can choose the links to open the web UIs in your browser, or enter the internal master node IP address with the relevant port for a specific web UI.
Other patterns for access
Other patterns also support accessing the web interfaces running on an EMR cluster in a private subnet. These patterns have their trade-offs, but are important to mention.
HTTP(S) proxy in the public subnet
One option is to run a component that accepts HTTP(S) traffic in the public VPC subnet and proxies requests to your EMR cluster in the private subnet. An easy setup for this would be to launch an Elastic Load Balancing load balancer in the public subnets that points to web UI endpoints on your cluster’s master node. Other flavors of this include running a HTTP proxy (such as NGINX or HAProxy) on an EC2 instance in the public subnet. This is very easy to set up, and because users do not need an SSH key, it can be easier for them to connect. However, this can make it more difficult for fine-grained user access and this method does not support all web UIs on your cluster.
Client/desktops within AWS
Generally, the closer you can be to the data and processing, the more you can do. This also applies to clients accessing the web interface. An excellent option is for customers to use Amazon Workspaces as a virtual desktop, and access the web interfaces on EMR from their Workspace, which can be configured to access a private subnet.
Conclusion
You can now create secure SSH tunnels to access the various web interfaces like Hue and the YARN Resource Manager UI running on your EMR cluster, even when that cluster is in a private subnet.
If you have questions or suggestions, please leave a comment below.
This is a guest post by Pratik Verma, Founder and Chief Product Officer at BlueTalon. Leonid Fedotov, Senior Solution Architect at BlueTalon, also contributed to this post.
Amazon Elastic MapReduce (Amazon EMR) makes it easy to quickly and cost-effectively process vast amounts of data in the cloud. EMR gets used for log, financial, fraud, and bioinformatics analysis, as well as many other big data use cases. Often, the data used in these analyses, such as customer information, transaction history, and other proprietary data, is sensitive from a business perspective and may even be subject to regulatory compliance.
BlueTalon is a leading provider of data-centric security solutions for Hadoop, SQL and Big Data environments on-premises and in the cloud. BlueTalon keeps enterprises in control of their data by allowing them to give users access to the data they need, not a byte more. BlueTalon solution works across AWS data services like EMR, Redshift and RDS.
In this blog post, we show how organizations can use BlueTalon to mitigate the risks associated with their use of sensitive data while taking full advantage of EMR.
BlueTalon provides capabilities for data-centric security:
Audits of user activity using a context-rich trail of queries users run that hit sensitive fields.
Precise control over data that is specific for each user identity or business role and specific for the data resource at the file, folder, table, column, row, cell, or partial-cell level.
Secure use of business data in policy decisions for real-world requirements, while maintaining complex access scenarios and relationship between users and data.
Using BlueTalon to enforce data security
BlueTalon’s data-centric security solution has three main components: a UI to create rules and visualize real-time audit, a Policy Engine to make fast run-time authorization decisions, and a collection of Enforcement Points that transparently enforce the decisions made by the Policy Engine.
In a typical Hadoop cluster, users specify computations using SQL queries in Hive, scripts in Pig, or MapReduce programs. For applications accessing data via Hive, the BlueTalon Hive enforcement point transparently proxies HiveServer2 at the network level and provides policy-protected data. The BlueTalon Policy Engine makes sophisticated, fine-grained policy decisions based on user and content criteria in-memory at run-time by re-engineering SQL requests for Hive. With the query modification technique, BlueTalon is able to ensure that end users get the same data, whether raw data is coming from local HDFS or Amazon S3, and that only policy-compliant data is pulled from storage by Hive.
For direct HDFS access, end users connect to and receive policy-protected data via the BlueTalon HDFS enforcement point that transparently proxies HDFS NameNode at network level and the BlueTalon Policy Engine makes policy decisions based on user and content criteria in-memory at run-time to provide folder and file level control on HDFS. With the enforcement point for HDFS, BlueTalon ensures that end-users can’t get around its security by going to HDFS to obtain data not accessible via Hive.
Using enforcement points, BlueTalon provides the following access controls for your data:
Field protection: Fields can be denied without breaking the application. As an example, a blank value compatible with the id field is returned instead of revealing the id values as they are stored on disk.
Record protection: The result set can be filtered to return a subset of the data, even when the field used in the filter criteria is not in the result set. In this example, the user is able to see only the 2 records with the East Coast zip codes, compared to 10 records on disk.
Cell protection: A specific field value for a specific record can be protected. In this example, the user is able to see the birthdate value for ‘Joyce McDonald’ but not ‘Kelly Adams’. Here as well, the date field is compatible with the format expected by the application.
Partial cell protection: Even portions of a cell may be protected. In this example, the user sees the last four digits of a Social Security number, rather than the number being hidden entirely.
The BlueTalon Policy Engine integrates with Active Directory for authenticating end-user credentials and mapping identities to business roles. It enforces authorization so that Hive provides only policy-compliant data to end users.
Deploying BlueTalon with Amazon EMR
In the following sections, you’ll learn how to deploy BlueTalon with EMR and configure the policies. A typical deployment looks like the following:
Prerequisites
You need to contact [email protected] to obtain an evaluation copy, an Amazon EC2 Linux instance for installing BlueTalon, and an Amazon EMR cluster in the same VPC. BlueTalon recommends using an m3.large instance with CentOS.
On the EC2 instance, install the BlueTalon Policy Engine and Audit packages, available as rpm packages, using the yum commands:
> yum search bluetalon
bluetalon-audit.x86_64 : BlueTalon data security for Hadoop. bluetalon-enforcementpoint.x86_64 : BlueTalon data security for Hadoop. bluetalon-policy.x86_64 : BlueTalon data security for Hadoop.
> yum install bluetalon-audit –y
> yum install bluetalon-policy –y
Run the setup script
After the BlueTalon packages are installed, run the setup script to configure and turn on the run-time services and UI associated with the two packages.
> bluetalon-audit-setup
Starting bt-audit-server service: [ OK ] Starting bt-audit-zookeeper service: [ OK ] Starting bt-audit-kafka service: [ OK ] Starting bt-audit-activity-monitor service: [ OK ]
BlueTalon Audit Product is installed…. URL to access BlueTalon Audit UI ec2-0-0-0-0.us-west-2.compute.amazonaws.com:8112/BlueTalonAudit
Starting bt-postgresql service: [ OK ] Starting bt-policy-engine service: [ OK ] Starting bt-sql-hooks-vds service: [ OK ] Starting bt-webserver service: [ OK ] Starting bt-HeartBeatService service: [ OK ]
BlueTalon Data Security Product for Hadoop is installed…. You can create rules using the BlueTalon Policy UI URL to access BlueTalon Policy UI ec2-0-0-0-0.us-west-2.compute.amazonaws.com:8111/BlueTalonConfig
After the run time services associated with the BlueTalon packages have started, you should be able to connect to the BlueTalon Policy Management and User Audit interfaces as displayed below.
Installing enforcement points
Install and configure the BlueTalon enforcement point packages for Hive and HDFS NameNode on the master node of the EMR cluster using the following commands:
Starting bt-enforcement-point-demods service: [ OK ]
The arguments to this command include:
Hive: The type of enforcement point to configure. Options include Hive, HDFS, and PostgreSQL.
10011: The port on which the enforcement point listens.
HiveDS: The name of the data domain in the BlueTalon UI to associate with this enforcement point.
This command configures a Hive enforcement point for the local HiveServer2 and creates an iptables entry to re-route HiveServer2 traffic to the BlueTalon enforcement point first.
The following command restarts NameNode with the embedded BlueTalon enforcement point process:
> bluetalon-enforcementpoint-setup HDFS Stopping NameNode process: [ OK ] Starting NameNode process: [ OK ]
Adding data domains
Open the BlueTalon Policy Management UI using a browser and add Hive and HDFS as data domains so that BlueTalon can look up the data resources (databases, tables, columns, folders, files, etc.) to create data access rules. This requires connectivity information for HiveServer2 and NameNode.
For HiveServer2:
default: Database name associated with Hive warehouse. Typically, ‘default’.
10.0.0.1: Hostname of the machine where HiveServer2 is running. Typically, the DNS of the master node in Amazon EMR.
10000: Port that HiveServer2 is listening on. Typically, ‘10000’.
10011: Port on which the enforcement point listens. Typically, ‘10011’.
HiveDS: The name of the data domain in the BlueTalon UI to associate with this enforcement point.
No Login: Credentials for connecting to HiveServer2 if required.
For HDFS:
10.0.0.1: Hostname of the machine where NameNode is running.
8020: Port on which NameNode is listening. Typically, ‘8020’.
HDFSDS: The name of the data domain in the BlueTalon UI to associate with this enforcement point.
Adding user domains
Using the BlueTalon Policy Management UI, add the directory as a user domain so that BlueTalon can authenticate user credentials and look up the business roles to which a user belongs. For more information about obtaining connectivity information, see Viewing Directory Information.
10.0.0.1: Hostname of the machine where Active Directory is running.
389: Port of the machine where Active Directory is running. Typically, ‘389’.
10011: Port that the Enforcement Point listens on. Typically, ‘10011’.
CN=hadoopadmin: Credentials for bind and query to Active Directory.
Creating rules for specifying data access
Using the BlueTalon UI, you can create rules specifying which users can access what data. This can be done using the Add Rule button on the Policy tab to open a tray. Two examples are shown below.
On the left is an example of a row-level rule that restricts access for user ‘admin1’ to records in the ‘people’ table for locations in West Coast zip codes only. On the right is an example of a masking rule on a sensitive field, ‘accounts.ssn’, which masks it completely.
Deploying policies
After the rules are created, deploy the policy to the BlueTalon Policy Engine using the Deploy button from the Deploy tab. After it’s deployed, the policy and rules become effective on the Policy Engine.
The screenshots below show the data protection with BlueTalon by making queries through the ‘beeline’ client.
With BlueTalon
beeline> !connect jdbc:hive2://<hostname of masternode>:10011/default
Without BlueTalon
beeline> !connect jdbc:hive2://<hostname of masternode>:10000/default
With BlueTalon, the row level protection count of records is 249.
Without BlueTalon, the row level protection count of records is 2499.
With BlueTalon protection, field ssn is masked with ‘XXXX’.
Auditing access
All access through the BlueTalon enforcement points is authorized against the BlueTalon Policy Engine and audited. The audit can be visualized in the BlueTalon User Audit UI.
Create a policy for a user ‘alice’ that allows a read on table ‘books_1’ for books with ‘price’ less than $30.00, masks the field ‘publisher’ and denies the ‘id’ of book completely.
Run the query directly and through BlueTalon to see the effect of the policy rules created in BlueTalon.
Data as stored in Hive:
Result with BlueTalon protection:
Conclusion
BlueTalon enables organizations to protect access to data efficiently in HDFS or Amazon S3, allow users to get needed data, and leverage the full potential of Hadoop in a secure manner.
If you have questions or suggestions, please leave a comment below.
Love to work on open source? Check out EMR’s careers page.
—————————————————————-
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
