Security updates have been issued by Arch Linux (jbig2dec), Debian (libspring-ldap-java, sam2p, and xorg-server), Fedora (postgresql), openSUSE (cacti, cacti-spine), and Ubuntu (ldns and libraw).
Security updates have been issued by Debian (ldns and swauth), Fedora (kernel and postgresql), Mageia (botan, krb5, and sssd), and Ubuntu (apport, linux, linux-aws, linux-gke, linux-kvm, linux-raspi2, linux-snapdragon, linux, linux-raspi2, linux-hwe, linux-lts-xenial, procmail, and samba).
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-quicksight-update-geospatial-visualization-private-vpc-access-and-more/
We don’t often recognize or celebrate anniversaries at AWS. With nearly 100 services on our list, we’d be eating cake and drinking champagne several times a week. While that might sound like fun, we’d rather spend our working hours listening to customers and innovating. With that said, Amazon QuickSight has now been generally available for a little over a year and I would like to give you a quick update!
QuickSight in Action
Today, tens of thousands of customers (from startups to enterprises, in industries as varied as transportation, legal, mining, and healthcare) are using QuickSight to analyze and report on their business data.
Here are a couple of examples:
Gemini provides legal evidence procurement for California attorneys who represent injured workers. They have gone from creating custom reports and running one-off queries to creating and sharing dynamic QuickSight dashboards with drill-downs and filtering. QuickSight is used to track sales pipeline, measure order throughput, and to locate bottlenecks in the order processing pipeline.
Jivochat provides a real-time messaging platform to connect visitors to website owners. QuickSight lets them create and share interactive dashboards while also providing access to the underlying datasets. This has allowed them to move beyond the sharing of static spreadsheets, ensuring that everyone is looking at the same and is empowered to make timely decisions based on current data.
Transfix is a tech-powered freight marketplace that matches loads and increases visibility into logistics for Fortune 500 shippers in retail, food and beverage, manufacturing, and other industries. QuickSight has made analytics accessible to both BI engineers and non-technical business users. They scrutinize key business and operational metrics including shipping routes, carrier efficient, and process automation.
Looking Back / Looking Ahead
The feedback on QuickSight has been incredibly helpful. Customers tell us that their employees are using QuickSight to connect to their data, perform analytics, and make high-velocity, data-driven decisions, all without setting up or running their own BI infrastructure. We love all of the feedback that we get, and use it to drive our roadmap, leading to the introduction of over 40 new features in just a year. Here’s a summary:
- December 2016 – QuickSight Enterprise Edition.
- February 2017 – Support for Amazon Athena, Scheduled Refresh of SPICE data.
- April 2017 – KPI Charts, Export to CVS, AD Connector; Availability in US East (Ohio); Audit Logging with AWS CloudTrail.
- May 2017 – Presto and Apache Spark Connectors, Federated Single Sign-On Using SAML 2.0.
- June 2017 – Support for Amazon Redshift Spectrum, 1-Click Visualization of S3 Analytics.
- August 2017 – Availability in Asia Pacific (Singapore) and Asia Pacific (Sydney).
- September 2017 – Search & Filter Groups, Amazon S3 Analytics Connector, Create Analyses From Dashboards, Import Custom Date Formats.
- October 2017 – Combo Charts, Row-Level Security.
Looking forward, we are watching an interesting trend develop within our customer base. As these customers take a close look at how they analyze and report on data, they are realizing that a serverless approach offers some tangible benefits. They use Amazon Simple Storage Service (S3) as a data lake and query it using a combination of QuickSight and Amazon Athena, giving them agility and flexibility without static infrastructure. They also make great use of QuickSight’s dashboards feature, monitoring business results and operational metrics, then sharing their insights with hundreds of users. You can read Building a Serverless Analytics Solution for Cleaner Cities and review Serverless Big Data Analytics using Amazon Athena and Amazon QuickSight if you are interested in this approach.
New Features and Enhancements
We’re still doing our best to listen and to learn, and to make sure that QuickSight continues to meet your needs. I’m happy to announce that we are making seven big additions today:
Geospatial Visualization – You can now create geospatial visuals on geographical data sets.
Private VPC Access – You can now sign up to access a preview of a new feature that allows you to securely connect to data within VPCs or on-premises, without the need for public endpoints.
Flat Table Support – In addition to pivot tables, you can now use flat tables for tabular reporting. To learn more, read about Using Tabular Reports.
Calculated SPICE Fields – You can now perform run-time calculations on SPICE data as part of your analysis. Read Adding a Calculated Field to an Analysis for more information.
Wide Table Support – You can now use tables with up to 1000 columns.
Other Buckets – You can summarize the long tail of high-cardinality data into buckets, as described in Working with Visual Types in Amazon QuickSight.
HIPAA Compliance – You can now run HIPAA-compliant workloads on QuickSight.
Everyone seems to want this feature! You can now take data that contains a geographic identifier (country, city, state, or zip code) and create beautiful visualizations with just a few clicks. QuickSight will geocode the identifier that you supply, and can also accept lat/long map coordinates. You can use this feature to visualize sales by state, map stores to shipping destinations, and so forth. Here’s a sample visualization:
Private VPC Access Preview
If you have data in AWS (perhaps in Amazon Redshift, Amazon Relational Database Service (RDS), or on EC2) or on-premises in Teradata or SQL Server on servers without public connectivity, this feature is for you. Private VPC Access for QuickSight uses an Elastic Network Interface (ENI) for secure, private communication with data sources in a VPC. It also allows you to use AWS Direct Connect to create a secure, private link with your on-premises resources. Here’s what it looks like:
If you are ready to join the preview, you can sign up today.
The AWS US East/West Region has received a Provisional Authority to Operate (P-ATO) from the Joint Authorization Board (JAB) at the Federal Risk and Authorization Management Program (FedRAMP) Moderate baseline.
Though AWS has maintained an AWS US East/West Region Agency-ATO since early 2013, this announcement represents AWS’s carefully deliberated move to the JAB for the centralized maintenance of our P-ATO for 10 services already authorized. This also includes the addition of 10 new services to our FedRAMP program (see the complete list of services below). This doubles the number of FedRAMP Moderate services available to our customers to enable increased use of the cloud and support modernized IT missions. Our public sector customers now can leverage this FedRAMP P-ATO as a baseline for their own authorizations and look to the JAB for centralized Continuous Monitoring reporting and updates. In a significant enhancement for our partners that build their solutions on the AWS US East/West Region, they can now achieve FedRAMP JAB P-ATOs of their own for their Platform as a Service (PaaS) and Software as a Service (SaaS) offerings.
In line with FedRAMP security requirements, our independent FedRAMP assessment was completed in partnership with a FedRAMP accredited Third Party Assessment Organization (3PAO) on our technical, management, and operational security controls to validate that they meet or exceed FedRAMP’s Moderate baseline requirements. Effective immediately, you can begin leveraging this P-ATO for the following 20 services in the AWS US East/West Region:
- Amazon Aurora (MySQL)*
- Amazon CloudWatch Logs*
- Amazon DynamoDB
- Amazon Elastic Block Store
- Amazon Elastic Compute Cloud
- Amazon EMR*
- Amazon Glacier*
- Amazon Kinesis Streams*
- Amazon RDS (MySQL, Oracle, Postgres*)
- Amazon Redshift
- Amazon Simple Notification Service*
- Amazon Simple Queue Service*
- Amazon Simple Storage Service
- Amazon Simple Workflow Service*
- Amazon Virtual Private Cloud
- AWS CloudFormation*
- AWS CloudTrail*
- AWS Identity and Access Management
- AWS Key Management Service
- Elastic Load Balancing
* Services with first-time FedRAMP Moderate authorizations
We continue to work with the FedRAMP Project Management Office (PMO), other regulatory and compliance bodies, and our customers and partners to ensure that we are raising the bar on our customers’ security and compliance needs.
To learn more about how AWS helps customers meet their security and compliance requirements, see the AWS Compliance website. To learn about what other public sector customers are doing on AWS, see our Government, Education, and Nonprofits Case Studies and Customer Success Stories. To review the public posting of our FedRAMP authorizations, see the FedRAMP Marketplace.
– Chris Gile, Senior Manager, AWS Public Sector Risk and Compliance
Post Syndicated from Peter Pereira original https://aws.amazon.com/blogs/security/how-aws-managed-microsoft-ad-helps-to-simplify-the-deployment-and-improve-the-security-of-active-directory-integrated-net-applications/
Companies using .NET applications to access sensitive user information, such as employee salary, Social Security Number, and credit card information, need an easy and secure way to manage access for users and applications.
For example, let’s say that your company has a .NET payroll application. You want your Human Resources (HR) team to manage and update the payroll data for all the employees in your company. You also want your employees to be able to see their own payroll information in the application. To meet these requirements in a user-friendly and secure way, you want to manage access to the .NET application by using your existing Microsoft Active Directory identities. This enables you to provide users with single sign-on (SSO) access to the .NET application and to manage permissions using Active Directory groups. You also want the .NET application to authenticate itself to access the database, and to limit access to the data in the database based on the identity of the application user.
Microsoft Active Directory supports these requirements through group Managed Service Accounts (gMSAs) and Kerberos constrained delegation (KCD). AWS Directory Service for Microsoft Active Directory, also known as AWS Managed Microsoft AD, enables you to manage gMSAs and KCD through your administrative account, helping you to migrate and develop .NET applications that need these native Active Directory features.
In this blog post, I give an overview of how to use AWS Managed Microsoft AD to manage gMSAs and KCD and demonstrate how you can configure a gMSA and KCD in six steps for a .NET application:
- Create your AWS Managed Microsoft AD.
- Create your Amazon RDS for SQL Server database.
- Create a gMSA for your .NET application.
- Deploy your .NET application.
- Configure your .NET application to use the gMSA.
- Configure KCD for your .NET application.
The following diagram shows the components of a .NET application that uses Amazon RDS for SQL Server with a gMSA and KCD. The diagram also illustrates authentication and access and is numbered to show the six key steps required to use a gMSA and KCD. To deploy this solution, the AWS Managed Microsoft AD directory must be in the same Amazon Virtual Private Cloud (VPC) as RDS for SQL Server. For this example, my company name is Example Corp., and my directory uses the domain name,
Deploy the solution
The following six steps (numbered to correlate with the preceding diagram) walk you through configuring and using a gMSA and KCD.
1. Create your AWS Managed Microsoft AD directory
2. Create your Amazon RDS for SQL Server database
Using the RDS console, create your Amazon RDS for SQL Server database instance in the same Amazon VPC where your directory is running, and enable Windows Authentication. To enable Windows Authentication, select your directory in the Microsoft SQL Server Windows Authentication section in the Configure Advanced Settings step of the database creation workflow (see the following screenshot).
In my example, I create my Amazon RDS for SQL Server
db-example database, and enable Windows Authentication to allow my
db-example database to authenticate against my
3. Create a gMSA for your .NET application
Now that you have deployed your directory, database, and application, you can create a gMSA for your .NET application.
To perform the next steps, you must install the Active Directory administration tools on a Windows server that is joined to your AWS Managed Microsoft AD directory domain. If you do not have a Windows server joined to your directory domain, you can deploy a new Amazon EC2 for Microsoft Windows Server instance and join it to your directory domain.
To create a gMSA for your .NET application:
- Log on to the instance on which you installed the Active Directory administration tools by using a user that is a member of the
Adminssecurity group or the
Managed Service Accounts Adminssecurity group in your organizational unit (OU). For my example, I use the
Adminuser in the
- Identify which .NET application servers (hosts) will run your .NET application. Create a new security group in your OU and add your .NET application servers as members of this new group. This allows a group of application servers to use a single gMSA, instead of creating one gMSA for each server. In my example, I create a group,
App_server_grp, in my
exampleOU. I also add
Appserver1, which is my .NET application server computer name, as a member of this new group.
- Create a gMSA in your directory by running Windows PowerShell from the Start menu. The basic syntax to create the gMSA at the Windows PowerShell command prompt follows.
In my example, the
example.com, and the
PrincipalsAllowedToRetrieveManagedPasswordis the recently created security group,
To confirm you created the gMSA, you can run the
Get-ADServiceAccountcommand from the PowerShell command prompt.
You also can confirm you created the gMSA by opening the Active Directory Users and Computers utility located in your Administrative Tools folder, expand the domain (
example.comin my case), and expand the Managed Service Accounts folder.
4. Deploy your .NET application
Deploy your .NET application on IIS on Amazon EC2 for Windows Server instances. For this step, I assume you are the application’s expert and already know how to deploy it. Make sure that all of your instances are joined to your directory.
5. Configure your .NET application to use the gMSA
You can configure your .NET application to use the gMSA to enforce strong password security policy and ensure password rotation of your service account. This helps to improve the security and simplify the management of your .NET application. Configure your .NET application in two steps:
- Grant to gMSA the required permissions to run your .NET application in the respective application folders. This is a critical step because when you change the application pool identity account to use gMSA, downtime can occur if the gMSA does not have the application’s required permissions. Therefore, make sure you first test the configurations in your development and test environments.
- Configure your application pool identity on IIS to use the gMSA as the service account. When you configure a gMSA as the service account, you include the
$at the end of the gMSA name. You do not need to provide a password because AWS Managed Microsoft AD automatically creates and rotates the password. In my example, my service account is
gMSAexample$, as shown in the following screenshot.
You have completed all the steps to use gMSA to create and rotate your .NET application service account password! Now, you will configure KCD for your .NET application.
6. Configure KCD for your .NET application
You now are ready to allow your .NET application to have access to other services by using the user identity’s permissions instead of the application service account’s permissions. Note that KCD and gMSA are independent features, which means you do not have to create a gMSA to use KCD. For this example, I am using both features to show how you can use them together. To configure a regular service account such as a user or local built-in account, see the Kerberos constrained delegation with ASP.NET blog post on MSDN.
In my example, my goal is to delegate to the
gMSAexample account the ability to enforce the user’s permissions to my
db-example SQL Server database, instead of the
gMSAexample account’s permissions. For this, I have to update the
msDS-AllowedToDelegateTo gMSA attribute. The value for this attribute is the service principal name (SPN) of the service instance that you are targeting, which in this case is the
db-example Amazon RDS for SQL Server database.
The SPN format for the
msDS-AllowedToDelegateTo attribute is a combination of the service class, the Kerberos authentication endpoint, and the port number. The Amazon RDS for SQL Server Kerberos authentication endpoint format is
[database_name].[domain_name]. The value for my
msDS-AllowedToDelegateTo attribute is
1433 are the SQL Server Database service class and port number standards, respectively.
Follow these steps to perform the
msDS-AllowedToDelegateTo gMSA attribute configuration:
- Log on to your Active Directory management instance with a user identity that is a member of the
Kerberos Delegation Adminssecurity group. In this case, I will use
- Open the Active Directory Users and Groups utility located in your Administrative Tools folder, choose View, and then choose Advanced Features.
- Expand your domain name (
example.comin this example), and then choose the Managed Service Accounts security group. Right-click the gMSA account for the application pool you want to enable for Kerberos delegation, choose Properties, and choose the Attribute Editor tab.
- Search for the
msDS-AllowedToDelegateToattribute on the Attribute Editor tab and choose Edit.
- Enter the
MSSQLSvc/db-example.example.com:1433value and choose Add.
- Choose OK and Apply, and your KCD configuration is complete.
Congratulations! At this point, your application is using a gMSA rather than an embedded static user identity and password, and the application is able to access SQL Server using the identity of the application user. The gMSA eliminates the need for you to rotate the application’s password manually, and it allows you to better scope permissions for the application. When you use KCD, you can enforce access to your database consistently based on user identities at the database level, which prevents improper access that might otherwise occur because of an application error.
In this blog post, I demonstrated how to simplify the deployment and improve the security of your .NET application by using a group Managed Service Account and Kerberos constrained delegation with your AWS Managed Microsoft AD directory. I also outlined the main steps to get your .NET environment up and running on a managed Active Directory and SQL Server infrastructure. This approach will make it easier for you to build new .NET applications in the AWS Cloud or migrate existing ones in a more secure way.
For additional information about using group Managed Service Accounts and Kerberos constrained delegation with your AWS Managed Microsoft AD directory, see the AWS Directory Service documentation.
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-ec2-update-x1e-instances-in-five-more-sizes-and-a-stronger-sla/
Earlier this year we launched the x1e.32xlarge instances in four AWS Regions with 4 TB of memory. Today, two months after that launch, customers are using these instances to run high-performance relational and NoSQL databases, in-memory databases, and other enterprise applications that are able to take advantage of large amounts of memory.
Five More Sizes of X1e
I am happy to announce that we are extending the memory-optimized X1e family with five additional instance sizes. Here’s the lineup:
|Model||vCPUs||Memory (GiB)||SSD Storage (GB)||Networking Performance|
|x1e.xlarge||4||122||120||Up to 10 Gbps
|x1e.2xlarge||8||244||240||Up to 10 Gbps|
|x1e.4xlarge||16||488||480||Up to 10 Gbps|
|x1e.8xlarge||32||976||960||Up to 10 Gbps|
The instances are powered by quad socket Intel® Xeon® E7 8880 processors running at 2.3 GHz, with large L3 caches and plenty of memory bandwidth. ENA networking and EBS optimization are standard, with up to 14 Gbps of dedicated throughput (depending on instance size) to EBS.
As part of today’s launch we are also making all sizes of X1e available in the Asia Pacific (Sydney) Region. This means that you can now launch them in On-Demand and Reserved Instance form in the US East (Northern Virginia), US West (Oregon), EU (Ireland), Asia Pacific (Tokyo), and Asia Pacific (Sydney) Regions.
Stronger EC2 SLA
I also have another piece of good news!
Effective immediately, we are increasing the EC2 Service Level Agreement (SLA) for both EC2 and EBS to 99.99%, for all regions and for all AWS customers. This change was made possible by our continuous investment in infrastructure and quality of service, along with our focus on operational excellence.
Contributed by Otavio Ferreira, Manager, Software Development, AWS Messaging
Like other developers around the world, you may be tackling increasingly complex business problems. A key success factor, in that case, is the ability to break down a large project scope into smaller, more manageable components. A service-oriented architecture guides you toward designing systems as a collection of loosely coupled, independently scaled, and highly reusable services. Microservices take this even further. To improve performance and scalability, they promote fine-grained interfaces and lightweight protocols.
However, the communication among isolated microservices can be challenging. Services are often deployed onto independent servers and don’t share any compute or storage resources. Also, you should avoid hard dependencies among microservices, to preserve maintainability and reusability.
If you apply the pub/sub design pattern, you can effortlessly decouple and independently scale out your microservices and serverless architectures. A pub/sub messaging service, such as Amazon SNS, promotes event-driven computing that statically decouples event publishers from subscribers, while dynamically allowing for the exchange of messages between them. An event-driven architecture also introduces the responsiveness needed to deal with complex problems, which are often unpredictable and asynchronous.
What is event-driven computing?
Given the context of microservices, event-driven computing is a model in which subscriber services automatically perform work in response to events triggered by publisher services. This paradigm can be applied to automate workflows while decoupling the services that collectively and independently work to fulfil these workflows. Amazon SNS is an event-driven computing hub, in the AWS Cloud, that has native integration with several AWS publisher and subscriber services.
Which AWS services publish events to SNS natively?
Several AWS services have been integrated as SNS publishers and, therefore, can natively trigger event-driven computing for a variety of use cases. In this post, I specifically cover AWS compute, storage, database, and networking services, as depicted below.
- Auto Scaling: Helps you ensure that you have the correct number of Amazon EC2 instances available to handle the load for your application. You can configure Auto Scaling lifecycle hooks to trigger events, as Auto Scaling resizes your EC2 cluster.As an example, you may want to warm up the local cache store on newly launched EC2 instances, and also download log files from other EC2 instances that are about to be terminated. To make this happen, set an SNS topic as your Auto Scaling group’s notification target, then subscribe two Lambda functions to this SNS topic. The first function is responsible for handling scale-out events (to warm up cache upon provisioning), whereas the second is in charge of handling scale-in events (to download logs upon termination).
- AWS Elastic Beanstalk: An easy-to-use service for deploying and scaling web applications and web services developed in a number of programming languages. You can configure event notifications for your Elastic Beanstalk environment so that notable events can be automatically published to an SNS topic, then pushed to topic subscribers.As an example, you may use this event-driven architecture to coordinate your continuous integration pipeline (such as Jenkins CI). That way, whenever an environment is created, Elastic Beanstalk publishes this event to an SNS topic, which triggers a subscribing Lambda function, which then kicks off a CI job against your newly created Elastic Beanstalk environment.
- Elastic Load Balancing: Automatically distributes incoming application traffic across Amazon EC2 instances, containers, or other resources identified by IP addresses.You can configure CloudWatch alarms on Elastic Load Balancing metrics, to automate the handling of events derived from Classic Load Balancers. As an example, you may leverage this event-driven design to automate latency profiling in an Amazon ECS cluster behind a Classic Load Balancer. In this example, whenever your ECS cluster breaches your load balancer latency threshold, an event is posted by CloudWatch to an SNS topic, which then triggers a subscribing Lambda function. This function runs a task on your ECS cluster to trigger a latency profiling tool, hosted on the cluster itself. This can enhance your latency troubleshooting exercise by making it timely.
- Amazon S3: Object storage built to store and retrieve any amount of data.You can enable S3 event notifications, and automatically get them posted to SNS topics, to automate a variety of workflows. For instance, imagine that you have an S3 bucket to store incoming resumes from candidates, and a fleet of EC2 instances to encode these resumes from their original format (such as Word or text) into a portable format (such as PDF).In this example, whenever new files are uploaded to your input bucket, S3 publishes these events to an SNS topic, which in turn pushes these messages into subscribing SQS queues. Then, encoding workers running on EC2 instances poll these messages from the SQS queues; retrieve the original files from the input S3 bucket; encode them into PDF; and finally store them in an output S3 bucket.
- Configuring Amazon S3 Event Notifications
- Configuring Amazon S3 Buckets for Amazon SNS Notifications (Walkthrough)
- Messaging Fan-out Pattern for Serverless Architectures Using Amazon SNS (Multimedia Encoding Example)
- Amazon EFS: Provides simple and scalable file storage, for use with Amazon EC2 instances, in the AWS Cloud.You can configure CloudWatch alarms on EFS metrics, to automate the management of your EFS systems. For example, consider a highly parallelized genomics analysis application that runs against an EFS system. By default, this file system is instantiated on the “General Purpose” performance mode. Although this performance mode allows for lower latency, it might eventually impose a scaling bottleneck. Therefore, you may leverage an event-driven design to handle it automatically.Basically, as soon as the EFS metric “Percent I/O Limit” breaches 95%, CloudWatch could post this event to an SNS topic, which in turn would push this message into a subscribing Lambda function. This function automatically creates a new file system, this time on the “Max I/O” performance mode, then switches the genomics analysis application to this new file system. As a result, your application starts experiencing higher I/O throughput rates.
- Amazon Glacier: A secure, durable, and low-cost cloud storage service for data archiving and long-term backup.You can set a notification configuration on an Amazon Glacier vault so that when a job completes, a message is published to an SNS topic. Retrieving an archive from Amazon Glacier is a two-step asynchronous operation, in which you first initiate a job, and then download the output after the job completes. Therefore, SNS helps you eliminate polling your Amazon Glacier vault to check whether your job has been completed, or not. As usual, you may subscribe SQS queues, Lambda functions, and HTTP endpoints to your SNS topic, to be notified when your Amazon Glacier job is done.
- AWS Snowball: A petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data.You can leverage Snowball notifications to automate workflows related to importing data into and exporting data from AWS. More specifically, whenever your Snowball job status changes, Snowball can publish this event to an SNS topic, which in turn can broadcast the event to all its subscribers.As an example, imagine a Geographic Information System (GIS) that distributes high-resolution satellite images to users via Web browser. In this example, the GIS vendor could capture up to 80 TB of satellite images; create a Snowball job to import these files from an on-premises system to an S3 bucket; and provide an SNS topic ARN to be notified upon job status changes in Snowball. After Snowball changes the job status from “Importing” to “Completed”, Snowball publishes this event to the specified SNS topic, which delivers this message to a subscribing Lambda function, which finally creates a CloudFront web distribution for the target S3 bucket, to serve the images to end users.
- Amazon RDS: Makes it easy to set up, operate, and scale a relational database in the cloud.RDS leverages SNS to broadcast notifications when RDS events occur. As usual, these notifications can be delivered via any protocol supported by SNS, including SQS queues, Lambda functions, and HTTP endpoints.As an example, imagine that you own a social network website that has experienced organic growth, and needs to scale its compute and database resources on demand. In this case, you could provide an SNS topic to listen to RDS DB instance events. When the “Low Storage” event is published to the topic, SNS pushes this event to a subscribing Lambda function, which in turn leverages the RDS API to increase the storage capacity allocated to your DB instance. The provisioning itself takes place within the specified DB maintenance window.
- Amazon ElastiCache: A web service that makes it easy to deploy, operate, and scale an in-memory data store or cache in the cloud.ElastiCache can publish messages using Amazon SNS when significant events happen on your cache cluster. This feature can be used to refresh the list of servers on client machines connected to individual cache node endpoints of a cache cluster. For instance, an ecommerce website fetches product details from a cache cluster, with the goal of offloading a relational database and speeding up page load times. Ideally, you want to make sure that each web server always has an updated list of cache servers to which to connect.To automate this node discovery process, you can get your ElastiCache cluster to publish events to an SNS topic. Thus, when ElastiCache event “AddCacheNodeComplete” is published, your topic then pushes this event to all subscribing HTTP endpoints that serve your ecommerce website, so that these HTTP servers can update their list of cache nodes.
- Amazon Redshift: A fully managed data warehouse that makes it simple to analyze data using standard SQL and BI (Business Intelligence) tools.Amazon Redshift uses SNS to broadcast relevant events so that data warehouse workflows can be automated. As an example, imagine a news website that sends clickstream data to a Kinesis Firehose stream, which then loads the data into Amazon Redshift, so that popular news and reading preferences might be surfaced on a BI tool. At some point though, this Amazon Redshift cluster might need to be resized, and the cluster enters a ready-only mode. Hence, this Amazon Redshift event is published to an SNS topic, which delivers this event to a subscribing Lambda function, which finally deletes the corresponding Kinesis Firehose delivery stream, so that clickstream data uploads can be put on hold.At a later point, after Amazon Redshift publishes the event that the maintenance window has been closed, SNS notifies a subscribing Lambda function accordingly, so that this function can re-create the Kinesis Firehose delivery stream, and resume clickstream data uploads to Amazon Redshift.
- AWS DMS: Helps you migrate databases to AWS quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database.DMS also uses SNS to provide notifications when DMS events occur, which can automate database migration workflows. As an example, you might create data replication tasks to migrate an on-premises MS SQL database, composed of multiple tables, to MySQL. Thus, if replication tasks fail due to incompatible data encoding in the source tables, these events can be published to an SNS topic, which can push these messages into a subscribing SQS queue. Then, encoders running on EC2 can poll these messages from the SQS queue, encode the source tables into a compatible character set, and restart the corresponding replication tasks in DMS. This is an event-driven approach to a self-healing database migration process.
- Amazon Route 53: A highly available and scalable cloud-based DNS (Domain Name System). Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources.You can set CloudWatch alarms and get automated Amazon SNS notifications when the status of your Route 53 health check changes. As an example, imagine an online payment gateway that reports the health of its platform to merchants worldwide, via a status page. This page is hosted on EC2 and fetches platform health data from DynamoDB. In this case, you could configure a CloudWatch alarm for your Route 53 health check, so that when the alarm threshold is breached, and the payment gateway is no longer considered healthy, then CloudWatch publishes this event to an SNS topic, which pushes this message to a subscribing Lambda function, which finally updates the DynamoDB table that populates the status page. This event-driven approach avoids any kind of manual update to the status page visited by merchants.
- AWS Direct Connect (AWS DX): Makes it easy to establish a dedicated network connection from your premises to AWS, which can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections.You can monitor physical DX connections using CloudWatch alarms, and send SNS messages when alarms change their status. As an example, when a DX connection state shifts to 0 (zero), indicating that the connection is down, this event can be published to an SNS topic, which can fan out this message to impacted servers through HTTP endpoints, so that they might reroute their traffic through a different connection instead. This is an event-driven approach to connectivity resilience.
In addition to SNS, event-driven computing is also addressed by Amazon CloudWatch Events, which delivers a near real-time stream of system events that describe changes in AWS resources. With CloudWatch Events, you can route each event type to one or more targets, including:
- SNS topics
- Amazon SQS queues
- Amazon EC2 instances
- Amazon ECS tasks
- Amazon Kinesis Streams
- Amazon Kinesis Firehose delivery streams
- AWS Lambda functions
- AWS Step Functions state machines
- AWS CodePipeline pipelines
Many AWS services publish events to CloudWatch. As an example, you can get CloudWatch Events to capture events on your ETL (Extract, Transform, Load) jobs running on AWS Glue and push failed ones to an SQS queue, so that you can retry them later.
Amazon SNS is a pub/sub messaging service that can be used as an event-driven computing hub to AWS customers worldwide. By capturing events natively triggered by AWS services, such as EC2, S3 and RDS, you can automate and optimize all kinds of workflows, namely scaling, testing, encoding, profiling, broadcasting, discovery, failover, and much more. Business use cases presented in this post ranged from recruiting websites, to scientific research, geographic systems, social networks, retail websites, and news portals.
Start now by visiting Amazon SNS in the AWS Management Console, or by trying the AWS 10-Minute Tutorial, Send Fan-out Event Notifications with Amazon SNS and Amazon SQS.
Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2017/11/16/staying-busy-between-code-pushes/
Staying Busy Between Code Pushes.
Maintaining a regular cadence of pushing out releases, adding new features, implementing bug fixes and staying on top of support requests is important for any software to thrive; but especially important for open source software due to its rapid pace. It’s easy to lose yourself in code and forget that events are happening all the time – in every corner of the world, where we can learn, share knowledge, and meet like-minded individuals to build better software, together. There are so many amazing events we’d like to participate in, but there simply isn’t enough time (or budget) to fit them all in. Here’s what we’ve been up to recently; between code pushes.
Øredev Conference | Malmö, Sweden: Øredev is one of the biggest developer conferences in Scandinavia, and Grafana Labs jumped at the chance to be a part of it. In early November, Grafana Labs Principal Developer, Carl Bergquist, gave a great talk on “Monitoring for Everyone”, which discussed the concepts of monitoring and why everyone should care, different ways to monitor your systems, extending your monitoring to containers and microservices, and finally what to monitor and alert on. Watch the video of his talk below.
InfluxDays | San Francisco, CA: Dan Cech, our Director of Platform Services, spoke at InfluxDays in San Francisco on Nov 14, and Grafana Labs sponsored the event. InfluxDB is a popular data source for Grafana, so we wanted to connect to the InfluxDB community and show them how to get the most out of their data. Dan discussed building dashboards, choosing the best panels for your data, setting up alerting in Grafana and a few sneak peeks of the upcoming Grafana 5.0. The video of his talk is forthcoming, but Dan has made his presentation available.
PromCon | Munich, Germany: PromCon is the Prometheus-focused event of the year. In August, Carl Bergquist, had the opportunity to speak at PromCon and take a deep dive into Grafana and Prometheus. Many attendees at PromCon were already familiar with Grafana, since it’s the default dashboard tool for Prometheus, but Carl had a trove of tricks and optimizations to share. He also went over some major changes and what we’re currently working on.
CNCF Meetup | New York, NY: Grafana Co-founder and CEO, Raj Dutt, particpated in a panel discussion with the folks of Packet and the Cloud Native Computing Foundation. The discussion focused on the success stories, failures, rationales and in-the-trenches challenges when running cloud native in private or non “public cloud” datacenters (bare metal, colocation, private clouds, special hardware or networking setups, compliance and security-focused deployments).
Percona Live | Dublin: Daniel Lee traveled to Dublin, Ireland this fall to present at the database conference Percona Live. There he showed the new native MySQL support, along with a number of upcoming features in Grafana 5.0. His presentation is available to download.
Big Monitoring Meetup | St. Petersburg, Russian Federation: Alexander Zobnin, our developer located in Russia, is the primary maintainer of our popular Zabbix plugin. He attended the Big Monitoring Meetup to discuss monitoring, Grafana dashboards and democratizing metrics.
Why observability matters – now and in the future | Webinar: Our own Carl Bergquist and Neil Gehani, Director of Product at Weaveworks, to discover best practices on how to get started with monitoring both your application and infrastructure. Start capturing metrics that matter, aggregate and visualize them in a useful way that allows for identifying bottlenecks and proactively preventing incidents. View Carl’s presentation.
We’re going to maintain this momentum with a number of upcoming events, and hope you can join us.
- Cloud native applications and computing
- Central orchestration processing
- And more.
Nov. 29, 1pm EST: Check out how you can use popular open source projects, for performance monitoring of your Infrastructure, Application, and Cloud faster, easier, and to scale. In this webinar, Daniel Lee from Grafana Labs, and Chris Churilo from InfluxData, will provide you with step by step instruction from download & configure, to collecting metrics and building dashboards and alerts.
Last, but certainly not least, the next GrafanaCon is right around the corner. GrafanaCon EU (to be held in Amsterdam, Netherlands, March 1-2. 2018),is a two-day event with talks centered around Grafana and the surrounding ecosystem. In addition to the latest features and functionality of Grafana, you can expect to see and hear from members of the monitoring community like Graphite, Prometheus, InfluxData, Elasticsearch Kubernetes, and more. Head to grafanacon.org to see the latest speakers confirmed. We have speakers from Automattic, Bloomberg, CERN, Fastly, Tinder and more!
The Grafana Labs team is spread across the globe. Having a “post-geographic” company structure give us the opportunity to take part in events wherever they may be held in the world. As our team continues to grow, we hope to take part in even more events, and hope you can find the time to join us.
Security updates have been issued by Arch Linux (konversation), Debian (graphicsmagick and konversation), Fedora (git-annex, ImageMagick, kernel, and libgcrypt), Oracle (kernel), Red Hat (httpd), SUSE (firefox, nss), and Ubuntu (perl and postgresql-9.3, postgresql-9.5, postgresql-9.6).
Security updates have been issued by Debian (graphicsmagick, imagemagick, mupdf, postgresql-common, ruby2.3, and wordpress), Fedora (tomcat), Gentoo (cacti, chromium, eGroupWare, hostapd, imagemagick, libXfont2, lxc, mariadb, vde, wget, and xorg-server), Mageia (flash-player-plugin and libjpeg), openSUSE (ansible, ImageMagick, java-1_8_0-openjdk, krb5, redis, shadow, virtualbox, and webkit2gtk3), Red Hat (rh-eclipse46-jackson-databind and rh-eclipse47-jackson-databind), SUSE (java-1_8_0-openjdk, mysql, openssl, and storm, storm-kit), and Ubuntu (perl).
Security updates have been issued by Arch Linux (lib32-openssl, libextractor, postgresql, and postgresql-old-upgrade), Debian (bchunk, postgresql-9.4, postgresql-9.6, postgresql-common, roundcube, and tomcat7), Gentoo (libxml2), SUSE (kvm, openssl1, and qemu), and Ubuntu (postgresql-common).
Security updates have been issued by Debian (apr, apr-util, chromium-browser, libpam4j, and mupdf), Fedora (community-mysql and modulemd), Mageia (git), openSUSE (libsass, libwpd, qemu, sssd, and SuSEfirewall2), Red Hat (Red Hat JBoss Enterprise Application Platform and Red Hat JBoss Enterprise Application Platform 7.0), SUSE (qemu), and Ubuntu (openssl).
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-price-list-api-update-new-query-and-metadata-functions/
The original AWS Price List API, as described in New – AWS Price List API, gave you access to prices in JSON and CSV form by way of structured URLs. While this worked well for some types of cost management tools, the size and complexity of the files made them difficult to download and tedious to parse. Today we are updating the API by adding new functions that allow you to perform fine-grained price queries that return only the prices that you need. This will allow you to make use of the prices in mobile and browser-based applications.
Here are the new functions:
DescribeServices – Returns sets of attribute keys that are used to define products within a service. For example, the keys returned for EC2 will include physicalProcessor, memory, operatingSystem, location, and tenancy.
GetAttributeValues – Returns all of the allowable values for a given attribute key. For example, values for the operatingSystem key include Windows, RHEL, Linux, and SUSE; values for the location key include US East (N. Virginia) and Asia Pacific (Mumbai).
GetProducts -Returns all of the products, along with their public prices, that match a filter expression based on service name and attribute value.
Here’s how I list all of the services and attributes:
The output starts like this:
Here’s how I get all of the values for all of EC2’s pricing attributes:
The output starts like this:
And here’s how I use the service name and attribute values to obtain price listings for EC2 instances with 64 vCPUs, 256 GiB of memory, pre-installed SQL Server Enterprise, in the Asia Pacific (Mumbai) Region. Each price is a JSON string:
The output starts like this (there’s a lot more):
Read Using the AWS Price List API to learn more about these functions and the data that they return.
The new functions are available now and you can start using them today in the US East (Northern Virginia) and Asia Pacific (Mumbai) Regions to access metadata and price listings for all public AWS Regions and AWS GovCloud (US), at no charge.
To see a real-world example of how to use these functions, take a look at the new post, Controlling Projected User Costs Through Monthly Budget Policies on the AWS Management Tools Blog.
WPSeku is a black box WordPress Security scanner that can be used to scan remote WordPress installations to find security issues and vulnerabilities.
Features of WPSeku WordPress Security Scanner
WPSeku supports various types of scanning including:
- Testing for XSS Vulnerabilities
- Testing for SQL Injection Vulnerabilities
- Testing for LFI Vulnerabilities
- Bruteforce login via xmlrpc
- Username Enumeration
- Proxy Support
- Method (GET/POST)
- Custom Wordlists
- Custom user-agent
It also uses the WPVulnDB Vulnerability Database API at https://wpvulndb.com/api.
Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/just-in-case-you-missed-it-catching-up-on-some-recent-aws-launches/
So many launches and cloud innovations, that you simply may not believe. In order to catch up on some service launches and features, this post will be a round-up of some cool releases that happened this summer and through the end of September.
The launches and features I want to share with you today are:
- AWS IAM for Authenticating Database Users for RDS MySQL and Amazon Aurora
- Amazon SES Reputation Dashboard
- Amazon SES Open and Click Tracking Metrics
- Serverless Image Handler by the Solutions Builder Team
- AWS Ops Automator by the Solutions Builder Team
Let’s dive in, shall we!
AWS IAM for Authenticating Database Users for RDS MySQL and Amazon Aurora
Wished you could manage access to your Amazon RDS database instances and clusters using AWS IAM? Well, wish no longer. Amazon RDS has launched the ability for you to use IAM to manage database access for Amazon RDS for MySQL and Amazon Aurora DB.
What I like most about this new service feature is, it’s very easy to get started. To enable database user authentication using IAM, you would select a checkbox Enable IAM DB Authentication when creating, modifying, or restoring your DB instance or cluster. You can enable IAM access using the RDS console, the AWS CLI, and/or the Amazon RDS API.
After configuring the database for IAM authentication, client applications authenticate to the database engine by providing temporary security credentials generated by the IAM Security Token Service. These credentials can be used instead of providing a password to the database engine.
You can learn more about using IAM to provide targeted permissions and authentication to MySQL and Aurora by reviewing the Amazon RDS user guide.
Amazon SES Reputation Dashboard
In order to aid Amazon Simple Email Service customers’ in utilizing best practice guidelines for sending email, I am thrilled to announce we launched the Reputation Dashboard to provide comprehensive reporting on email sending health. To aid in proactively managing emails being sent, customers now have visibility into overall account health, sending metrics, and compliance or enforcement status.
The Reputation Dashboard will provide the following information:
- Account status: A description of your account health status.
- Healthy – No issues currently impacting your account.
- Probation – Account is on probation; Issues causing probation must be resolved to prevent suspension
- Pending end of probation decision – Your account is on probation. Amazon SES team member must review your account prior to action.
- Shutdown – Your account has been shut down. No email will be able to be sent using Amazon SES.
- Pending shutdown – Your account is on probation and issues causing probation are unresolved.
- Bounce Rate: Percentage of emails sent that have bounced and bounce rate status messages.
- Complaint Rate: Percentage of emails sent that recipients have reported as spam and complaint rate status messages.
- Notifications: Messages about other account reputation issues.
Amazon SES Open and Click Tracking Metrics
Another exciting feature recently added to Amazon SES is support for Email Open and Click Tracking Metrics. With Email Open and Click Tracking Metrics feature, SES customers can now track when email they’ve sent has been opened and track when links within the email have been clicked. Using this SES feature will allow you to better track email campaign engagement and effectiveness.
How does this work?
When using the email open tracking feature, SES will add a transparent, miniature image into the emails that you choose to track. When the email is opened, the mail application client will load the aforementioned tracking which triggers an open track event with Amazon SES. For the email click (link) tracking, links in email and/or email templates are replaced with a custom link. When the custom link is clicked, a click event is recorded in SES and the custom link will redirect the email user to the link destination of the original email.
You can take advantage of the new open tracking and click tracking features by creating a new configuration set or altering an existing configuration set within SES. After choosing either; Amazon SNS, Amazon CloudWatch, or Amazon Kinesis Firehose as the AWS service to receive the open and click metrics, you would only need to select a new configuration set to successfully enable these new features for any emails you want to send.
AWS Solutions: Serverless Image Handler & AWS Ops Automator
The AWS Solution Builder team has been hard at work helping to make it easier for you all to find answers to common architectural questions to aid in building and running applications on AWS. You can find these solutions on the AWS Answers page. Two new solutions released earlier this fall on AWS Answers are Serverless Image Handler and the AWS Ops Automator.
Serverless Image Handler was developed to provide a solution to help customers dynamically process, manipulate, and optimize the handling of images on the AWS Cloud. The solution combines Amazon CloudFront for caching, AWS Lambda to dynamically retrieve images and make image modifications, and Amazon S3 bucket to store images. Additionally, the Serverless Image Handler leverages the open source image-processing suite, Thumbor, for additional image manipulation, processing, and optimization.
AWS Ops Automator solution helps you to automate manual tasks using time-based or event-based triggers to automatically such as snapshot scheduling by providing a framework for automated tasks and includes task audit trails, logging, resource selection, scaling, concurrency handling, task completion handing, and API request retries. The solution includes the following AWS services:
- AWS CloudFormation: a templates to launches the core framework of microservices and solution generated task configurations
- Amazon DynamoDB: a table which stores task configuration data to defines the event triggers, resources, and saves the results of the action and the errors.
- Amazon CloudWatch Logs: provides logging to track warning and error messages
- Amazon SNS: topic to send messages to a subscribed email address to which to send the logging information from the solution
Have fun exploring and coding.
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-hipaa-eligibility-post-update-october-2017-sixteen-additional-services/
Our Health Customer Stories page lists just a few of the many customers that are building and running healthcare and life sciences applications that run on AWS. Customers like Verge Health, Care Cloud, and Orion Health trust AWS with Protected Health Information (PHI) and Personally Identifying Information (PII) as part of their efforts to comply with HIPAA and HITECH.
Sixteen More Services
In my last HIPAA Eligibility Update I shared the news that we added eight additional services to our list of HIPAA eligible services. Today I am happy to let you know that we have added another sixteen services to the list, bringing the total up to 46. Here are the newest additions, along with some short descriptions and links to some of my blog posts to jog your memory:
Amazon Aurora with PostgreSQL Compatibility – This brand-new addition to Amazon Aurora allows you to encrypt your relational databases using keys that you create and manage through AWS Key Management Service (KMS). When you enable encryption for an Amazon Aurora database, the underlying storage is encrypted, as are automated backups, read replicas, and snapshots. Read New – Encryption at Rest for Amazon Aurora to learn more.
Amazon CloudWatch Logs – You can use the logs to monitor and troubleshoot your systems and applications. You can monitor your existing system, application, and custom log files in near real-time, watching for specific phrases, values, or patterns. Log data can be stored durably and at low cost, for as long as needed. To learn more, read Store and Monitor OS & Application Log Files with Amazon CloudWatch and Improvements to CloudWatch Logs and Dashboards.
Amazon Connect – This self-service, cloud-based contact center makes it easy for you to deliver better customer service at a lower cost. You can use the visual designer to set up your contact flows, manage agents, and track performance, all without specialized skills. Read Amazon Connect – Customer Contact Center in the Cloud and New – Amazon Connect and Amazon Lex Integration to learn more.
Amazon ElastiCache for Redis – This service lets you deploy, operate, and scale an in-memory data store or cache that you can use to improve the performance of your applications. Each ElastiCache for Redis cluster publishes key performance metrics to Amazon CloudWatch. To learn more, read Caching in the Cloud with Amazon ElastiCache and Amazon ElastiCache – Now With a Dash of Redis.
Amazon Kinesis Streams – This service allows you to build applications that process or analyze streaming data such as website clickstreams, financial transactions, social media feeds, and location-tracking events. To learn more, read Amazon Kinesis – Real-Time Processing of Streaming Big Data and New: Server-Side Encryption for Amazon Kinesis Streams.
Amazon RDS for MariaDB – This service lets you set up scalable, managed MariaDB instances in minutes, and offers high performance, high availability, and a simplified security model that makes it easy for you to encrypt data at rest and in transit. Read Amazon RDS Update – MariaDB is Now Available to learn more.
Amazon RDS SQL Server – This service lets you set up scalable, managed Microsoft SQL Server instances in minutes, and also offers high performance, high availability, and a simplified security model. To learn more, read Amazon RDS for SQL Server and .NET support for AWS Elastic Beanstalk and Amazon RDS for Microsoft SQL Server – Transparent Data Encryption (TDE) to learn more.
AWS Batch – This service lets you run large-scale batch computing jobs on AWS. You don’t need to install or maintain specialized batch software or build your own server clusters. Read AWS Batch – Run Batch Computing Jobs on AWS to learn more.
AWS CloudHSM – A cloud-based Hardware Security Module (HSM) for key storage and management at cloud scale. Designed for sensitive workloads, CloudHSM lets you manage your own keys using FIPS 140-2 Level 3 validated HSMs. To learn more, read AWS CloudHSM – Secure Key Storage and Cryptographic Operations and AWS CloudHSM Update – Cost Effective Hardware Key Management at Cloud Scale for Sensitive & Regulated Workloads.
AWS Key Management Service – This service makes it easy for you to create and control the encryption keys used to encrypt your data. It uses HSMs to protect your keys, and is integrated with AWS CloudTrail in order to provide you with a log of all key usage. Read New AWS Key Management Service (KMS) to learn more.
AWS Lambda – This service lets you run event-driven application or backend code without thinking about or managing servers. To learn more, read AWS Lambda – Run Code in the Cloud, AWS Lambda – A Look Back at 2016, and AWS Lambda – In Full Production with New Features for Mobile Devs.
[email protected] – You can use this new feature of AWS Lambda to run Node.js functions across the global network of AWS locations without having to provision or manager servers, in order to deliver rich, personalized content to your users with low latency. Read [email protected] – Intelligent Processing of HTTP Requests at the Edge to learn more.
AWS Snowball Edge – This is a data transfer device with 100 terabytes of on-board storage as well as compute capabilities. You can use it to move large amounts of data into or out of AWS, as a temporary storage tier, or to support workloads in remote or offline locations. To learn more, read AWS Snowball Edge – More Storage, Local Endpoints, Lambda Functions.
AWS Snowmobile – This is an exabyte-scale data transfer service. Pulled by a semi-trailer truck, each Snowmobile packs 100 petabytes of storage into a ruggedized 45-foot long shipping container. Read AWS Snowmobile – Move Exabytes of Data to the Cloud in Weeks to learn more (and to see some of my finest LEGO work).
AWS Storage Gateway – This hybrid storage service lets your on-premises applications use AWS cloud storage (Amazon Simple Storage Service (S3), Amazon Glacier, and Amazon Elastic File System) in a simple and seamless way, with storage for volumes, files, and virtual tapes. To learn more, read The AWS Storage Gateway – Integrate Your Existing On-Premises Applications with AWS Cloud Storage and File Interface to AWS Storage Gateway.
And there you go! Check out my earlier post for a list of resources that will help you to build applications that comply with HIPAA and HITECH.
Post Syndicated from Sara Rodas original https://aws.amazon.com/blogs/aws/aws-online-tech-talks-november-2017/
Leaves are crunching under my boots, Halloween is tomorrow, and pumpkin is having its annual moment in the sun – it’s fall everybody! And just in time to celebrate, we have whipped up a fresh batch of pumpkin spice Tech Talks. Grab your planner (Outlook calendar) and pencil these puppies in. This month we are covering re:Invent, serverless, and everything in between.
November 2017 – Schedule
Noted below are the upcoming scheduled live, online technical sessions being held during the month of November. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts.
Webinars featured this month are:
Monday, November 6
9:00 – 9:40 AM PDT: Set it and Forget it: Auto Scaling Target Tracking Policies
Tuesday, November 7
9:00 – 9:40 AM PDT: Real-time Application Monitoring with Amazon Kinesis and Amazon CloudWatch
10:30 – 11:10 AM PDT: Simplify Microsoft Windows Server Management with Amazon Lightsail
12:00 – 12:40 PM PDT: Deep Dive on Amazon SES What’s New
Wednesday, November 8
10:30 – 11:10 AM PDT: Migrating Your Oracle Database to PostgreSQL
12:00 – 12:40 PM PDT: Run Your CI/CD Pipeline at Scale for a Fraction of the Cost
Thursday, November 9
10:30 – 11:10 AM PDT: Migrating Your Oracle Database to PostgreSQL
9:00 – 9:40 AM PDT: Managing Container Images with Amazon ECR
12:00 – 12:40 PM PDT: Amazon Elasticsearch Service Security Deep Dive
Monday, November 13
10:30 – 11:10 AM PDT: AWS re:Invent 2017: Know Before You Go
5:00 – 5:40 PM PDT: AWS re:Invent 2017: Know Before You Go
Tuesday, November 14
9:00 – 9:40 AM PDT: Sentiment Analysis Using Apache MXNet and Gluon
10:30 – 11:10 AM PDT: Bringing Characters to Life with Amazon Polly Text-to-Speech
12:00 – 12:40 PM PDT: Essential Capabilities of an IoT Cloud Platform
Wednesday, November 15
Security & Identity
9:00 – 9:40 AM PDT: How to Integrate AWS Directory Service with Office365
10:30 – 11:10 AM PDT: Disaster Recovery Options with AWS
Hands on Lab
12:30 – 2:00 PM PDT: Hands on Lab: Windows Workloads
Thursday, November 16
9:00 – 9:40 AM PDT: Building Serverless Websites with [email protected]
Hands on Lab
12:30 – 2:00 PM PDT: Hands on Lab: Deploy .NET Code to AWS from Visual Studio
Security updates have been issued by Arch Linux (apr, apr-util, chromium, and wget), CentOS (tomcat and tomcat6), Debian (curl, git-annex, golang, shadowsocks-libev, and wget), Fedora (libextractor and sssd), Gentoo (apache, asterisk, jython, oracle-jdk-bin, and xorg-server), openSUSE (chromium, curl, gcc48, GraphicsMagick, hostapd, kernel, libjpeg-turbo, libvirt, mysql-community-server, openvpn, SDL2, tcpdump, and wget), Oracle (tomcat and tomcat6), Red Hat (chromium-browser, tomcat, and tomcat6), Scientific Linux (tomcat and tomcat6), Slackware (php and wget), SUSE (firefox, mozilla-nss, kernel, wget, and xen), and Ubuntu (mysql-5.5, poppler, and wget).
Post Syndicated from Gordon Heinrich original https://aws.amazon.com/blogs/big-data/build-a-data-lake-foundation-with-aws-glue-and-amazon-s3/
A data lake is an increasingly popular way to store and analyze data that addresses the challenges of dealing with massive volumes of heterogeneous data. A data lake allows organizations to store all their data—structured and unstructured—in one centralized repository. Because data can be stored as-is, there is no need to convert it to a predefined schema.
Many organizations understand the benefits of using AWS as their data lake. For example, Amazon S3 is a highly durable, cost-effective object start that supports Open Data Formats while decoupling storage from compute, and it works with all the AWS analytic services. Although Amazon S3 provides the foundation of a data lake, you can add other services to tailor the data lake to your business needs. For more information about building data lakes on AWS, see What is a Data Lake?
Because one of the main challenges of using a data lake is finding the data and understanding the schema and data format, Amazon recently introduced AWS Glue. AWS Glue significantly reduces the time and effort that it takes to derive business insights quickly from an Amazon S3 data lake by discovering the structure and form of your data. AWS Glue automatically crawls your Amazon S3 data, identifies data formats, and then suggests schemas for use with other AWS analytic services.
This post walks you through the process of using AWS Glue to crawl your data on Amazon S3 and build a metadata store that can be used with other AWS offerings.
AWS Glue features
AWS Glue is a fully managed data catalog and ETL (extract, transform, and load) service that simplifies and automates the difficult and time-consuming tasks of data discovery, conversion, and job scheduling. AWS Glue crawls your data sources and constructs a data catalog using pre-built classifiers for popular data formats and data types, including CSV, Apache Parquet, JSON, and more.
Because AWS Glue is integrated with Amazon S3, Amazon RDS, Amazon Athena, Amazon Redshift, and Amazon Redshift Spectrum—the core components of a modern data architecture—it works seamlessly to orchestrate the movement and management of your data.
The AWS Glue Data Catalog is compatible with Apache Hive Metastore and supports popular tools such as Hive, Presto, Apache Spark, and Apache Pig. It also integrates directly with Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.
In addition, the AWS Glue Data Catalog features the following extensions for ease-of-use and data-management functionality:
- Discover data with search
- Identify and parse files with classification
- Manage changing schemas with versioning
For more information, see the AWS Glue product details.
Amazon S3 data lake
AWS Glue is an essential component of an Amazon S3 data lake, providing the data catalog and transformation services for modern data analytics.
In the preceding figure, data is staged for different analytic use cases. Initially, the data is ingested in its raw format, which is the immutable copy of the data. The data is then transformed and enriched to make it more valuable for each use case. In this example, the raw CSV files are transformed into Apache Parquet for use by Amazon Athena to improve performance and reduce cost.
The data can also be enriched by blending it with other datasets to provide additional insights. An AWS Glue crawler creates a table for each stage of the data based on a job trigger or a predefined schedule. In this example, an AWS Lambda function is used to trigger the ETL process every time a new file is added to the Raw Data S3 bucket. The tables can be used by Amazon Athena, Amazon Redshift Spectrum, and Amazon EMR to query the data at any stage using standard SQL or Apache Hive. This configuration is a popular design pattern that delivers Agile Business Intelligence to derive business value from a variety of data quickly and easily.
In this walkthrough, you define a database, configure a crawler to explore data in an Amazon S3 bucket, create a table, transform the CSV file into Parquet, create a table for the Parquet data, and query the data with Amazon Athena.
Discover the data
Sign in to the AWS Management Console and open the AWS Glue console. You can find AWS Glue in the Analytics section. AWS Glue is currently available in US East (N. Virginia), US East (Ohio), and US West (Oregon). Additional AWS Regions are added frequently.
The first step to discovering the data is to add a database. A database is a collection of tables.
- In the console, choose Add database. In Database name, type nycitytaxi, and choose Create.
- Choose Tables in the navigation pane. A table consists of the names of columns, data type definitions, and other metadata about a dataset.
- Add a table to the database nycitytaxi.You can add a table manually or by using a crawler. A crawler is a program that connects to a data store and progresses through a prioritized list of classifiers to determine the schema for your data. AWS Glue provides classifiers for common file types like CSV, JSON, Avro, and others. You can also write your own classifier using a grok pattern.
- To add a crawler, enter the data source: an Amazon S3 bucket named s3://aws-bigdata-blog/artifacts/glue-data-lake/data/. This S3 bucket contains the data file consisting of all the rides for the green taxis for the month of January 2017.
- Choose Next.
- For IAM role, choose the default role AWSGlueServiceRoleDefault in the drop-down list.
- For Frequency, choose Run on demand. The crawler can be run on demand or set to run on a schedule.
- For Database, choose nycitytaxi.It is important to understand how AWS Glue deals with schema changes so that you can select the appropriate method. In this example, the table is updated with any change. For more information about schema changes, see Cataloging Tables with a Crawler in the AWS Glue Developer Guide.
- Review the steps, and choose Finish. The crawler is ready to run. Choose Run it now.
When the crawler has finished, one table has been added.
- Choose Tables in the left navigation pane, and then choose data. This screen describes the table, including schema, properties, and other valuable information.
Transform the data from CSV to Parquet format
Now you can configure and run a job to transform the data from CSV to Parquet. Parquet is a columnar format that is well suited for AWS analytics services like Amazon Athena and Amazon Redshift Spectrum.
- Under ETL in the left navigation pane, choose Jobs, and then choose Add job.
- For the Name, type nytaxi-csv-parquet.
- For the IAM role, choose AWSGlueServiceRoleDefault.
- For This job runs, choose A proposed script generated by AWS Glue.
- Provide a unique Amazon S3 path to store the scripts.
- Provide a unique Amazon S3 directory for a temporary directory.
- Choose Next.
- Choose data as the data source.
- Choose Create tables in your data target.
- Choose Parquet as the format.
- Choose a new location (a new prefix location without any existing objects) to store the results.
- Verify the schema mapping, and choose Finish.
- View the job.This screen provides a complete view of the job and allows you to edit, save, and run the job.AWS Glue created this script. However, if required, you can create your own.
- Choose Save, and then choose Run job.
Add the Parquet table and crawler
When the job has finished, add a new table for the Parquet data using a crawler.
- For Crawler name, type nytaxiparquet.
- Choose S3 as the Data store.
- Include the Amazon S3 path chosen in the ETL
- For the IAM role, choose AWSGlueServiceRoleDefault.
- For Database, choose nycitytaxi.
- For Frequency, choose Run on demand.
After the crawler has finished, there are two tables in the nycitytaxi database: a table for the raw CSV data and a table for the transformed Parquet data.
Analyze the data with Amazon Athena
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is capable of querying CSV data. However, the Parquet file format significantly reduces the time and cost of querying the data. For more information, see the blog post Analyzing Data in Amazon S3 using Amazon Athena.
To use AWS Glue with Amazon Athena, you must upgrade your Athena data catalog to the AWS Glue Data Catalog. For more information about upgrading your Athena data catalog, see this step-by-step guide.
- Open the AWS Management Console for Athena. The Query Editor displays both tables in the nycitytaxi
- Open the AWS Management Console for Athena. The Query Editor displays both tables in the nycitytaxi
You can query the data using standard SQL.
- Choose the nytaxigreenparquet
Select * From "nycitytaxi"."data" limit 10;
- Choose Run Query.
This post demonstrates how easy it is to build the foundation of a data lake using AWS Glue and Amazon S3. By using AWS Glue to crawl your data on Amazon S3 and build an Apache Hive-compatible metadata store, you can use the metadata across the AWS analytic services and popular Hadoop ecosystem tools. This combination of AWS services is powerful and easy to use, allowing you to get to business insights faster.
If you have questions or suggestions, please comment below.
See the following blog posts for more information:
- Build a Schema-On-Read Analytics Pipeline Using Amazon Athena
- Harmonize, Query, and Visualize Data from Various Providers using AWS Glue, Amazon Athena, and Amazon QuickSight
About the author
Gordon Heinrich is a Solutions Architect working with global systems integrators. He works with our partners and customers to provide them architectural guidance for building data lakes and using AWS analytic services. In his spare time, he enjoys spending time with his family, skiing, hiking, and mountain biking in Colorado.
Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2017/10/27/timeshiftgrafanabuzz-1w-issue-19/
This week, we were busy prepping for our latest stable release, Grafana 4.6! This is a sizeable release that adds some key new functionality, but there’s no time to pat ourselves on the back – now it’s time to focus on Grafana 5.0! In the meantime, find out more about what’s in 4.6 in our release blog post, and let us know what you think of the new features and enhancements.
From the Blogosphere
Lyft’s Envoy dashboards: Lyft developed Envoy to relieve operational and reliability headaches. Envoy is a “service mesh” substrate that provides common utilities such as service discovery, load balancing, rate limiting, circuit breaking, stats, logging, tracing, etc. to application architectures. They’ve recently shared their Envoy dashboards, and walk you through their setup.
Monitoring Data in a SQL Table with Prometheus and Grafana Joseph recently built a proof-of-concept to add monitoring and alerting on the results of a Microsoft SQL Server query. Since he knew he’d eventually want to monitor many other things, from many other sources, he chose Prometheus and Grafana as his starting point. In this article, he walks us through his steps of exposing SQL queries to Prometheus, collecting metrics, alerting, and visualizing the results in Grafana.
Crypto Exchange Trading Data Discovering interesting public Grafana dashboards has been happening more and more lately. This week, I came across a dashboard visualizing trading data on the crypto exchanges. If you have a public dashboard you’d like shared, Let us know.
GrafanaCon EU Early Bird is Ending
Early bird discounts will be ending October 31; this is your last chance to take advantage of the discounted tickets!
Each week we review updated plugins to ensure code quality and compatibility before publishing them on grafana.com. This process can take time, and we appreciate all of the communication from plugin authors. This week we have two plugins that received some major TLC. These are two very popular plugins, so we encourage you to update. We’ve made updating easy; for on-prem Grafana, use the Grafana-cli tool, or update with 1 click if you are using Hosted Grafana.
Zabbix App Plugin – The Zabbix App Plugin just got a big update! Here are just a few of the changes:
- PostgreSQL support for Direct DB Connection.
- Triggers query mode, which allows counting active alerts by group, host and application, #141.
- sortSeries() function that sorts multiple timeseries by name, #447, thanks to @mdorenkamp.
- percentil() function, thanks to @pedrohrf.
- Zabbix System Status example dashboard.
Wroldmap Panel Plugin – The Worldmap panel also got a new update. Zooming with the mouse wheel has been turned off, as it was too easy to accidentally zoom in when scrolling the page. You can zoom in with the mouse by either double-clicking or using shift+drag to zoom in on an area.
- Support for new data source integration, the Dynamic JSON endpoint #103, thanks @LostInBrittany
- Fix for using floats in thresholds #79, thanks @fabienpomerol
- Turned off mouse wheel zoom
In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We have some awesome talks lined up this November. Hope to see you at one of these events!
11.1.17 | New York, NY – Cloud Native in the Datacenter: Why, How & Who
Raj Dutt – Panel Discussion | Join the Cloud Native Computing Foundation and Packet for an interactive discussion about the success stories, failures, rationales and in-the-trenches challenges when running cloud native in private or non “public cloud” datacenters. This means bare metal, colocation, private clouds, special hardware or networking setups, compliance and security-focused deployments. We’ll talk about successes, current initiatives within the community to enable these types of deployments and work left to be done to make cloud native platforms truly portable.
11.7.17 – 11.10.17 | Malmö, Sweden – Øredev
Carl Bergquist – Monitoring for Everyone | Once your product starts to provide value you might want to make sure that the system is behaving and performing as you expect. That’s when you start looking for something to verify that everything is working as expected. Aka monitoring. But what is monitoring? What kind of monitoring tools are there? How have they evolved lately?
Tweet of the Week
We scour Twitter each week to find an interesting/beautiful dashboard and show it off! #monitoringLove
— Quality NOC S.L. (@QualityNoc) October 23, 2017
Nice – but dashboards are meant for sharing! You should upload that to our list of Icinga2 dashboards.
How are we doing?
Well, that wraps up another week! How we’re doing? Submit a comment on this article below, or post something at our community forum. Help us make these weekly roundups better!