Tag Archives: AWS re:Invent

Increase collaboration and securely share cloud knowledge with AWS re:Post Private

2023-11-26 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/increase-collaboration-and-securely-share-cloud-knowledge-with-aws-repost-private/

Today we’re launching AWS re:Post Private, a fully managed knowledge service to accelerate cloud adoption, improve productivity, and drive innovation. re:Post Private allows organizations to increase collaboration and access knowledge resources built for your cloud community. It includes curated collections of technical content and training materials from AWS. The content is tailored specifically for your organization’s use cases, along with private discussion and collaboration forums for the members of your organization and your AWS account team.

As its name implies, you can think of it as a private version of AWS re:Post, with private content and access limited to people that belong to your organization and your AWS Account team.

Organizations of all sizes and verticals are increasingly moving their operations to the cloud. To ensure cloud adoption success, organizations must have the right skills and structure in place. The optimal way to achieve this is by setting up a centralized cloud center of excellence (CCOE). A CCOE is a centralized governance function for the organization and acts in a consultative role for central IT, business-unit IT, and cloud service consumers in the business. According to Gartner, a CCOE has three pillars: governance, brokerage, and community. The community pillar establishes the cloud community of practice (COP) that brings together stakeholders and facilitates cloud collaboration. It helps organizations adapt themselves for cloud adoption by promoting COP member interaction and facilitating cloud-related training and skills development.

AWS re:Post Private facilitates the creation, structure, and management of an internal cloud community of practice. It allows you to build a custom knowledge base that is searchable, reusable, and scalable. It allows community members to post private questions and answers and publish articles. It combines the benefits of traditional forums, such as community discussion and collaboration, with the benefits of an integrated information experience.

AWS re:Post Private is a fully managed service: there is no need to operate complex knowledge management and collaboration technologies or to develop custom solutions.

AWS re:Post Private also facilitates your interactions with AWS Support. You can create a support case directly from your private re:Post, and you can convert case resolution to reusable knowledge visible to all in your organization.

You choose in which AWS Region re:Post Private stores your data and who has access. All data at rest and in transit is encrypted using industry-standard algorithms. Your administrator chooses between using AWS-managed encryption keys or keys you manage and control.

Your organization’s Technical Account Managers are automatically added to your private re:Post. You can select other persons to invite among your organization and AWS teams, such as your AWS Solutions Architect. Only your private re:Post administrators need an AWS account. All other users can federate from your organization’s identity provider, such as Microsoft Active Directory.

Let’s see how to create a re:Post Private
To get started with AWS re:Post Private, as an administrator, I point my browser to the re:Post section of the AWS Management Console. I select Create private re:Post and enter the information needed to create a private re:Post for my organization, my team, or my project.

I can choose the Data encryption parameters and whether or not I enable Service access for Support case integration. When I’m ready, I select Create this re:Post.

Once the private re:Post is created, I can grant access to users and groups. User and group information comes from AWS IAM Identity Center and your identity provider. Invited users receive an email inviting them to connect to the private re:Post and create their profile.

That’s pretty much it for the administrator part. Once the private re:Post is created, I receive an endpoint name that I can share with the rest of my organization.

Let’s see how to use re:Post Private
As a member of the organization, I navigate to re:Post Private using the link I received from the administrator. I authenticate with the usual identity service of my organization, and I am redirected to the re:Post Private landing page.

On the top menu, I can select a tab to view the contents for Questions, Community Articles, Selections, Tags, Topics, Community Groups, or My Dashboard. This should be familiar if you already use the public knowledge service AWS re:Post that adopted a similar structure.

Further down on the page, I see the popular topics and the top contributors in my organization.I also have access to Questions and Community Groups. I can search the available content by keyword, tags, author, and so on.

Pricing and availability
You can create your organization’s AWS re:Post Private in the following AWS Regions: US West (Oregon) and Europe (Frankfurt).

AWS re:Post Private is available to customers having an AWS Enterprise or Enterprise On-Ramp support plan. re:Post Private offers a free tier that allows you to explore and try out standard capabilities for six months. There is no limit on the number of users in the free tier, and content storage is limited to 10 GB. When you reach the free storage limit, the plan is converted to the paid standard tier.

With AWS re:Post Private Standard tier, you only pay for what you use. We charge based on the number of users per month. Please visit the re:Post Private pricing page for more information.

Get started today and activate AWS re:Post Private for your organization.

— seb

Use anomaly detection with AWS Glue to improve data quality (preview)

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-anomaly-detection-with-aws-glue-to-improve-data-quality-preview/

We are launching a preview of a new AWS Glue Data Quality feature that will help to improve your data quality by using machine learning to detect statistical anomalies and unusual patterns. You get deep insights into data quality issues, data quality scores, and recommendations for rules that you can use to continuously monitor for anomalies, all without having to write any code.

Data quality counts
AWS customers already build data integration pipelines to extract and transform data. They set up data quality rules to ensure that the resulting data is of high quality and can be used to make accurate business decisions. In many cases, these rules assess the data based on criteria that were chosen and locked in at a specific point in time, reflecting the current state of the business. However, as the business environment changes and the properties of the data shift, the rules are not always reviewed and updated.

For example, a rule could be set to verify that daily sales are at least ten thousand dollars for an early-stage business. As the business succeeds and grows, the rule should be checked and updated from time to time, but in practice this rarely happens. As a result, if there’s an unexpected drop in sales, the outdated rule does not activate, and no one is happy.

Anomaly detection in action
To detect unusual patterns and to gain deeper insights into data, organizations try to create their own adaptive systems or turn to costly commercial solutions that require specific technical skills and specialized business knowledge.

To address this widespread challenge, Glue Data Quality now makes use of machine learning (ML).

Once activated, this cool new addition to Glue Data Quality gathers statistics as fresh data arrives, using ML and dynamic thresholds to learn from past patterns while looking outliers and unusual data patterns. This process produces observations and also visualizes trends so that you can quickly gain a better understanding of the anomaly.

You will also get rule recommendations as part of the Observations, and you can easily and progressively add them to your data pipelines. Rules can enforce an action such as stopping your data pipelines. In the past, you could only write static rules. Now, you can write Dynamic rules that have auto-adjusting thresholds and AnomalyDetection Rules that grasp recurring patterns and spot deviations. When you use rules as part of data pipelines, they can stop the data flow so that a data engineer can review, fix and resume.

To use anomaly detection, I add an Evaluate Data Quality node to my job:

I select the node and click Add analyzer to choose a statistic and the columns:

Glue Data Quality learns from the data to recognize patterns and then generates observations that will be shown in the Data quality tab:

And a visualization:

After I review the observations I add new rules. The first one sets adaptive thresholds that check the row count is between the smallest of the last 10 runs and the largest of the last 20 runs. The second one looks for unusual patters, for example RowCount being abnormally high on weekends:

Join the preview
This new capability is available in preview in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland). To learn more, read Data Quality Anomaly Detection]].

Stay tuned for a detailed blog post when this feature launches!

Learn more

Data Quality Anomaly Detection

— Jeff;

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities

2023-11-26 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/mutual-authentication-for-application-load-balancer-to-reliably-verify-certificate-based-client-identities/

Today, we are announcing support for mutually authenticating clients that present X509 certificates to Application Load Balancer. With this new feature, you can now offload client authentication to the load balancer, ensuring only trusted clients communicate with their backend applications. This new capability is built on S2N, AWS’s open source Transport Layer Security (TLS) implementation that provides strong encryption and protections against zero-day vulnerabilities, which developers can trust.

Mutual authentication (mTLS) is commonly used for business-to-business (B2B) applications such as online banking, automobile, or gaming devices to authenticate devices using digital certificates. Companies typically use it with a private certificate authority (CA) to authenticate their clients before granting access to data and services.

Customers have implemented mutual authentication using self-created or third-party solutions that require additional time and management overhead. These customers spend their engineering resources to build the functionality into their backend, update their code to keep up with the latest security patches, and invest heavily in infrastructure to create and rotate certificates.

With mutual authentication on Application Load Balancer, you have a fully managed, scalable, and cost-effective solution that enables you to use your developer resources to focus on other critical projects. Your ALB will authenticate clients with revocation checks and pass client certificate information to the target, which can be used for authorization by applications.

Getting started with mutual authentication on ALB
To enable mutual authentication on ALB, choose Create Application Load Balancer by the ALB wizard on Amazon EC2 console. When you select HTTPS in the Listeners and routing section, you can see more settings such as security policy, default server certificate, and a new client certificate handling option to support mutual authentication.

With Mutual authentication (mTLS) enabled, you can configure how listeners handle requests that present client certificates. This includes how your Application Load Balancer authenticates certificates and the amount of certificate metadata that is sent to your backend targets.

Mutual authentication has two options. The Passthrough option sends all the client certificate chains received from the client to your backend application using HTTP headers. The mTLS-enabled Application Load Balancer gets the client certificate in the handshake, establishes a TLS connection, and then sends whatever it gets in HTTPS headers to the target application. The application will need to verify the client certificate chain to authenticate the client.

With the Verify with trust store option, Application Load Balancer and client verify each other’s identity and establish a TLS connection to encrypt communication between them. We introduce a new trust store feature, and you can upload any CA bundle with roots and/or intermediate certificates generated by AWS Private Certificate Authority or any other third party CA as the source of trust to validate your client certificates.

It requires selecting an existing trust store or creating a new one. Trust stores contain your CAs, trusted certificates, and, optionally, certificate revocation lists (CRLs). The load balancer uses a trust store to perform mutual authentication with clients.

To use this option and create a new trust store, choose Trust Stores in the left menu of the Amazon EC2 console and choose Create trust store.

You can choose a CA certificate bundle in PEM format and, optionally, CRLs from your Amazon Simple Storage Service (Amazon S3) bucket. A CA certificate bundle is a group of CA certificates (root or intermediate) used by a trust store. CRLs can be used when a CA revokes client certificates that have been compromised, and you need to reject those revoked certificates. You can replace a CA bundle, and add or remove CRLs from the trust store after creation.

You can use the AWS Command Line Interface (AWS CLI) with new APIs such as create-trust-store to upload CA information, configure the mutual-authentication-mode on the Application Load Balancer listener, and send user certificate information to targets.

$ aws elbv2 create-trust-store --name my-tls-name \
    --ca-certificates-bundle-s3-bucket channy-certs \
    --ca-certificates-bundle-s3-key Certificates.pem \
    --ca-certificates-bundle-s3-object-version <version>
>> arn:aws:elasticloadbalancing:root:file1
$ aws elbv2 create-listener --load balancer-arn <value> \
    --protocol HTTPS \
    --port 443 \
    --mutual-authentication Mode=verify,
      TrustStoreArn=<arn:aws:elasticloadbalancing:root:file1>

If you already have your own private CA, such as AWS Private CA, third-party CA, or self-signed CA, you can upload their CA bundle or CRLs to the Application Load Balancer trust store to enable mutual authentication.

To test the mutual authentication on Application Load Balancer, follow the step-by-step instructions to make a self-signed CA bundle and client certificate using OpenSSL, upload them to the Amazon S3 bucket, and use them with an ELB trust store.

You can use curl with the --key and --cert parameters to send the client certificate as part of the request:

$ curl --key my_client.key --cert my_client.pem https://api.yourdomain.com

Mutual authentication can fail if a client presents an invalid or expired certificate, fails to present a certificate, cannot find a trust chain, or if any links in the trust chain have expired, or the certificate is on the revocation list.

Application Load Balancer will close the connections whenever it fails to authenticate a client and will record new connection logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the client’s IP address, handshake latency, TLS cipher used, and client certificate details. You can use these connection logs to analyze request patterns and troubleshoot issues.

To learn more, see Mutual authentication on Application Load Balancer in the AWS documentation.

Now available
Mutual authentication on Application Load Balancer is now available in all commercial AWS Regions where Application Load Balancer is available, except China. With no upfront costs or commitments required, you only pay for what you use. To learn more, see the Elastic Load Balancing pricing page.

Give it a try now and send feedback to AWS re:Post for Amazon EC2 or through your usual AWS Support contacts.

Learn more:
Application Load Balancer product page

— Channy

Check your AWS Free Tier usage programmatically with a new API

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/check-your-aws-free-tier-usage-programmatically-with-a-new-api/

Starting today, you can check your usage of the AWS Free Tier using the new AWS Free Tier API. You can use the API directly with the AWS Command Line Interface (AWS CLI) or integrate it into an application with the AWS SDKs.

The AWS Free Tier program provides you with the ability to explore and try out AWS services free of charge up to specified limits for each service. The AWS Free Tier includes three different types of offerings:

Always free offers allow customers to use a service for free up to specified limits as long as they are an AWS customer.
12 months free offers allow customers to use a service for free up to specified limits for one year from the date the account was activated.
Short-term trials are free to use for a specified period or up to a one-time limit, depending on the service.

Once you begin turning on AWS resources and interacting with AWS services that offer a free tier, you need to keep track of your progress toward the free tier limit so that you know when to expect to switch to pay-as-you-go pricing.

There are a few ways you can keep track of your AWS Free Tier usage:

The usage alerts in the Billing preferences of the AWS Billing and Cost Management console are enabled by default (unless the account was created via AWS Organizations) and send you emails when you exceed 85 percent of the Free Tier limit for each service.
You can create a zero-spend or a monthly-cost budget in the Budgets section of the Billing and Cost Management console. Using templates, it just requires a couple of clicks and the email address to notify.
The Free Tier page in the Billing and Cost Management console tells you the service, the type of offer, the current usage, and the forecasted usage for each offer in the current billing period.
The new GetFreeTierUsage API provides the same information in the Free Tier page with a structured format that you can use programmatically.

Let’s see how this new API works in practice.

Using the AWS Free Tier API with the AWS CLI
I got access to a new account created in the last few months. Here, I use the AWS Command Line Interface (AWS CLI) to call the GetFreeTierUsage API.

aws freetier get-free-tier-usage

The response is a JSON document that contains a description of the current usage for each offer that is applicable to this account during this billing period. For simplicity, I only show a few offers here.

{
    "freeTierUsages": [
        {
            "service": "Amazon Simple Queue Service",
            "operation": "",
            "usageType": "Requests",
            "region": "global",
            "actualUsageAmount": 294387.0,
            "forecastedUsageAmount": 679354.6153846154,
            "limit": 1000000.0,
            "unit": "Requests",
            "description": "1000000.0 Requests are always free per month as part of AWS Free Usage Tier (Global-Requests)",
            "freeTierType": "Always Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "",
            "usageType": "EBS:VolumeUsage",
            "region": "global",
            "actualUsageAmount": 9.0,
            "forecastedUsageAmount": 33.0,
            "limit": 30.0,
            "unit": "GB-Mo",
            "description": "30.0 GB-Mo for free for 12 months as part of AWS Free Usage Tier (Global-EBS:VolumeUsage)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances:0002",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 476.0,
            "forecastedUsageAmount": 851.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 225.0,
            "forecastedUsageAmount": 485.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Redshift",
            "operation": "RunComputeNode:0001",
            "usageType": "Node:dc2.large",
            "region": "global",
            "actualUsageAmount": 367.0,
            "forecastedUsageAmount": 735.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free per month during a short-term trial as part of AWS Free Usage Tier (Global-Node:dc2.large)",
            "freeTierType": "Free Trial"
        },
        ...
    ]
}

In the freeTierUsages list, I find some of the most common offers:

Two compute offers for Amazon Elastic Compute Cloud (Amazon EC2).
- The offer with operation RunInstances:0002 is for Windows.
- The offer with operation RunInstances is for Linux.
- The value of the operation property is the same as the platform details and usage operation displayed on the Instances or AMIs pages in the Amazon EC2 console. For more information, see the AMI billing information fields in the Amazon EC2 User Guide.
One storage offer for Amazon Elastic Block Store (Amazon EBS) volumes. This and the two Amazon EC2 compute offers have freeTierType equal to 12 Months Free.
An Always Free offer for Amazon Simple Queue Service (Amazon SQS).
A Free Trial (short-term) offer for Amazon Redshift.

Let’s have a look at some properties of these offers:

description gives a readable explanation of what the offer is about.
freeTierType tells the type of offer: Always Free, 12 Months Free, or Free Trial (short-term).
unit describes the unit used to measure usage for the offer. For example, Hrs (hours) for EC2 instances, GB-Mo (GB per month) for EBS volumes, Requests for Amazon SQS, and so on.

Three interesting properties are the limit of the offer (limit), the actual usage amount of the offer (actualUsageAmount), and the forecasted usage amount (forecastedUsageAmount) at the end of the billing period (the current month). They are all based on the unit used by the offer. For example, the Windows and Linux compute offers each have a limit of 750 hours per month. For the storage offer, the limit is 30 GB per month. For Amazon SQS, the limit of the offer is one million requests per month.

Details on the limits and services provided for free are detailed in each card on the AWS Free Tier page and on the pricing page of each service. The actual and forecast usage amounts provided by the AWS Free Tier API are estimated up to three times per day, similar to AWS Cost and Usage Reports.

If the forecast usage is greater than the limit for the offer, I should expect to switch to pay-as-you-go pricing before the end of the billing period if I continue to use the service in the same way. Actual usage is no longer tracked by the GetFreeTierUsage API once the limit is reached. This means that the actual usage amount cannot be greater than its limit. If that’s the case, the corresponding offer is not returned by the API.

For example, I look for the offers for which the forecast is greater than the limit using the --query option of the AWS CLI:

aws freetier get-free-tier-usage --query 'freeTierUsages[?forecastedUsageAmount > limit]'

{
    "freeTierUsages": [
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "",
            "usageType": "EBS:VolumeUsage",
            "region": "global",
            "actualUsageAmount": 9.0,
            "forecastedUsageAmount": 33.0,
            "limit": 30.0,
            "unit": "GB-Mo",
            "description": "30.0 GB-Mo for free for 12 months as part of AWS Free Usage Tier (Global-EBS:VolumeUsage)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances:0002",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 476.0,
            "forecastedUsageAmount": 851.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        }
    ]
}

According to this result, if I want to stay within the Free Tier limits, I can check how I use EBS volumes and Amazon EC2 compute with Windows.

For example, I am currently using 476 hours out of the 750 available in a month for Windows EC2 instances. At this pace, I am forecasted to cross the limit and reach about 851 hours. If I am concerned by the costs, I can switch off my Windows instances when not in use or during the night.

Things to know
Previously, the Free Tier API was not publicly available and was used internally for the Free Tier page in the AWS Billing console, where you can find the same data. We hope that making the GetFreeTierUsage API publicly available can help you have fun with AWS, have better use of the AWS Free Tier offers, and be aware of what is free and what to do when you get close to or over a limit.

Using this information, you can build custom reporting that meets your business needs. For example, if you want to avoid compute costs, you can programmatically stop or hibernate your EC2 instances or set the size of an EC2 Auto Scaling group to zero. You can use any of the AWS SDKs to create a web app or integrate this data in a monitoring solution.

More generally, you can send additional emails or notifications (for example, using Amazon SES or Amazon SNS) when the usage of an offer is close to its limit. This can help you get the maximum benefit of an offer without incurring additional costs. You can also do this with AWS Budgets if you set a usage budget amount to the Free Tier limit.

If an offer is no longer applicable to this account (for example, because it expired at the end of the previous month), the corresponding item is not included in the list. If you save the results from previous invocations of the API, you can compare the list of offers with those reported during the previous billing cycle to see which offers have recently expired.

To learn more about keeping track of your AWS Free Tier usage, we created these three 10-minute courses on AWS Skill Builder, an online learning center where you can learn from AWS experts and build cloud skills online:

— Danilo

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-use-amazon-cloudwatch-to-consolidate-hybrid-multi-cloud-and-on-premises-metrics/

You can now consolidate metrics from your hybrid, multicloud, and on-premises data sources using Amazon CloudWatch and process them in a consistent, unified fashion. You can query, visualize, and alarm on any and all of the metrics, regardless of their source. In addition to giving you a unified view, this new feature will help you to identify trends and issues that span multiple parts and aspects of your infrastructure.

When I first heard about this new feature, I thought, “Wait, I can do that with PutMetricData, what’s the big deal?” Quite a bit, as it turns out. PutMetricData stores the metrics in CloudWatch, but this cool new feature fetches them on demand, directly from the source.

Instead of storing data, you select and configure connectors that pull data from Amazon Managed Service for Prometheus, generic Prometheus, Amazon OpenSearch Service, Amazon RDS for MySQL, Amazon RDS for PostgreSQL, CSV files stored in Amazon Simple Storage Service (Amazon S3), and Microsoft Azure Monitor. Each connector is a AWS Lambda function that is deployed from a AWS CloudFormation template. CloudWatch invokes the appropriate Lambda functions as needed and makes use of the returned metrics immediately — they are not buffered or kept around.

Creating and using connectors
To get started I open the CloudWatch Console, click All metrics, and activate the Multi source query tab, then I click Create and manage data sources:

And then I do it again:

Then I choose a data source type:

CloudWatch will then prompt me for the details that it needs to create and set up the connector for my data source. For example, if I select Amazon RDS – MySQL, I give my data source a name, choose the RDS database instance, and specify the connection info:

When I click Create data source, a Lambda function, a Lambda Permission, an IAM role, a Secrets Manager Secret, a Log Group, and a AWS CloudFormation Stack will be created in my account:

Then, when I am ready to reference the data source and make use of the metrics that it provides, I enter a SQL query that returns timestamps and values for the metric:

Inside the Lambda function
The code for the Custom – getting started template is short, simple, and easy to understand. It implements handlers for two events:

DescribeGetMetricData – This handler returns a string that includes the name of the connector, default values for the arguments to the other handler, and a text description in Markdown format that is displayed in the custom data source query builder in the CloudWatch console.

GetMetricData – This handler returns a metric name, 1-dimensional array of timestamps and metric values, all for a time range that is provided as arguments to the handler.

If you spend a few minutes examining this code you should be able to see how to write functions to connect to your own data sources.

Things to know
Here are a couple of things to keep in mind about this powerful new feature:

Regions – You can create and use data connectors in all commercial AWS Regions; a connector that is running in one region can connect to and retrieve data from services and endpoints in other regions and other AWS accounts.

Pricing – There is no extra charge for the connectors. You pay for the invocations of the Lambda functions and for any other AWS infrastructure that you create.

— Jeff;

Announcing cross-region data replication for Amazon WorkSpaces

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/cross-region-data-replication-for-amazon-workspaces/

You can now use cross-region data replication to provide business continuity for your Amazon WorkSpaces users. Snapshots are taken every 12 hours, replicated to the desired destination region, and are used to provide a recovery point objective (RPO) of 12-24 hours.

Multi-Region Resilience Review
In her 2022 post Advancing business continuity with Amazon WorkSpaces Multi-Region Resilience, my colleague Ariel introduced you to the initial version of this feature and showed you how to use it to set up standby virtual desktops available for your users. After it has been set up, users log in with Amazon WorkSpaces registration codes that include fully qualified domain names (FQDNs). If the WorkSpaces in the primary region are unavailable, the users are redirected to standby WorkSpaces in the secondary region.

The standby WorkSpaces are available for a small, fixed monthly fee for infrastructure and storage, with a low flat rate change for each hour of usage during the month. Together, this feature and this business model make it easy and economical for you to maintain a standby deployment.

Cross-Region Data Replication
Today we are making this feature even more powerful by adding one-way cross-region data replication. Applications, documents, and other resources stored on the primary WorkSpace are snapshotted every 12 hours and copied to the region hosting the secondary WorkSpace. You get an additional layer of redundancy, enhanced data protection, and can minimize productivity that would otherwise be lost to disruptions. This is particularly helpful if users have installed and configured applications on top of the base image since they won’t have to repeat these steps on the secondary WorkSpace.

Here’s how it all works:

Normal Operation – During normal operation, the users in your WorkSpaces fleet are using the primary region. EBS snapshots of the system (C:) and data (D:) drives are created every 12 hours. Multi-Region Resilience runs in the secondary region and checks for fresh snapshots regularly. When it finds them, it initiates a copy to the secondary region. As the copies arrive in the secondary region they are used to update the secondary WorkSpace.

Failover Detection – As part of the setup process, you will follow Configure your DNS service and setup DNS routing policies to set up DNS routing policies and optional Amazon Route 53 health checks to manage cross-Region redirection.

Failover – If a large-scale event (LSE) affects the primary region and the primary WorkSpace, the failover detection that I just described goes in to effect. When users try to reconnect, they are redirected to the secondary region, the latest snapshots are used to launch a WorkSpace for them, and they are back up and running with access to data and apps that are between 12 and 24 hours old.

Failback – At the conclusion of the LSE, the users manually back up any data that they have created on the secondary WorkSpace and log out of it. Then they log in again, and this time they will be directed to the primary region and WorkSpace, where they can restore their backups and continue to work.

Getting Set Up
As a WorkSpaces administrator, I start by locating the desired primary WorkSpace:

I select it and choose Create Standby WorkSpaces from the Actions menu:

I select the desired region for the secondary WorkSpace and click Next:

Then I choose the right directory in the region, and again click Next:

If the primary WorkSpace is encrypted, I must enter the ARN of the KMS key in the secondary region (or, even better, use a multi-Region key). I check Enable data replication and confirm that I am authorizing an additional monthly charge:

On the next page I review my choices and click Create to initiate the creation of the secondary WorkSpace in the region that I selected.

As mentioned earlier I also need to set up Multi-Region Resilience. This includes setting up a domain name to use as a WorkSpaces registration code, setting up Route 53 health checks, and using them to power routing policies.

Things to Know
Here are a couple of important things to know about Cross-Region Data Replication:

Directories – You can use a self-managed Active Directory, AWS Managed AD or AD Connector configured as described in this post. Simple AD is not supported.

Snapshots – The first-ever EBS snapshot for a particular data volume is full, and subsequent snapshots are incremental. As a result, the first replication for a given WorkSpace will likely take longer than subsequent ones. Snapshots are initiated on a schedule that is internal to WorkSpaces and you cannot control the timing.

Encryption – You can use this feature with Encrypted WorkSpaces as long as you use the same AWS Key Management Service (AWS KMS) keys in the primary and secondary regions. You can also use multi-Region keys.

Bundles – You can use the Windows 10 and Windows 11 bundles, and you can also BYOL.

Accounts – Every AWS account has a fixed limit on the number of pending EBS snapshots. This may affect your ability to use this feature with large fleets of WorkSpaces.

Pricing – You pay a fixed monthly fee based on the amount of storage configured for each primary WorkSpace. See the Amazon WorkSpaces Pricing page for more information.

— Jeff;

Amazon Transcribe Call Analytics adds new generative AI-powered call summaries (preview)

2023-11-26 Veliswa Boya

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/amazon-transcribe-call-analytics-adds-new-generative-ai-powered-call-summaries-preview/

We are announcing generative artificial intelligence (AI)-powered call summarization in Amazon Transcribe Call Analytics in preview. Powered by Amazon Bedrock, this feature helps businesses improve customer experience, and agent and supervisor productivity by automatically summarizing customer service calls. Amazon Transcribe Call Analytics provides machine learning (ML)-powered analytics that allows contact centers to understand the sentiment, trends, and policy compliance of customer conversations to improve their experience and identify crucial feedback. A single API call is all it takes to extract transcripts, rich insights, and summaries from your customer conversations.

We understand that as a business, you want to maintain an accurate historical record of key conversation points, including action items associated with each conversation. To do this, agents summarize notes after the conversation has ended and enter these in their CRM system, a process that is time-consuming and subject to human error. Now imagine the customer trust erosion that follows when the agent fails to correctly capture and act upon important action items discussed during conversations.

How it works
Starting today, to assist agents and supervisors with the summarization of customer conversations, Amazon Transcribe Call Analytics will generate a concise summary of a contact center interaction that captures key components such as why the customer called, how the issue was addressed, and what follow-up actions were identified. After completing a customer interaction, agents can directly proceed to help the next customer since they don’t have to summarize a conversation, resulting in reduced customer wait times and improved agent productivity. Further, supervisors can review the summary when investigating a customer issue to get a gist of the conversation, without having to listen to the entire call recording or read the transcript.

Exploring Amazon Transcribe Call Analytics in the console
To see how this works visually, I first create an Amazon Simple Storage Service (Amazon S3) bucket in the relevant AWS Region. I then upload the audio file to the S3 bucket.

To create an analytics job that transcribes the audio and provides additional analytics about the conversation that the customer and the agent were having, I go to the Amazon Transcribe Call Analytics console. I select Post-call Analytics in the left hand navigation bar and then choose Create job.

Next I enter a job name making sure to keep the language settings based on the language in the audio file.

In the Amazon S3 URI path, I provide the link to the audio file uploaded in the first screenshot shown in this post.

In Role name, I select Create an IAM role which will have access to the Amazon S3 bucket, then choose Next.

I enable Generative call summarization, and then choose Create job.

After a few minutes, the job’s status will change from In progress to Complete, indicating that it was completed successfully.

Select the job, and the next screen will show the transcript and a new tab, Generative call summarization – preview.

You can also download the transcript to view the analytics and summary.

Now available
Generative call summarization in Amazon Transcribe Call Analytics is available today in English in US East (N. Virginia) and US West (Oregon).

With generative call summarization in Amazon Transcribe Call Analytics, you pay as you go and are billed monthly based on tiered pricing. For more information, see Amazon Transcribe pricing.

Learn more:

– Veliswa

Build generative AI apps using AWS Step Functions and Amazon Bedrock

2023-11-26 Marcia Villalba

Post Syndicated from Marcia Villalba original https://aws.amazon.com/blogs/aws/build-generative-ai-apps-using-aws-step-functions-and-amazon-bedrock/

Today we are announcing two new optimized integrations for AWS Step Functions with Amazon Bedrock. Step Functions is a visual workflow service that helps developers build distributed applications, automate processes, orchestrate microservices, and create data and machine learning (ML) pipelines.

In September, we made available Amazon Bedrock, the easiest way to build and scale generative artificial intelligence (AI) applications with foundation models (FMs). Bedrock offers a choice of foundation models from leading providers like AI21 Labs, Anthropic, Cohere, Stability AI, and Amazon, along with a broad set of capabilities that customers need to build generative AI applications, while maintaining privacy and security. You can use Amazon Bedrock from the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDKs.

The new Step Functions optimized integrations with Amazon Bedrock allow you to orchestrate tasks to build generative AI applications using Amazon Bedrock, as well as to integrate with over 220 AWS services. With Step Functions, you can visually develop, inspect, and audit your workflows. Previously, you needed to invoke an AWS Lambda function to use Amazon Bedrock from your workflows, adding more code to maintain them and increasing the costs of your applications.

Step Functions provides two new optimized API actions for Amazon Bedrock:

InvokeModel – This integration allows you to invoke a model and run the inferences with the input provided in the parameters. Use this API action to run inferences for text, image, and embedding models.
CreateModelCustomizationJob – This integration creates a fine-tuning job to customize a base model. In the parameters, you specify the foundation model and the location of the training data. When the job is completed, your custom model is ready to be used. This is an asynchronous API, and this integration allows Step Functions to run a job and wait for it to complete before proceeding to the next state. This means that the state machine execution will pause while the create model customization job is running and will resume automatically when the task is complete.

The InvokeModel API action accepts requests and responses that are up to 25 MB. However, Step Functions has a 256 kB limit on state payload input and output. In order to support larger payloads with this integration, you can define an Amazon Simple Storage Service (Amazon S3) bucket where the InvokeModel API reads data from and writes the result to. These configurations can be provided in the parameters section of the API action configuration parameters section.

How to get started with Amazon Bedrock and AWS Step Functions
Before getting started, ensure that you create the state machine in a Region where Amazon Bedrock is available. For this example, use US East (N. Virginia), us-east-1.

From the AWS Management Console, create a new state machine. Search for “bedrock,” and the two available API actions will appear. Drag the InvokeModel to the state machine.

You can now configure that state in the menu on the right. First, you can define which foundation model you want to use. Pick a model from the list, or get the model dynamically from the input.

Then you need to configure the model parameters. You can enter the inference parameters in the text box or load the parameters from Amazon S3.

If you keep scrolling in the API action configuration, you can specify additional configuration options for the API, such as the S3 destination bucket. When this field is specified, the API action stores the API response in the specified bucket instead of returning it to the state output. Here, you can also specify the content type for the requests and responses.

When you finish configuring your state machine, you can create and run it. When the state machine runs, you can visualize the execution details, select the Amazon Bedrock state, and check its inputs and outputs.

Using Step Functions, you can build state machines as extensively as you need, combining different services to solve many problems. For example, you can use Step Functions with Amazon Bedrock to create applications using prompt chaining. This is a technique for building complex generative AI applications by passing multiple smaller and simpler prompts to the FM instead of a very long and detailed prompt. To build a prompt chain, you can create a state machine that calls Amazon Bedrock multiple times to get an inference for each of the smaller prompts. You can use the parallel state to run all these tasks in parallel and then use an AWS Lambda function that unifies the responses of the parallel tasks into one response and generates a result.

Available now
AWS Step Functions optimized integrations for Amazon Bedrock are limited to the AWS Regions where Amazon Bedrock is available.

You can get started with Step Functions and Amazon Bedrock by trying out a sample project from the Step Functions console.

– Marcia

New Cost Optimization Hub centralizes recommended actions to save you money

2023-11-26 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/new-cost-optimization-hub-to-find-all-recommended-actions-in-one-place-for-saving-you-money/

Today, we are announcing Cost Optimization Hub, a new AWS Billing and Cost Management feature that makes it easy for you to identify, filter, aggregate, and quantify savings for AWS cost optimization recommendations.

With the new Cost Optimization Hub, you can interactively query cost optimization recommendations such as idle resource detection, resource rightsizing, and purchasing options across multiple AWS Regions and AWS accounts in your organizations without any data aggregation and processing. You can find out how much you’ll save if you implement those recommendations and easily compare and prioritize recommendations by savings.

Andy Jassy, CEO of Amazon, told shareholders, “We’re trying to build customer relationships (and a business) that outlast all of us, and as a result, our AWS sales and support teams are spending much of their time helping customers optimize their AWS spend so they can better weather this uncertain economy” in his 2022 Letter to Shareholders.

Cost Optimization Hub gathers all cost-optimizing recommended actions across AWS Cloud Financial Management (CFM) services, including AWS Cost Explorer and AWS Compute Optimizer, in one place. It incorporates customer-specific pricing and discounts into these recommendations, and it deduplicates findings and savings to give a consolidated view of your cost optimization opportunities.

If you are a FinOps team or infrastructure management team member who wants to understand your cost optimization opportunities in aggregate, such as which AWS accounts or AWS Regions have the most cost optimization opportunities, you should start with Cost Optimization Hub.

You can easily analyze cost optimization opportunities with built-in filters and grouping options. For example, after understanding which AWS account has the most cost optimization opportunities, you can identify the top cost optimization strategies, such as stopping idle resources, rightsizing, and Graviton migration. If you identify which AWS Region has the highest number of rightsizing opportunities, you can get a list of rightsizing recommendations for the Region. It will redirect you to the Compute Optimizer console through deep linking to validate the details, such as the projected CPU utilization if you implement the change.

Getting started with Cost Optimization Hub
To get started, choose Cost Optimization Hub in the left navigation menu of the AWS Billing and Cost Management Console. You can opt in by selecting Enable. There is a 24-hour wait time for Cost Optimization Hub to populate data initially and data will be refreshed daily afterward.

After opt-in, you can see the dashboard of cost optimization recommendations by AWS account, AWS Region, and tag key. If you want to see the list of resources available for optimization, choose View opportunities.

Cost Optimization Hub supports six types of cost-optimizing recommended actions, including:

Stop – Stop idle or unused resources to save up to 100 percent of the resources’ cost.
Rightsize – Move to a smaller Amazon EC2 instance type, Amazon EBS volume, AWS Lambda memory size, or AWS Fargate task size
Upgrade – Move to a later-generation product, such as moving from EBS io1 volume type to io2.
Graviton migration – Move from EC2 instance types with x86-based processors to EC2 instance types with AWS Graviton-based processors to save costs.
Purchase Savings Plans – Purchase Compute Savings Plans, EC2 Instance Savings Plans, and Amazon SageMaker Savings Plans
Purchase Reserved Instances – Purchase Amazon EC2, Amazon RDS, Amazon DynamoDB, Amazon ElastiCache, and Amazon Redshift Reserved Instances.

You can see the resource type, top recommended action, and estimated monthly savings. You can also filter the list by AWS account, AWS Region, implementation effort, and tag key as the group-by dimension.

You also can classify each recommendation as “Is resources restart needed” or “Is rollback possible.” If you specify Is resources restart needed=No as the filter, you can only see recommendations that don’t require you to restart your resources, such as EBS volume recommendations. Similarly, If you specify Is rollback possible=Yes as the filter, you can only see recommendations that can be rolled back.

If you select a specific source, for example, right-sizing EC2 instance, you can view details and connect to Amazon EC2 and the AWS Compute Optimizer console. Note that estimated monthly savings is a quick approximation of future savings. The actual savings you will realize are dependent on your future AWS usage patterns.

You can also interactively query through AWS Command Line Interface (AWS CLI) and AWS SDKs. Here’s a sample query to find the recommendations about deleting and rightsizing resources:

$ aws cost-optimization-hub list-recommendations

The preceding query gives you the following results:

{
   "items":[
      {
         "recommendationId":"MDA2MDI1ODQ1MTA1XzQ5MzNhYzZlLWZmYTUtNGI2ZC04YzBkLTAxYWE3Y2JlNjNlYg==",
         "accountId":"006025845105",
         "region":"Global",
         "resourceId":"006025845105_ComputeSavingsPlans",
         "currentResourceType":"ComputeSavingsPlans",
         "recommendedResourceType":"ComputeSavingsPlans",
         "estimatedMonthlySavings":1506.591472696,
         "estimatedSavingsPercentage":55.46400024,
         "estimatedMonthlyCost":2716.341169146,
         "currencyCode":"USD",
         "implementationEffort":"VeryLow",
         "restartNeeded":false,
         "actionType":"PurchaseSavingsPlans",
         "rollbackPossible":false,
         "recommendedResourceSummary":"$1.628/hour with three years term",
         "lastRefreshTimestamp":"2023-10-23T16:54:13-07:00",
         "recommendationLookbackPeriodInDays":30,
         "source":"CostExplorer"
      },
      {
         "recommendationId":"MDA2MDI1ODQ1MTA1XzhiZTRlNTczLTE0MDctNGIzOS05MmY3LTdmN2EzOTU2Y2ZkYw==",
         "accountId":"006025845105",
         "region":"us-east-1",
         "resourceId":"arn:aws:lambda:us-east-1:006025845105:function:Lambda-recommendation-testing:$LATEST",
         "resourceArn":"arn:aws:lambda:us-east-1:006025845105:function:Lambda-recommendation-testing:$LATEST",
         "currentResourceType":"LambdaFunction",
         "recommendedResourceType":"LambdaFunction",
         "estimatedMonthlySavings":3.1682091425308054e-06,
         "estimatedSavingsPercentage":1.936368871741565,
         "estimatedMonthlyCost":0.00016044778307703665,
         "currencyCode":"USD",
         "implementationEffort":"Low",
         "restartNeeded":false,
         "actionType":"Rightsize",
         "rollbackPossible":true,
         "currentResourceSummary":"128 MB memory",
         "recommendedResourceSummary":"160 MB memory",
         "lastRefreshTimestamp":"2023-10-24T04:07:35.364000-07:00",
         "recommendationLookbackPeriodInDays":14,
         "source":"ComputeOptimizer"
      }
   ]
}

For more information about new Cost Optimization Hub APIs, see the Cost Optimization Hub API documentation.

Now available
Cost Optimization Hub is now generally available for all customers. There is no additional charge for this new capability. You can now get started and view cost optimization recommendations across all AWS Regions.

To learn more, see the Cost Optimization Hub page and send feedback to AWS re:Post for Cost Optimization or through your usual AWS Support contacts.

— Channy

Amazon CloudWatch Logs now offers automated pattern analytics and anomaly detection

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/amazon-cloudwatch-logs-now-offers-automated-pattern-analytics-and-anomaly-detection/

Searching through log data to find operational or business insights often feels like looking for a needle in a haystack. It usually requires you to manually filter and review individual log records. To help you with that, Amazon CloudWatch has added new capabilities to automatically recognize and cluster patterns among log records, extract noteworthy content and trends, and notify you of anomalies using advanced machine learning (ML) algorithms trained using decades of Amazon and AWS operational data.

Specifically, CloudWatch now offers the following:

The Patterns tab on the Logs Insights page finds recurring patterns in your query results and lets you analyze them in detail. This makes it easier to find what you’re looking for and drill down into new or unexpected content in your logs.
The Compare button in the time interval selector on the Logs Insights page lets you quickly compare the query result for the selected time range to a previous period, such as the previous day, week, or month. In this way, it takes less time to see what has changed compared to a previous stable scenario.
The Log Anomalies page in the Logs section of the navigation pane automatically surfaces anomalies found in your logs while they are processed during ingestion.

Let’s see how these work in practice with a typical troubleshooting journey. I will look at some application logs to find key patterns, compare two time periods to understand what changed, and finally see how detecting anomalies can help discover issues.

Finding recurring patterns in the logs
In the CloudWatch console, I choose Logs Insights from the Logs section of the navigation pane. To start, I have selected which log groups I want to query. In this case, I select a log group of a Lambda function that I want to inspect and choose Run query.

In the Pattern tab, I see the patterns that have been found in these log groups. One of the patterns seems to be an error. I can select it to quickly add it as a filter to my query and focus on the logs that contain this pattern. For now, I choose the magnifying glass icon to analyze the pattern.

In the Pattern inspect window, a histogram with the occurrences of the pattern in the selected time period is shown. After the histogram, samples from the logs are provided.

The variable parts of the pattern (such as numbers) have been extracted as “tokens.” I select the Token values tab to see the values for a token. I can select a token value to quickly add it as a filter to the query and focus on the logs that contain this pattern with this specific value.

I can also look at the Related patterns tab to see other logs that typically occurred at the same time as the pattern I am analyzing. For example, if I am looking at an ERROR log that was always written alongside a DEBUG log showing more details, I would see that relationship there.

Comparing logs with a previous period
To better understand what is happening, I choose the Compare button in the time interval selector. This updates the query to compare results with a previous period. For example, I choose Previous day to see what changed compared to yesterday.

In the Patterns tab, I notice that there has actually been a 10 percent decrease in the number of errors, so the current situation might not be too bad.

I choose the magnifying glass icon on the pattern with severity type ERROR to see a full comparison of the two time periods. The graph overlaps the occurrences of the pattern over the two periods (now and yesterday in this case) inside the selected time range (one hour).

Errors are decreasing but are still there. To reduce those errors, I make some changes to the application. I come back after some time to compare the logs, and a new ERROR pattern is found that was not present in the previous time period.

My update probably broke something, so I roll back to the previous version of the application. For now, I’ll keep it as it is because the number of errors is acceptable for my use case.

Detecting anomalies in the log
I am reassured by the decrease in errors that I discovered comparing the logs. But how can I know if something unexpected is happening? Anomaly detection for CloudWatch Logs looks for unexpected patterns in the logs as they are processed during ingestion and can be enabled at log group level.

I select Log groups in the navigation pane and type a filter to see the same log group I was looking at before. I choose Configure in the Anomaly detection column and select an Evaluation frequency of 5 minutes. Optionally, I can use a longer interval (up to 60 minutes) and add patterns to process only specific log events for anomaly detection.

After I activate anomaly detection for this log group, incoming logs are constantly evaluated against historical baselines. I wait for a few minutes and, to see what has been found, I choose Log anomalies from the Logs section of the navigation pane.

To simplify this view, I can suppress anomalies that I am not interested in following. For now, I choose one of the anomalies in order to inspect the corresponding pattern in a way similar to before.

After this additional check, I am convinced there are no urgent issues with my application. With all the insights I collected with these new capabilities, I can now focus on the errors in the logs to understand how to solve them.

Things to know
Amazon CloudWatch automated log pattern analytics is available today in all commercial AWS Regions where Amazon CloudWatch Logs is offered excluding the China (Beijing), the China (Ningxia), and Israel (Tel Aviv) Regions.

The patterns and compare query features are charged according to existing Logs Insights query costs. Comparing a one-hour time period against another one-hour time period is equivalent to running a single query over a two-hour time period. Anomaly detection is included as part of your log ingestion fees, and there is no additional charge for this feature. For more information, see CloudWatch pricing.

Simplify how you analyze logs with CloudWatch automated log pattern analytics.

— Danilo

Amazon Managed Service for Prometheus collector provides agentless metric collection for Amazon EKS

2023-11-26 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/amazon-managed-service-for-prometheus-collector-provides-agentless-metric-collection-for-amazon-eks/

Today, I’m happy to announce a new capability, Amazon Managed Service for Prometheus collector, to automatically and agentlessly discover and collect Prometheus metrics from Amazon Elastic Kubernetes Service (Amazon EKS). Amazon Managed Service for Prometheus collector consists of a scraper that discovers and collects metrics from Amazon EKS applications and infrastructure without needing to run any collectors in-cluster.

This new capability provides fully managed Prometheus-compatible monitoring and alerting with Amazon Managed Service for Prometheus. One of the significant benefits is that the collector is fully managed, automatically right-sized, and scaled for your use case. This means you don’t have to run any compute for collectors to collect the available metrics. This helps you optimize metric collection costs to monitor your applications and infrastructure running on EKS.

With this launch, Amazon Managed Service for Prometheus now supports two major modes of Prometheus metrics collection: AWS managed collection, a fully managed and agentless collector, and customer managed collection.

Getting started with Amazon Managed Service for Prometheus Collector
Let’s take a look at how to use AWS managed collectors to ingest metrics using this new capability into a workspace in Amazon Managed Service for Prometheus. Then, we will evaluate the collected metrics in Amazon Managed Service for Grafana.

When you create a new EKS cluster using the Amazon EKS console, you now have the option to enable AWS managed collector by selecting Send Prometheus metrics to Amazon Managed Service for Prometheus. In the Destination section, you can also create a new workspace or select your existing Amazon Managed Service for Prometheus workspace. You can learn more about how to create a workspace by following the getting started guide.

Then, you have the flexibility to define your scraper configuration using the editor or upload your existing configuration. The scraper configuration controls how you would like the scraper to discover and collect metrics. To see possible values you can configure, please visit the Prometheus Configuration page.

Once you’ve finished the EKS cluster creation, you can go to the Observability tab on your cluster page to see the list of scrapers running in your EKS cluster.

The next step is to configure your EKS cluster to allow the scraper to access metrics. You can find the steps and information on Configuring your Amazon EKS cluster.

Once your EKS cluster is properly configured, the collector will automatically discover metrics from your EKS cluster and nodes. To visualize the metrics, you can use Amazon Managed Grafana integrated with your Prometheus workspace. Visit the Set up Amazon Managed Grafana for use with Amazon Managed Service for Prometheus page to learn more.

The following is a screenshot of metrics ingested by the collectors and visualized in an Amazon Managed Grafana workspace. From here, you can run a simple query to get the metrics that you need.

Using AWS CLI and APIs
Besides using the Amazon EKS console, you can also use the APIs or AWS Command Line Interface (AWS CLI) to add an AWS managed collector. This approach is useful if you want to add an AWS managed collector into an existing EKS cluster or make some modifications to the existing collector configuration.

To create a scraper, you can run the following command:

aws amp create-scraper \ 
       --source eksConfiguration="{clusterArn=<EKS-CLUSTER-ARN>,securityGroupIds=[<SG-SECURITY-GROUP-ID>],subnetIds=[<SUBNET-ID>]}" \ 
       --scrape-configuration configurationBlob=<BASE64-CONFIGURATION-BLOB> \ 
       --destination=ampConfiguration={workspaceArn="<WORKSPACE_ARN>"}

You can get most of the parameter values from the respective AWS console, such as your EKS cluster ARN and your Amazon Managed Service for Prometheus workspace ARN. Other than that, you also need to define the scraper configuration defined as configurationBlob.

Once you’ve defined the scraper configuration, you need to encode the configuration file into base64 encoding before passing the API call. The following is the command that I use in my Linux development machine to encode sample-configuration.yml into base64 and copy it onto the clipboard.

$ base64 sample-configuration.yml | pbcopy

Now Available
The Amazon Managed Service for Prometheus collector capability is now available to all AWS customers in all AWS Regions where Amazon Managed Service for Prometheus is supported.

Learn more:

Amazon Managed Service for Prometheus product page
AWS managed collectors documentation page

Happy building!
— Donnie

Optimize your storage costs for rarely-accessed files with Amazon EFS Archive

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/optimize-your-storage-costs-for-rarely-accessed-files-with-amazon-efs-archive/

Today, we are introducing EFS Archive, a new storage class for Amazon Elastic File System (Amazon EFS) optimized for long-lived data that is rarely accessed.

With this launch, Amazon EFS supports three Regional storage classes:

EFS Standard – Powered by SSD storage and designed to deliver submillisecond latency for active data.
EFS Infrequent Access (EFS IA) – Cost-optimized for data accessed only a few times a quarter, and that doesn’t need the submillisecond latencies of EFS Standard.
EFS Archive – Cost-optimized for long-lived data accessed a few times a year or less and offering similar performance to EFS IA.

All Regional storage classes deliver gigabytes-per-second throughput and hundreds of thousands of IOPS performance and are designed for eleven nines of durability.

You don’t need to manually pick and choose a storage class for your file systems because EFS lifecycle management can automatically migrate files across storage classes based on their access patterns. This allows you to have a single shared file system that contains files processed in very different ways: from active latency-sensitive to cold rarely-accessed data.

Many datasets have subsets of data that are valuable for generating insights but aren’t often used. With EFS Archive, you can store rarely accessed data cost-effectively while keeping it in the same shared file system as other data. This simplified storage approach allows end users and applications to collaborate on large shared datasets in one place, making it easier and quicker to set up and scale analytics workloads.

Using EFS Archive, you can optimize costs for workloads with large file-based datasets that contain a mix of active and inactive data such as user shares, machine learning (ML) training datasets, SaaS applications, and data retained for regulatory compliance like financial transactions and medical records.

Let’s see how this works in practice.

Using EFS Archive storage
To use the new EFS Archive storage class, I need to configure lifecycle management for the file system. In the Amazon EFS console, I select one of my file systems and choose Edit. To use EFS Archive storage, the file system Throughput mode must be Elastic. Elastic Throughput is the recommended choice for most workloads because it is designed to provide applications with as much throughput as they need with pay-as-you-use pricing.

Now, I configure Lifecycle management to transition files into EFS IA or EFS Archive based on my workload’s access patterns.

My workloads rarely use files older than one month. Files older than a quarter are not used by normal activities but need to be kept for a longer time. Based on these considerations, I select to automatically transition files to EFS IA after 30 days and to EFS Archive after 90 days since the last access. These are the default settings for new file systems.

When one of my old files is accessed, it’s usually an indicator that is being used in a new analysis, so it’ll become active again for some period. For this reason, I use the option to transition files back to Standard storage on their first access in IA or Archive storage.

I save changes, and that’s it! This file system will now automatically use different storage classes based on how files are being processed by my applications.

Things to know
EFS Archive is available today in all AWS Regions where Amazon EFS is offered, excluding those based in China.

To offer a more cost-optimized experience for colder, rarely-accessed files, EFS Archive offers 50 percent lower storage cost than EFS IA with a three times higher request charge when data is accessed. For more information, see Amazon EFS pricing.

You can use EFS Archive with existing file systems by configuring the file system lifecycle policies. New file systems are created by default with a lifecycle policy that automatically transitions files to EFS IA after 30 days and to EFS Archive after 90 days since the last access.

Optimize your storage costs by configuring lifecycle management for your Amazon EFS file systems.

— Danilo

New Amazon CloudWatch log class for infrequent access logs at a reduced price

2023-11-26 Marcia Villalba

Post Syndicated from Marcia Villalba original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-log-class-for-infrequent-access-logs-at-a-reduced-price/

Amazon CloudWatch Logs announces today a new log class called Infrequent Access. This new log class offers a tailored set of capabilities at a lower cost for infrequently accessed logs, enabling customers to consolidate all their logs in one place in a cost-effective manner.

As customers’ applications continue to scale and grow, so does the volume of logs generated. To limit the increase of logging costs, many customers are forced to make hard trade-offs. For example, some customers limit the logs generated by their applications, which can hinder the visibility of the application, or choose a different solution for different log types, which adds complexity and inefficiencies in managing different logging solutions. For instance, customers may send logs needed for real-time analytics and alerting to CloudWatch Logs and send more detailed logs needed for debugging and troubleshooting to a lower-cost solution that doesn’t have as many features as CloudWatch. In the end, these workarounds can impact the observability of the application, because customers need to navigate across multiple solutions to see their logs.

The Infrequent Access log class allows you to build a holistic observability solution using CloudWatch by centralizing all your logs in one place to ingest, query, and store your logs in a cost-efficient way. Infrequent Access is 50 percent lower per GB ingestion price than Standard log class. It provides a tailored set of capabilities for customers that don’t need advanced features like Live Tail, metric extraction, alarming, or data protection that the Standard log class provides. With Infrequent Access, you can still get the benefits of fully managed ingestion, storage, and the ability to deep dive using CloudWatch Logs Insights.

The following table shows a side-by-side comparison of the features that the new Infrequent Access and the Standard log classes have.

Feature	Infrequent Access log class	Standard log class
Fully managed ingestion and storage	Available	Available
Cross-account	Available	Available
Encryption with KMS	Available	Available
Logs Insights	Available	Available
Subscription filters / Export to S3	Not available	Available
GetLogEvents / FilterLogEvents	Not available	Available
Contributor, Container, and Lambda Insights	Not available	Available
Metric filter and alerting	Not available	Available
Data protection	Not available	Available
Embedded metric format (EMF)	Not available	Available
Live Tail	Not available	Available

When to use the new Infrequent Access log class
Use the Infrequent Access log class when you have a new workload that doesn’t require advanced features provided by the Standard log class. One important consideration is that when you create a log group with a specific log class, you cannot change that log group log class afterward.

The Infrequent Access log class is suitable for debug logs or web server logs because they are quite verbose and rarely require any of the advanced functionality that the Standard log class provides.

Another good workload for the Infrequent Access log class is an Internet of Things (IoT) fleet sending detailed logs that are only accessed for after the fact forensic analysis after the event. In addition, the Infrequent Access log class is a good choice for workloads where logs need to be stored for compliance because they will be queried infrequently.

Getting started
To get started using the new Infrequent Access log class, create a new log group in the CloudWatch Logs console and select the new Infrequent Access log class. You can create logs groups with the new Infrequent Access log class not only from the AWS Management Console but also from the AWS Command Line Interface (AWS CLI), AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), and AWS SDKs.

Once you have the new log group created, you can start using it in your workloads. For this example, I will configure a web application to send debug logs to this log group. After a while of the web application executes for a while, you can go back to the log group, where you see a new log stream.

When you select a log stream, you will be directed to CloudWatch Logs Insights.

Using the same familiar CloudWatch Logs Insights experience you get with Standard Class, you can create queries and search those logs to find relevant information, and you can analyze all the logs quickly in one place.

Available now
The new Infrequent Access log class is now available in all AWS Regions except the China and GovCloud Regions. You can start using it and enjoy a more cost-effective way to collect, store, and analyze your logs in a fully managed experience.

To learn more about the new log class, you can check the CloudWatch Logs user guide dedicated page for the Infrequent Access log class.

– Marcia

Your DevOps and Developer Productivity guide to re:Invent 2023

2023-11-21 Anubhav Rao

Post Syndicated from Anubhav Rao original https://aws.amazon.com/blogs/devops/your-devops-and-developer-productivity-guide-to-reinvent-2023/

Your DevOps and Developer Productivity guide to re:Invent 2023

ICYMI – AWS re:Invent is less than a week away! We can’t wait to join thousands of builders in person and virtually for another exciting event. Still need to save your spot? You can register here.

With so much planned for the DevOps and Developer Productivity (DOP) track at re:Invent, we’re highlighting the most exciting sessions for technology leaders and developers in this post. Sessions span intermediate (200) through expert (400) levels of content in a mix of interactive chalk talks, hands-on workshops, and lecture-style breakout sessions.

You will experience the future of efficient development at the DevOps and Developer Productivity track and get a chance to talk to AWS experts about exciting services, tools, and new AI capabilities that optimize and automate your software development lifecycle. Attendees will leave re:Invent with the latest strategies to accelerate development, use generative AI to improve developer productivity, and focus on high-value work and innovation.

How to reserve a seat in the sessions

Reserved seating is available for registered attendees to secure seats in the sessions of their choice. Reserve a seat by signing in to the attendee portal and navigating to Event, then Sessions.

Do not miss the Innovation Talk led by Vice President of AWS Generative Builders, Adam Seligman. In DOP225-INT Build without limits: The next-generation developer experience at AWS, Adam will provide updates on the latest developer tools and services, including generative AI-powered capabilities, low-code abstractions, cloud development, and operations. He’ll also welcome special guests to lead demos of key developer services and showcase how they integrate to increase productivity and innovation.

DevOps and Developer Productivity breakout sessions

What are breakout sessions?

AWS re:Invent breakout sessions are lecture-style and 60 minutes long. These sessions are delivered by AWS experts and typically reserve 10–15 minutes for Q&A at the end. Breakout sessions are recorded and made available on-demand after the event.

Level 200 — Intermediate

DOP201 | Best practices for Amazon CodeWhisperer Generative AI can create new content and ideas, including conversations, stories, images, videos, and music. Learning how to interact with generative AI effectively and proficiently is a skill worth developing. Join this session to learn about best practices for engaging with Amazon CodeWhisperer, which uses an underlying foundation model to radically improve developer productivity by generating code suggestions in real time.

DOP202 | Realizing the developer productivity benefits of Amazon CodeWhisperer Developers spend a significant amount of their time writing undifferentiated code. Amazon CodeWhisperer radically improves productivity by generating code suggestions in real time to alleviate this burden. In this session, learn how CodeWhisperer can “write” much of this undifferentiated code, allowing developers to focus on business logic and accelerate the pace of their innovation.

DOP205 | Accelerate development with Amazon CodeCatalyst In this session, explore the newest features in Amazon CodeCatalyst. Learn firsthand how these practical additions to CodeCatalyst can simplify application delivery, improve team collaboration, and speed up the software development lifecycle from concept to deployment.

DOP206 | AWS infrastructure as code: A year in review AWS provides services that help with the creation, deployment, and maintenance of application infrastructure in a programmatic, descriptive, and declarative way. These services help provide rigor, clarity, and reliability to application development. Join this session to learn about the new features and improvements for AWS infrastructure as code with AWS CloudFormation and AWS Cloud Development Kit (AWS CDK) and how they can benefit your team.

DOP207 | Build and run it: Streamline DevOps with machine learning on AWS While organizations have improved how they deliver and operate software, development teams still run into issues when performing manual code reviews, looking for hard-to-find defects, and uncovering security-related problems. Developers have to keep up with multiple programming languages and frameworks, and their productivity can be impaired when they have to search online for code snippets. Additionally, they require expertise in observability to successfully operate the applications they build. In this session, learn how companies like Fidelity Investments use machine learning–powered tools like Amazon CodeWhisperer and Amazon DevOps Guru to boost application availability and write software faster and more reliably.

DOP208 | Continuous integration and delivery for AWS AWS provides one place where you can plan work, collaborate on code, build, test, and deploy applications with continuous integration/continuous delivery (CI/CD) tools. In this session, learn about how to create end-to-end CI/CD pipelines using infrastructure as code on AWS.

DOP209 | Governance and security with infrastructure as code In this session, learn how to use AWS CloudFormation and the AWS CDK to deploy cloud applications in regulated environments while enforcing security controls. Find out how to catch issues early with cdk-nag, validate your pipelines with cfn-guard, and protect your accounts from unintended changes with CloudFormation hooks.

DOP210 | Scale your application development with Amazon CodeCatalyst Amazon CodeCatalyst brings together everything you need to build, deploy, and collaborate on software into one integrated software development service. In this session, discover the ways that CodeCatalyst helps developers and teams build and ship code faster while spending more time doing the work they love.

DOP211 | Boost developer productivity with Amazon CodeWhisperer Generative AI is transforming the way that developers work. Writing code is already getting disrupted by tools like Amazon CodeWhisperer, which enhances developer productivity by providing real-time code completions based on natural language prompts. In this session, get insights into how to evaluate and measure productivity with the adoption of generative AI–powered tools. Learn from the AWS Disaster Recovery team who uses CodeWhisperer to solve complex engineering problems by gaining efficiency through longer productivity cycles and increasing velocity to market for ongoing fixes. Hear how integrating tools like CodeWhisperer into your workflows can boost productivity.

DOP212 | New AWS generative AI features and tools for developers Explore how generative AI coding tools are changing the way developers and companies build software. Generative AI–powered tools are boosting developer and business productivity by automating tasks, improving communication and collaboration, and providing insights that can inform better decision-making. In this session, see the newest AWS tools and features that make it easier for builders to solve problems with minimal technical expertise and that help technical teams boost productivity. Walk through how organizations like FINRA are exploring generative AI and beginning their journey using these tools to accelerate their pace of innovation.

DOP220 | Simplify building applications with AWS SDKs AWS SDKs play a vital role in using AWS services in your organization’s applications and services. In this session, learn about the current state and the future of AWS SDKs. Explore how they can simplify your developer experience and unlock new capabilities. Discover how SDKs are evolving, providing a consistent experience in multiple languages and empowering you to do more with high-level abstractions to make it easier to build on AWS. Learn how AWS SDKs are built using open source tools like Smithy, and how you can use these tools to build your own SDKs to serve your customers’ needs.

DevOps and Developer Productivity chalk talks

What are chalk talks?

Chalk Talks are highly interactive sessions with a small audience. Experts lead you through problems and solutions on a digital whiteboard as the discussion unfolds. Each begins with a short lecture (10–15 minutes) delivered by an AWS expert, followed by a 45- or 50-minute Q&A session with the audience.

Level 300 — Advanced

DOP306 | Streamline DevSecOps with a complete software development service Security is not just for application code—the automated software supply chains that build modern software can also be exploited by attackers. In this chalk talk, learn how you can use Amazon CodeCatalyst to incorporate security tests into every aspect of your software development lifecycle while maintaining a great developer experience. Discover how CodeCatalyst’s flexible actions-based CI/CD workflows streamline the process of adapting to security threats.

DOP309-R | AI for DevOps: Modernizing your DevOps operations with AWS As more organizations move to microservices architectures to scale their businesses, applications increasingly have become distributed, requiring the need for even greater visibility. IT operations professionals and developers need more automated practices to maintain application availability and reduce the time and effort required to detect, debug, and resolve operational issues. In this chalk talk, discover how you can use AWS services, including Amazon CodeWhisperer, Amazon CodeGuru and Amazon DevOps Guru, to start using AI for DevOps solutions to detect, diagnose, and remedy anomalous application behavior.

DOP310-R | Better together: GitHub Actions, Amazon CodeCatalyst, or AWS CodeBuild Learn how combining GitHub Actions with Amazon CodeCatalyst or AWS CodeBuild can maximize development efficiency. In this chalk talk, learn about the tradeoffs of using GitHub Actions runners hosted on Amazon EC2 or Amazon ECS with GitHub Actions hosted on CodeCatalyst or CodeBuild. Explore integration with other AWS services to enhance workflow automation. Join this talk to learn how GitHub Actions on AWS can take your development processes to the next level.

DOP311 | Building infrastructure as code with AWS CloudFormation AWS CloudFormation helps you manage your AWS infrastructure as code, increasing automation and supporting infrastructure-as-code best practices. In this chalk talk, learn the fundamentals of CloudFormation, including templates, stacks, change sets, and stack dependencies. See a demo of how to describe your AWS infrastructure in a template format and provision resources in an automated, repeatable way.

DOP312 | Creating custom constructs with AWS CDK Join this chalk talk to get answers to your questions about creating, publishing, and sharing your AWS CDK constructs publicly and privately. Learn about construct levels, how to test your constructs, how to discover and use constructs in your AWS CDK projects, and explore Construct Hub.

DOP313-R | Multi-account and multi-Region deployments at scale Many AWS customers are implementing multi-account strategies to more easily manage their cloud infrastructure and improve their security and compliance postures. In this chalk talk, learn about various options for deploying resources into multiple accounts and AWS Regions using AWS developer tools, including AWS CodePipeline, AWS CodeDeploy, and Amazon CodeCatalyst.

DOP314 | Simplifying cloud infrastructure creation with the AWS CDK The AWS Cloud Development Kit (AWS CDK) is an open source software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. In this chalk talk, get an introduction to the AWS CDK and see a demo of how it can simplify infrastructure creation. Through code examples and diagrams, see how the AWS CDK lets you use familiar programming languages for declarative infrastructure definition. Also learn how it provides higher-level abstractions and constructs over native CloudFormation.

DOP317 | Applying Amazon’s DevOps culture to your team In this chalk talk, learn how Amazon helps its developers rapidly release and iterate software while maintaining industry-leading standards on security, reliability, and performance. Learn about the culture of two-pizza teams and how to maintain a culture of DevOps in a large enterprise. Also, discover how you can help build such a culture at your own organization.

DOP318 | Testing for resilience with AWS Fault Injection Simulator As cloud-based systems grow in scale and complexity, there is increased need to test distributed systems for resiliency. AWS Fault Injection Simulator (FIS) allows you to stress test your applications to understand failure modes and build more resilient services. Through code examples and diagrams, see how to set up and run fault injection experiments on AWS. By the end of this session, understand how FIS helps identify weaknesses and validate improvements to build more resilient cloud-based systems.

DOP319-R | Zero-downtime deployment strategies AWS services support a wealth of deployment options to meet your needs, ranging from in-place updates to blue/green deployment to continuous configuration with feature flags. In this chalk talk, hear about multiple options for deploying changes to Amazon EC2, Amazon ECS, and AWS Lambda compute platforms using AWS CodeDeploy, AWS AppConfig, AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), and Amazon CodeCatalyst.

DOP320 | Build a path to production with Amazon CodeCatalyst blueprints Amazon CodeCatalyst uses blueprints to configure your software projects in the service. Blueprints instruct CodeCatalyst on how to set up a code repository with working sample code, define cloud infrastructure, and run pre-configured CI/CD workflows for your project. In this session, learn how blueprints in CodeCatalyst can give developers a compliant software service they’ll want to use on AWS.

DOP321-R | Code faster with Amazon CodeWhisperer Traditionally, building applications requires developers to spend a lot of time manually writing code and trying to learn and keep up with new frameworks, SDKs, and libraries. In the last three years, AI models have grown exponentially in complexity and sophistication, enabling the creation of tools like Amazon CodeWhisperer that can generate code suggestions in real time based on a natural language description of the task. In this session, learn how CodeWhisperer can accelerate and enhance your software development with code generation, reference tracking, security scans, and more.

DOP324 | Accelerating application development with AWS client-side tools Did you know AWS has more than just services? There are dozens of AWS client-side tools and libraries designed to make developing quality applications easier. In this chalk talk, explore some of the tools available in your development workspace. Learn more about command line tooling (AWS CLI), libraries (AWS SDK), IDE integrations, and application frameworks that can accelerate your AWS application development. The audience helps set the agenda so there’s sure to be something for every builder.

DevOps and Developer Productivity workshops

What are workshops?

Workshops are two-hour interactive learning sessions where you work in small group teams to solve problems using AWS services. Each workshop starts with a short lecture (10–15 minutes) by the main speaker, and the rest of the time is spent working as a group.

Level 300 — Advanced

DOP301 | Boost your application availability with AIOps on AWS As applications become increasingly distributed and complex, developers and IT operations teams can benefit from more automated practices to maintain application availability and reduce the time and effort spent detecting, debugging, and resolving operational issues manually. In this workshop, learn how AWS AIOps solutions can help you make the shift toward more automation and proactive mechanisms so your IT team can innovate faster. The workshop includes use cases spanning multiple AWS services such as AWS Lambda, Amazon DynamoDB, Amazon API Gateway, Amazon RDS, and Amazon EKS. Learn how you can reduce MTTR and quickly identify issues within your AWS infrastructure. You must bring your laptop to participate.

DOP302 | Build software faster with Amazon CodeCatalyst In this workshop, learn about creating continuous integration and continuous delivery (CI/CD) pipelines using Amazon CodeCatalyst. CodeCatalyst is a unified software development service on AWS that brings together everything teams need to plan, code, build, test, and deploy applications with continuous CI/CD tools. You can utilize AWS services and integrate AWS resources into your projects by connecting your AWS accounts. With all of the stages of an application’s lifecycle in one tool, you can deliver quality software quickly and confidently. You must bring your laptop to participate.

DOP303-R | Continuous integration and delivery on AWS In this workshop, learn to create end-to-end continuous integration and continuous delivery (CI/CD) pipelines using AWS Cloud Development Kit (AWS CDK). Review the fundamental concepts of continuous integration, continuous deployment, and continuous delivery. Then, using TypeScript/Python, define an AWS CodePipeline, AWS CodeBuild, and AWS CodeCommit workflow. You must bring your laptop to participate.

DOP304 | Develop AWS CDK resources to deploy your applications on AWS In this workshop, learn how to build and deploy applications using infrastructure as code with AWS Cloud Development Kit (AWS CDK). Create resources using AWS CDK and learn maintenance and operations tips. In addition, get an introduction to building your own constructs. You must bring your laptop to participate.

DOP305 | Develop AWS CloudFormation templates to manage your infrastructure In this workshop, learn how to develop and test AWS CloudFormation templates. Create CloudFormation templates to deploy and manage resources and learn about CloudFormation language features that allow you to reuse and extend templates for many scenarios. Explore testing tools that can help you validate your CloudFormation templates, including cfn-lint and CloudFormation Guard. You must bring your laptop to participate.

DOP307-R | Hands-on with Amazon CodeWhisperer In this workshop, learn how to build applications faster and more securely with Amazon CodeWhisperer. The workshop begins with several examples highlighting how CodeWhisperer incorporates your comments and existing code to produce results. Then dive into a series of challenges designed to improve your productivity using multiple languages and frameworks. You must bring your laptop to participate.

DOP308 | Enforcing development standards with Amazon CodeCatalyst In this workshop, learn how Amazon CodeCatalyst can accelerate the application development lifecycle within your organization. Discover how your cloud center of excellence (CCoE) can provide standardized code and workflows to help teams get started quickly and securely. In addition, learn how to update projects as organization standards evolve. You must bring your laptop to participate.

Level 400 — Expert

DOP401 | Get better at building AWS CDK constructs In this workshop, dive deep into how to design AWS CDK constructs, which are reusable and shareable cloud components that help you meet your organization’s security, compliance, and governance requirements. Learn how to build, test, and share constructs representing a single AWS resource, as well as how to create higher-level abstractions that include built-in defaults and allow you to provision multiple AWS resources. You must bring your laptop to participate.

DevOps and Developer Productivity builders’ sessions

What are builders’ sessions?

These 60-minute group sessions are led by an AWS expert and provide an interactive learning experience for building on AWS. Builders’ sessions are designed to create a hands-on experience where questions are encouraged.

Level 300 — Advanced

DOP322-R | Accelerate data science coding with Amazon CodeWhisperer Generative AI removes the heavy lifting that developers experience today by writing much of the undifferentiated code, allowing them to build faster. Helping developers code faster could be one of the most powerful uses of generative AI that we will see in the coming years—and this framework can also be applied to data science projects. In this builders’ session, explore how Amazon CodeWhisperer accelerates the completion of data science coding tasks with extensions for JupyterLab and Amazon SageMaker. Learn how to build data processing pipeline and machine learning models with the help of CodeWhisperer and accelerate data science experiments in Python. You must bring your laptop to participate.

Level 400 — Expert

DOP402-R | Manage dev environments at scale with Amazon CodeCatalyst Amazon CodeCatalyst Dev Environments are cloud-based environments that you can use to quickly work on the code stored in the source repositories of your project. They are automatically created with pre-installed dependencies and language-specific packages so you can work on a new or existing project right away. In this session, learn how to create secure, reproducible, and consistent environments for VS Code, AWS Cloud9, and JetBrains IDEs. You must bring your laptop to participate.

DOP403-R | Hands-on with Amazon CodeCatalyst: Automating security in CI/CD pipelines In this session, learn how to build a CI/CD pipeline with Amazon CodeCatalyst and add the necessary steps to secure your pipeline. Learn how to perform tasks such as secret scanning, software composition analysis (SCA), static application security testing (SAST), and generating a software bill of materials (SBOM). You must bring your laptop to participate.

DevOps and Developer Productivity lightning talks

What are lightning talks?

Lightning talks are short, 20-minute demos led from a stage.

DOP221 | Amazon CodeCatalyst in real time: Deploying to production in minutes In this follow-up demonstration to DOP210, see how you can use an Amazon CodeCatalyst blueprint to build a production-ready application that is set up for long-term success. See in real time how to create a project using a CodeCatalyst Dev Environment and deploy it to production using a CodeCatalyst workflow.

DevOps and Developer Productivity code talks

What are code talks?

Code talks are 60-minute, highly-interactive discussions featuring live coding. Attendees are encouraged to dig in and ask questions about the speaker’s approach.

DOP203 | The future of development on AWS This code talk includes a live demo and an open discussion about how builders can use the latest AWS developer tools and generative AI to build production-ready applications in minutes. Starting at an Amazon CodeCatalyst blueprint and using integrated AWS productivity and security capabilities, see a glimpse of what the future holds for developing on AWS.

DOP204 | Tips and tricks for coding with Amazon CodeWhisperer Generative AI tools that can generate code suggestions, such as Amazon CodeWhisperer, are growing rapidly in popularity. Join this code talk to learn how CodeWhisperer can accelerate and enhance your software development with code generation, reference tracking, security scans, and more. Learn best practices for prompt engineering, and get tips and tricks that can help you be more productive when building applications.

Want to stay connected?

Get the latest updates for DevOps and Developer Productivity by following us on Twitter and visiting the AWS devops blog.

The attendee’s guide to the AWS re:Invent 2023 Compute track

2023-11-17 Chris Munns

Post Syndicated from Chris Munns original https://aws.amazon.com/blogs/compute/the-attendees-guide-to-the-aws-reinvent-2023-compute-track/

This post by Art Baudo – Principal Product Marketing Manager – AWS EC2, and Pranaya Anshu – Product Marketing Manager – AWS EC2

We are just a few weeks away from AWS re:Invent 2023, AWS’s biggest cloud computing event of the year. This event will be a great opportunity for you to meet other cloud enthusiasts, find productive solutions that can transform your company, and learn new skills through 2000+ learning sessions.

Even if you are not able to join in person, you can catch-up with many of the sessions on-demand and even watch the keynote and innovation sessions live.

If you’re able to join us, just a reminder we offer several types of sessions which can help maximize your learning in a variety of AWS topics. Breakout sessions are lecture-style 60-minute informative sessions presented by AWS experts, customers, or partners. These sessions are recorded and uploaded a few days after to the AWS Events YouTube channel.

re:Invent attendees can also choose to attend chalk-talks, builder sessions, workshops, or code talk sessions. Each of these are live non-recorded interactive sessions.

Chalk-talk sessions: Attendees will interact with presenters, asking questions and using a whiteboard in session.
Builder Sessions: Attendees participate in a one-hour session and build something.
Workshops sessions: Attendees join a two-hour interactive session where they work in a small team to solve a real problem using AWS services.
Code talk sessions: Attendees participate in engaging code-focused sessions where an expert leads a live coding session.

To start planning your re:Invent week, check-out some of the Compute track sessions below. If you find a session you’re interested in, be sure to reserve your seat for it through the AWS attendee portal.

Explore the latest compute innovations

This year AWS compute services have launched numerous innovations: From the launch of over 100 new Amazon EC2 instances, to the general availability of Amazon EC2 Trn1n instances powered by AWS Trainium and Amazon EC2 Inf2 instances powered by AWS Inferentia2, to a new way to reserve GPU capacity with Amazon EC2 Capacity Blocks for ML. There’s a lot of exciting launches to take in.

Explore some of these latest and greatest innovations in the following sessions:

CMP102 | What’s new with Amazon EC2
Provides an overview on the latest Amazon EC2 innovations. Hear about recent Amazon EC2 launches, learn how about differences between Amazon EC2 instances families, and how you can use a mix of instances to deliver on your cost, performance, and sustainability goals.
CMP217 | Select and launch the right instance for your workload and budget
Learn how to select the right instance for your workload and budget. This session will focus on innovations including Amazon EC2 Flex instances and the new generation of Intel, AMD, and AWS Graviton instances.
CMP219-INT | Compute innovation for any application, anywhere
Provides you with an understanding of the breadth and depth of AWS compute offerings and innovation. Discover how you can run any application, including enterprise applications, HPC, generative artificial intelligence (AI), containers, databases, and games, on AWS.

Customer experiences and applications with machine learning

Machine learning (ML) has been evolving for decades and has an inflection point with generative AI applications capturing widespread attention and imagination. More customers, across a diverse set of industries, choose AWS compared to any other major cloud provider to build, train, and deploy their ML applications. Learn about the generative AI infrastructure at Amazon or get hands-on experience building ML applications through our ML focused sessions, such as the following:

CMP206 | Behind-the-scenes look at generative AI infrastructure at Amazon
Learn how to power performant generative AI applications while keeping costs under control. Get a behind-the-scenes look at how purpose-built infrastructure from AWS including AWS Trainium and AWS Inferentia2 are used by Amazon teams.
CMP329-R | PyTorch best practices for generative AI & LLM inference architectures
Understand best practices for using PyTorch to deploy Large Language Models (LLMs) on a cluster of Amazon EC2 Inf2 instances managed by Amazon Elastic Kubernetes Service (EKS).
CMP402 | Build a generative AI chatbot using your own data with Amazon Titan
Build and deploy a generative AI-powered chatbot on AWS using AWS Inferentia, AWS Trainium, Amazon SageMaker, Amazon Bedrock, Amazon S3, and Amazon OpenSearch Service.

Discover what powers AWS compute

AWS has invested years designing custom silicon optimized for the cloud to deliver the best price performance for a wide range of applications and workloads using AWS services. Learn more about the AWS Nitro System, processors at AWS, and ML chips.

CMP306 | Deep dive into the AWS Nitro System
Delve further into the design, architecture, and new innovations to the AWS Nitro platform in this session.
CMP309 | Compute innovations enabled by the AWS Nitro System
Deep dive in to the AWS Nitro System, the underlying platform for modern EC2 instances, the Nitro System, which has allowed AWS to innovate faster, further reduce your costs, and deliver increased security.
CMP313 | AWS Graviton: The best price performance for your AWS workloads
Learn more about AWS Graviton, hear customer stories about recent Graviton adoptions, how you can migrate your workloads to Graviton while saving cost, and gaining sustainability benefits.

Optimize your compute costs

At AWS, we focus on delivering the best possible cost structure for our customers. Frugality is one of our founding leadership principles. Cost effective design continues to shape everything we do, from how we develop products to how we run our operations. Come learn of new ways to optimize your compute costs through AWS services, tools, and optimization strategies in the following sessions:

CMP207 | Capacity, availability, cost efficiency: Pick three
Use AWS strategies such as using attribute-based instance type selection, prioritizing operational flexibility, and Amazon EC2 Flex instances to help you get compute resources, when you need them, while keeping cost under control.
CMP211 | Smart savings: Amazon EC2 cost-optimization strategies
Discover the basics of cost optimization from AWS Savings Plans and Amazon EC2 Auto Scaling to more advanced strategies like Amazon EC2 Spot Instances, AWS Graviton, automation, and more.
CMP406 | Reduce costs and improve sustainability with AWS Graviton
Learn how using AWS Graviton based instances can provide up to 40% better price performance over comparable current-generation instances for a wide variety of workloads.

Check out workload-specific sessions

Amazon EC2 offers the broadest and deepest compute platform to help you best match the needs of your workload. More SAP, high performance computing (HPC), ML, and Windows workloads run on AWS than any other cloud. Join sessions focused around your specific workload to learn about how you can leverage AWS solutions to accelerate your innovations.

CMP103 | Bring your small business to the cloud with Amazon Lightsail
Join to learn how you can quickly deploy and configure a WordPress blog, point-of-sale system, ecommerce site, or remote Windows desktop on AWS for a simple monthly price.
CMP203 | Scaling SAP HANA workloads on AWS
Discover how AWS SAP HANA customers benefit from the flexibility and reliability of the AWS Cloud. learn how you can migrate your mission-critical SAP HANA workloads.
CMP213 | Confidently run your production HPC workloads on AWS
Explore the HPC portfolio of services and products available in the AWS Cloud. Learn how customers can scale simulations and modeling and other memory- and data-intensive technical workloads.
CMP318 | Build a spatial data lake with Visual Asset Management System
Get a hands-on look at how to use the open source Visual Asset Management System (VAMS), built by AWS, to deploy a foundation for spatial data lakes, helping business units better collaborate and use 3D data while avoiding redundancies.
CMP333 | Accelerate Apple application development with Amazon EC2 Mac instances
Dive deep into best practices to automate the provisioning, configuration, and scaling of your Amazon EC2 Mac instances.

Hear from AWS customers

AWS serves millions of customers of all sizes across thousands of use cases, every industry, and around the world. Hear customers dive into how AWS compute solutions have helped them transform their businesses.

CMP212 | Sustainable compute: Reducing costs and carbon emissions with AWS
Learn how to build sustainable and cost-efficient cloud operations that align with your business goals and see how Adobe is leveraging AWS Graviton processors to do the same.
CMP214 | HPC on AWS for semiconductors and healthcare life sciences
See a demo of how organizations like Arm are using AWS for demanding simulation workloads.
CMP320 | Powering Peloton’s journey to personalization with AWS
Learn how Peloton’s solution architecture and technology stack uses Amazon EC2, Amazon S3, and Amazon EKS to build and serve individualized class recommendations in real time with low latency for a great user experience.

Ready to unlock new possibilities?

The AWS Compute team looks forward to seeing you in Las Vegas. Come meet us at the Compute Booth in the Expo. And if you’re looking for more session recommendations, check-out additional re:Invent attendee guides curated by experts.

Unlock innovation in data and AI at AWS re:Invent 2023

2023-11-15 Pradeep Parmar

Post Syndicated from Pradeep Parmar original https://aws.amazon.com/blogs/big-data/unlock-innovation-in-data-and-ai-at-aws-reinvent-2023/

For organizations seeking to unlock innovation with data and AI, AWS re:Invent 2023 offers several opportunities. Attendees will discover services, strategies, and solutions for tackling any data challenge. In this post, we provide a curated list of keynotes, sessions, demos, and exhibits that will showcase how you can unlock innovation in data and AI using AWS for Data.

Keynotes

Several keynotes will shine a spotlight on data. For example, Dr. Swami Sivasubramanian, VP of Data and AI at AWS, keynotes Wednesday, November 29, from 8:30 AM – 10:30 AM on using company data to build differentiated generative AI applications that can unlock new levels of productivity and creativity across the organization.

Innovation Talks

Several Innovation Talks will provide insights into data topics. These 60-minute talks will also spotlight how AWS is empowering customers to innovate and overcome their most important business challenges:

STG227-INT | AWS storage: The backbone for your data-driven business, presented by Andy Warfield, Vice President and Distinguished Engineer; Tuesday, November 28 | 2:00 PM – 3:00 PM
AIM245-INT | Innovate faster with generative AI, presented by Dr. Bratin Saha, Vice President of AI & ML; Wednesday, November 29 | 1:00 PM – 2:00 PM
DAT212-INT | Future-proofing your applications with AWS databases, co-presented by Jeff Carter, Vice President of Databases and Migration Services, and Rahul Pathak, Vice President of Relational Database Engines; Wednesday, November 29 | 2:30 PM – 3:30 PM
AIM250-INT | Putting your data to work with generative AI, presented by Mai-Lan Tomsen Bukovec, Vice President of Technology; Thursday, November 30 | 12:30 PM – 1:30 PM
ANT219-INT | Data drives transformation: Data foundations with AWS analytics, presented by G2 Krishnamoorthy, Vice President of Analytics; Thursday, November 30 | 2:00 PM – 3:00 PM

Expo

FORMULA 1 cars are data powerhouses, with over 300 sensors collecting more than 1.1 million data points per second. F1 uses all that data with AWS to gain insights on race strategy and car performance. They also integrate some of those insights into the live TV broadcast to entertain and educate fans.

Stop by the AWS for Data booth in the AWS Village to get data strategy advice from an AI-powered Data Concierge created by the AWS Generative AI Innovation Center. You can race slot cars while seeing AWS technologies pull data in real time. You can also explore an augmented reality experience demonstrating F1 insights powered by AWS.

Sessions

Several educational sessions will explore best practices for deriving value from data on AWS. Examples include ANT335: Get the most out of your data warehousing workloads, DAT323: Discovering the vector database power of Amazon Aurora and Amazon RDS, and AIM337: Accelerate generative AI application development with Amazon Bedrock.

You can learn about data integration technologies and strategies with sessions such as ANT326: Set up a zero-ETL based analytics architecture for your organizations, ANT331: Build an end-to-end data strategy for analytics and generative AI, and ANT218: Unified and integrated near real-time analytics with zero-ETL.

We will also cover data governance with sessions such as ANT206: Modern data governance customer panel, ANT334: End-to-end data and machine learning governance on AWS, and AIM344: Scaling AI/ML governance.

Plan your re:Invent agenda

From data ingestion and storage to analysis and visualization, AWS provides a comprehensive, integrated, and governed set of data services. re:Invent offers a close look at how organizations can use these data services to transform their business. Browse the agenda to plan your re:Invent visit. You will leave inspired and equipped to use data like never before.

About the author

Pradeep S. Parmar is a Senior Product Marketing Manager at AWS. He focuses on data and AI services for technical decision makers. His prior experience includes leading product marketing for SaaS, networking, and server products at a startup, Cisco Systems and Sun Microsystems. Outside work, Pradeep dedicates time to volunteering with a non-profit wellness organization.

What’s cooking with Amazon Redshift at AWS re:Invent 2023

2023-11-15 Sunaina Abdul Salah

Post Syndicated from Sunaina Abdul Salah original https://aws.amazon.com/blogs/big-data/whats-cooking-with-amazon-redshift-at-aws-reinvent-2023/

AWS re:Invent is a powerhouse of a learning event and every time I have attended, I’ve been amazed at its scale and impact. There are keynotes packed with announcements from AWS leaders, training and certification opportunities, access to more than 2,000 technical sessions, an elaborate expo, executive summits, after-hours events, demos, and much more. The analytics team is waiting to engage with our customers and partners at the analytics kiosk in the expo hall.

For the latest and greatest with Amazon Redshift, our cloud data warehousing solution, I’ve curated a few must-attend sessions. The Amazon Redshift team will be there meeting with customers and discussing the newest announcements. And there are plenty of announcements you don’t want to miss! My personal favorite in this session list is the What’s new in Amazon Redshift session, led by Neeraja Rentachintala, Director of Product Management for Amazon Redshift and Matt Sandler, Senior Director of Data & Analytics from McDonalds giving a broad view of everything the team has been working on. But there are so many good ones on the list. Product and data engineers, data analysts, data scientists, data and technology leaders, and line of business owners can take their pick of sessions!

And here’s a tip—reserve your seat now so you can get into the session you want and avoid waiting in long lines. To access the session catalog and reserve your seat for one of our sessions, you must be registered for re:Invent. Register now!

The big stage

Adam Selipsky, Chief Executive Officer of Amazon Web Services – Keynote
Tuesday, 11/28 | 8:30 AM – 10:30 AM

Join Adam Selipsky, CEO of Amazon Web Services, as he shares his perspective on cloud transformation and highlights the latest innovations from data to infrastructure to AI and ML, helping you achieve your goals faster.

Swami Sivasubramanian, Vice President of Data and AI, AWS – Keynote
Wednesday, 11/29 | 8:30 AM – 10:30 AM

Swami’s session is all about data, so if that’s the thing that makes your world go round, then this is the big stage session for you. Generative AI is augmenting our productivity and creativity in new ways, while also being fueled by massive amounts of enterprise data and human intelligence, and Swami’s keynote will lay the groundwork for your understanding of AWS offerings in this realm.

Peter DeSantis, Senior Vice President of AWS Utility Computing – Keynote
Monday, 11/27 | 7:30 PM – 9:00 PM

Join Peter DeSantis, Senior Vice President of AWS Utility Computing, as he continues the Monday Night Live tradition of diving deep into the engineering that powers AWS services and keep a look out for any Amazon Redshift related announcements.

Innovation Talks

G2 Krishnamoorthy, Vice President of AWS Analytics
ANT219-INT | Data drives transformation: Data foundations with AWS analytics
Thursday, 11/30 | 2:00 PM – 3:00 PM

G2’s session discusses strategies for embedding analytics into your applications and ideas for building a data foundation that supports your business initiatives. With new capabilities for self-service and straightforward builder experiences, you can democratize data access for line of business users, analysts, scientists, and engineers. Hear also from Adidas, GlobalFoundries, and University of California, Irvine.

Sessions

We’ve got something for everyone, from deep learning sessions on building multi-cluster architectures and AWS’s zero-ETL approach to near-real-time analytics and ML in Amazon Redshift, powering self-service analytics for your organization and much more. Sessions can be big room breakout sessions, usually with a customer speaker, or more intimate and technical chalk talks, workshops, or builder sessions. Take a look, plan your week, and soak in the learning!

Monday, November 27

1:00 PM – 2:00 PM | ANT211 (Level 200 Break out session) | Powering self-service & near real-time analytics with Amazon Redshift
1:00 PM – 2:00 PM | ANT404 (Level 400 Chalk Talk) | Build a secure and highly available data strategy with Amazon Redshift

Tuesday, November 28

11:30 AM – 12:30 PM | ANT325 (Level 300 Break out session) | Amazon Redshift: A decade of innovation in cloud data warehousing
11:30 AM – 12:30 PM | ANT328 (Level 300 chalk talk) | Accessing open table formats for superior data lake analytics
2:30 PM – 3:30 PM | ANT322 (Level 300 Breakout session) | Modernize analytics by moving your data warehouse to Amazon Redshift
5:30 PM – 6:30 PM | ANT349 (Level 300 Chalk Talk) | Advanced real-time analytics and ML in your data warehouse
2:30 PM – 3:30 | ANT335 (Level 300 Chalk Talk) | Get the most out of your data warehousing workloads
2:00 PM – 4:00 PM | ANT310 (Workshop) | Share data across Regions & organizations for near real-time insights

Wednesday, November 29

12:00 PM – 2:00 PM | ANT307 (Workshop) | Connect and analyze all your data with zero-ETL approaches
4:00 PM – 5:00 PM | ANT308 (Builder session) | Build large-scale transactional data lakes with Apache Iceberg on AWS

Thursday, November 30

12:30 PM – 1:30 PM | ANT326 (Level 300) | Set up a zero-ETL-based analytics architecture for your organizations
2:00 PM – 3:00 PM | ANT203 (Level 200 breakout session) | What’s new in Amazon Redshift
2:30 PM – 3:30 PM | ANT342 (Level 300 chalk talk) | Scaling analytics with data lakes and data warehouses

Analytics kiosk in the AWS Village

When you are onsite, stop by the Analytics kiosk in the AWS re:Invent expo hall. Connect with experts, meet with book authors on data warehousing and analytics (at the Meet the Authors event on November 29 and 30, 3:00 PM – 4:00 PM), win prizes, and learn all about the latest innovations from our AWS Analytics services.

AWS Analytics Superheroes

We are excited to introduce the 2023 AWS Analytics Superheroes at this year’s re:Invent conference! Are you lightning fast and ultra-adaptable like Wire Weaver? A shapeshifting guardian and protector of data like Data Lynx? Or a digitally clairvoyant master of data insights like Cloud Sight? Join us at the Analytics kiosk to learn more about which AWS Analytics Superhero you are and receive superhero swag!

Get ready for re:Invent

To recap, here is what you can do to prepare:

Register for re:Invent 2023. Reserve your seats for your favorite sessions.
Check back for new sessions related to keynote announcements that will open up in the catalog after the keynote announcement is over. Those are the hot new launches!
Follow the AWS Databases & Analytics LinkedIn channel to stay up to date on everything re:Invent related.

See you in Vegas!

About the author

Sunaina AbdulSalah leads product marketing for Amazon Redshift. She focuses on educating customers about the impact of data warehousing and analytics and sharing AWS customer stories. She has a deep background in marketing and GTM functions in the B2B technology and cloud computing domains. Outside of work, she spends time with her family and friends and enjoys traveling.

AWS Speaker Profile: Zach Miller, Senior Worldwide Security Specialist Solutions Architect

2023-11-15 Roger Park

Post Syndicated from Roger Park original https://aws.amazon.com/blogs/security/aws-speaker-profile-zach-miller-senior-worldwide-security-specialist-solutions-architect/

In the AWS Speaker Profile series, we interview Amazon Web Services (AWS) thought leaders who help keep our customers safe and secure. This interview features Zach Miller, Senior Worldwide Security Specialist SA and re:Invent 2023 presenter of Securely modernize payment applications with AWS and Centrally manage application secrets with AWS Secrets Manager. Zach shares thoughts on the data protection and cloud security landscape, his unique background, his upcoming re:Invent sessions, and more.

How long have you been at AWS?

I’ve been at AWS for more than four years, and I’ve enjoyed every minute of it! I started as a consultant in Professional Services, and I’ve been a Security Solutions Architect for around three years.

How do you explain your job to your non-tech friends?

Well, my mother doesn’t totally understand my role, and she’s been known to tell her friends that I’m the cable company technician that installs your internet modem and router. I usually tell my non-tech friends that I help AWS customers protect their sensitive data. If I mention cryptography, I typically only get asked questions about cryptocurrency—which I’m not qualified to answer. If someone asks what cryptography is, I usually say it’s protecting data by using mathematics.

How did you get started in data protection and cryptography? What about it piqued your interest?

I originally went to school to become a network engineer, but I discovered that moving data packets from point A to point B wasn’t as interesting to me as securing those data packets. Early in my career, I was an intern at an insurance company, and I had a mentor who set up ethnical hacking lessons for me—for example, I’d come into the office and he’d have a compromised workstation preconfigured. He’d ask me to do an investigation and determine how the workstation was compromised and what could be done to isolate it and collect evidence. Other times, I’d come in and find my desk cabinets were locked with a padlock, and he wanted me to pick the lock. Security is particularly interesting because it’s an ever-evolving field, and I enjoy learning new things.

What’s been the most dramatic change you’ve seen in the data protection landscape?

One of the changes that I’ve been excited to see is an emphasis on encrypting everything. When I started my career, we’d often have discussions about encryption in the context of tradeoffs. If we needed to encrypt sensitive data, we’d have a conversation with application teams about the potential performance impact of encryption and decryption operations on their systems (for example, their databases), when to schedule downtime for the application to encrypt the data or rotate the encryption keys protecting the data, how to ensure the durability of their keys and make sure they didn’t lose data, and so on.

When I talk to customers about encryption on AWS today—of course, it’s still useful to talk about potential performance impact—but the conversation has largely shifted from “Should I encrypt this data?” to “How should I encrypt this data?” This is due to services such as AWS Key Management Service (AWS KMS) making it simpler for customers to manage encryption keys and encrypt and decrypt data in their applications with minimal performance impact or application downtime. AWS KMS has also made it simple to enable encryption of sensitive data—with over 120 AWS services integrated with AWS KMS, and services such as Amazon Simple Storage Service (Amazon S3) encrypting new S3 objects by default.

You are a frequent contributor to the AWS Security Blog. What were some of your recent posts about?

My last two posts covered how to use AWS Identity and Access Management (IAM) condition context keys to create enterprise controls for certificate management and how to use AWS Secrets Manager to securely manage and retrieve secrets in hybrid or multicloud workloads. I like writing posts that show customers how to use a new feature, or highlight a pattern that many customers ask about.

You are speaking in a couple of sessions at AWS re:Invent; what will your sessions focus on? What do you hope attendees will take away from your session?

I’m delivering two sessions at re:Invent this year. The first is a chalk talk, Centrally manage application secrets with AWS Secrets Manager (SEC221), that I’m delivering with Ritesh Desai, who is the General Manager of Secrets Manager. We’re discussing how you can securely store and manage secrets in your workloads inside and outside of AWS. We will highlight some recommended practices for managing secrets, and answer your questions about how Secrets Manager integrates with services such as AWS KMS to help protect application secrets.

The second session is also a chalk talk, Securely modernize payment applications with AWS (SEC326). I’m delivering this talk with Mark Cline, who is the Senior Product Manager of AWS Payment Cryptography. We will walk through an example scenario on creating a new payment processing application. We will discuss how to use AWS Payment Cryptography, as well as other services such as AWS Lambda, to build a simple architecture to help process and secure credit card payment data. We will also include common payment industry use cases such as tokenization of sensitive data, and how to include basic anti-fraud detection, in our example app.

What are you currently working on that you’re excited about?

My re:Invent sessions are definitely something that I’m excited about. Otherwise, I spend most of my time talking to customers about AWS Cryptography services such as AWS KMS, AWS Secrets Manager, and AWS Private Certificate Authority. I also lead a program at AWS that enables our subject matter experts to create and publish videos to demonstrate new features of AWS Security Services. I like helping people create videos, and I hope that our videos provide another mechanism for viewers who prefer information in a video format. Visual media can be more inclusive for customers with certain disabilities or for neurodiverse customers who find it challenging to focus on written text. Plus, you can consume videos differently than a blog post or text documentation. If you don’t have the time or desire to read a blog post or AWS public doc, you can listen to an instructional video while you work on other tasks, eat lunch, or take a break. I invite folks to check out the AWS Security Services Features Demo YouTube video playlist.

Is there something you wish customers would ask you about more often?

I always appreciate when customers provide candid feedback on our services. AWS is a customer-obsessed company, and we build our service roadmaps based on what our customers tell us they need. You should feel comfortable letting AWS know when something could be easier, more efficient, or less expensive. Many customers I’ve worked with have provided actionable feedback on our services and influenced service roadmaps, just by speaking up and sharing their experiences.

How about outside of work, any hobbies?

I have two toddlers that keep me pretty busy, so most of my hobbies are what they like to do. So I tend to spend a lot of time building elaborate toy train tracks, pushing my kids on the swings, and pretending to eat wooden toy food that they “cook” for me. Outside of that, I read a lot of fiction and indulge in binge-worthy TV.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Your guide to AWS Analytics at AWS re:Invent 2023

2023-11-13 Imtiaz Sayed

Post Syndicated from Imtiaz Sayed original https://aws.amazon.com/blogs/big-data/your-guide-to-aws-analytics-at-aws-reinvent-2023/

Join the AWS Analytics team at AWS re:Invent this year, where new ideas and exciting innovations come together.

For those in the data world, this post provides a curated guide for all analytics sessions that you can use to quickly schedule and build your itinerary. Book your spot early for the sessions you do not want to miss. You can do this through the attendee portal, and if you cannot make it in person, get a free pass to watch the live sessions online.

We are raising the bar this year on learning while having fun! Visit us at the AWS Analytics Kiosk in the AWS Village at the Expo to discover the AWS Analytics Superhero in you, participate in a playful quiz and AWS book signing events. Watch this space for additional details.

2023 AWS Analytics Superheroes

#AWSanalytics #awsfordata #awsreinvent2023

Keynotes

KEY002| Adam Selipsky (CEO, Amazon Web Services) | Tuesday, Nov. 28 | 8:30 AM – 10:30 AM (PDT)

Join Adam Selipsky, CEO of Amazon Web Services, as he shares his perspective on cloud transformation. He highlights innovations in data, infrastructure, and artificial intelligence and machine learning that are helping AWS customers achieve their goals faster, mine untapped potential, and create a better future.

KEY003| Swami Sivasubramanian (Vice President, Data and AI at AWS) | Nov. 29 | 8:30 AM – 10:30 AM (PDT)

A powerful relationship between humans, data, and AI is unfolding right before us. Generative AI is augmenting our productivity and creativity in new ways, while also being fueled by massive amounts of enterprise data and human intelligence. Join Swami Sivasubramanian, Vice President of Data and AI at AWS, to discover how you can use your company data to build differentiated generative AI applications and accelerate productivity for employees across your organization. Also hear from customer speakers with real-world examples of how they’ve used their data to support their generative AI use cases and create new experiences for their customers.

KEY005 | Dr. Werner Vogels (Vice President and Chief Technology Officer, Amazon.com) | Nov. 30 | 8:30 AM – 10:30 AM (PDT)

Join Dr. Werner Vogels, Amazon.com’s VP and CTO, for his twelfth re:Invent appearance. In his keynote, he covers best practices for designing resilient and cost-aware architectures. He also discusses why artificial intelligence is something every builder must consider when developing systems and the impact this will have in our world.

Analytics Innovation Talk

Data is the differentiator that drives your current business needs while simultaneously preparing you for the future. As your company transforms, you need a data foundation for business applications, new technical innovations, and data-driven business initiatives. Join G2 Krishnamoorthy, Vice President of AWS Analytics, to discuss strategies for embedding analytics into your applications and ideas for building a data foundation that supports your business initiatives. With new capabilities for self-service and simpler builder experiences, you can democratize data access for line-of-business users, analysts, scientists, and engineers. Hear inspiring stories from adidas, GlobalFoundries, and University of California, Irvine.

Breakout sessions

re:Invent breakout sessions are lecture-style, 1-hour long sessions delivered by AWS experts, customers, and partners.

Monday, Nov 27

Tuesday, Nov 28

Wednesday, Nov 29

Thursday, Nov 30

8:30 AM – 9:30 AM (PDT)

Wynn

ANT321 | How Rocket Companies run their data science platform on AWS

11:30 AM – 12:30 PM (PDT)

Mandalay Bay

ANT325 | Amazon Redshift: A decade of innovation in cloud data warehousing

9:00 AM – 10:00 AM (PDT)

MGM Grand

ANT319 | Building an open source data strategy on AWS

12:30 PM – 1:30 PM (PDT)

Mandalay Bay

ANT326 | Set up a zero-ETL-based analytics architecture for your organizations

10:00 AM – 11:00 AM (PDT)

Mandalay Bay

SVS307 | Scaling serverless data processing with Amazon Kinesis and Apache Kafka

12:30 PM – 1:30 PM (PDT)

Mandalay Bay

ANT202| What’s new in Amazon DataZone

9:00 AM – 10:00 AM (PDT)

Wynn

ANT204 | What’s new with Amazon EMR and Amazon Athena

2:00 PM – 3:00 PM (PDT)

Mandalay Bay

ANT203 | What’s new in Amazon Redshift

2:30 PM – 3:30 PM (PDT)

Mandalay Bay

ANT208-S | Build cloud data management across analytics, ML and generative AI applications (sponsored by Informatica)

2:30 PM – 3:30 PM (PDT)

Ceasars Forum

ANT322 | Modernize analytics by moving your data warehouse to Amazon Redshift

10:00 AM – 11:00 AM (PDT)

Ceasars Forum

ANT209 | Optimizing TCO for business-critical analytics

2:30 PM – 3:30 PM (PDT)

Ceasars Forum

ANT210 | Improve your search with vector capabilities in Amazon OpenSearch Service

3:00 PM – 4:00 PM (PDT)

MGM Grand

COM308 | Serverless data streaming: Amazon Kinesis Data Streams and AWS Lambda

4:00 PM – 5:00 PM (PDT)

Ceasars Forum

ANT329 | Best practices for analytics and generative AI on AWS

10:30 AM – 11:30 AM (PDT)

Wynn

ANT324 | Easily and securely prepare, share, and query data

3:30 PM – 4:30 PM (PDT)

Mandalay Bay

ANT220 | What’s new with AWS data integration

3:00 PM – 4:00 PM (PDT)

Wynn

ANT320 | How Electronic Arts modernized its data platform with Amazon EMR

4:00 PM – 5:00 PM (PDT)

Mandalay Bay

ANT317 | How Rivian builds real-time analytics from electric vehicles

10:30 AM – 11:30 AM (PDT)

MGM Grand

ANT303 | What’s new in AWS Lake Formation

4:00 PM – 5:00 PM (PDT)

Caesars Forum

ANT301 | What’s new in Amazon OpenSearch Service

3:00 PM – 4:00 PM (PDT)

Mandalay Bay

BSI203 | Enhance your applications with Amazon QuickSight embedded analytics

11:30 AM – 12:30 PM (PDT)

Ceasars Forum

ANT318 | Accelerate innovation with end-to-end serverless data architecture

4:30 PM – 5:30 PM (PDT)

Wynn

ANT207 | Understand your data with business context

1:00 PM – 2:00 PM (PDT)

Venetian

ANT201 | Accelerate innovation with real-time data

5:30 PM – 6:30 PM (PDT)

MGM Grand

ANT350 | Security analytics and observability with Amazon OpenSearch Service

1:00 PM – 2:00 PM (PDT)

Ceasars Forum

BSI101 | Generative BI in Amazon QuickSight

2:30 PM – 3:30 PM (PDT)

Ceasars Forum

BSI205 | What’s new with Amazon QuickSight

5:30 PM – 6:30 PM (PDT)

Mandalay Bay

ANT205 | Curate your data at scale

5:30 PM – 6:30 PM (PDT)

Mandalay Bay

ANT323 | Smarter, faster analytics with generative AI and ML

Chalk talks

Chalk talks are an hour-long, highly interactive content format with a small audience. Each begins with a short lecture delivered by an AWS expert, followed by a Q&A session with the audience.

Monday, Nov 27

Tuesday, Nov 28

Wednesday, Nov 29

Thursday, Nov 30

10:00 AM – 11:00 AM (PDT)

Wynn

OPN311 | Extending OpenSearch

11:00 AM – 12:00 PM (PDT)

Wynn

ANT336-R | Governed data sharing with Amazon DataZone and AWS Lake Formation [REPEAT]

10:00 AM – 11:00 AM (PDT)

Ceasars Forum

ANT316 | Fast-track streaming ETL with AWS streaming data services

11:00 AM – 12:00 PM (PDT)

Wynn

ANT347 | Using natural language to author data integration applications

11:30 AM – 12:30 PM (PDT)

Wynn

OPN308-R | Build and operate a Zero Trust Apache Kafka cluster [REPEAT]

11:00 AM – 12:00 PM (PDT)

Mandalay Bay

ANT338-R | Migrate legacy ETL to AWS Glue [REPEAT]

10:00 AM – 11:00 AM (PDT)

Wynn

ANT339 | Optimizing Apache Spark workloads with Amazon EMR Serverless

12:30 PM – 1:30 PM (PDT)

Mandalay Bay

ANT314 | Migrating self-managed Apache Flink to fully managed on AWS

2:30 PM – 3:30 PM (PDT)

Mandalay Bay

ANT333-R | Data integration with AWS Glue and Amazon MWAA [REPEAT]

11:30 AM – 12:30 PM (PDT)

Ceasars Forum

ANT328 | Accessing open table formats for superior data lake analytics

10:30 AM – 11:30 AM (PDT)

Ceasars Forum

STG328 | Powering your data lakes and analytics with Amazon S3

2:30 PM – 3:30 PM (PDT)

Mandalay Bay

ANT341 | Reduce downtime and optimize costs through data pipeline observability

4:00 PM – 5:00 PM (PDT)

MGM Grand

ANT315 | Accelerating value from data: Migrate from batch to stream processing

12:30 PM – 1:30 PM (PDT)

MGM Grand

BSI302 | Architecting governed BI for all users with Amazon QuickSight

11:30 AM – 12:30 PM (PDT)

Ceasars Forum

ANT340 | Powering observability with AI and Amazon OpenSearch Ingestion

2:30 PM – 3:30 PM (PDT)

Ceasars Forum

ANT342-R | Scaling analytics with data lakes and data warehouses [REPEAT]

4:00 PM – 5:00 PM (PDT)

Mandalay Bay

ANT311-R | Stream data and build transactional lakes with AWS Glue streaming [REPEAT]

12:30 PM – 1:30 PM (PDT)

Wynn

ANT403 | How to migrate your Apache Kafka workloads to Amazon MSK

2:30 PM – 3:30 PM (PDT)

Ceasars Forum

ANT346-R | Understanding zero-ETL with AWS analytics [REPEAT]

3:30 PM – 4:30 PM (PDT)

MGM Grand

STG336-R | Scale data lake access control and enhance governance with Amazon S3 [REPEAT]

4:30 PM – 5:30 PM (PDT)

Wynn

ANT337 | Identify and remediate security threats with Amazon OpenSearch Service

1:00 PM – 2:00 PM (PDT)

Ceasars Forum

ANT327-R | 5 steps to successfully migrating to Amazon OpenSearch Service [REPEAT]

2:30 PM – 3:30 PM (PDT)

Wynn

ANT334 | End-to-end data and machine learning governance on AWS

3:30 PM – 4:30 PM (PDT)

Ceasars Forum

ANT344 | Self-service analytics for all

.4:30 PM – 5:30 PM (PDT)

Mandalay Bay

BSI301-R | Architectural patterns for embedded analytics using Amazon QuickSight [REPEAT]

2:00 PM – 3:00 PM (PDT)

MGM Grand

BSI401 | DevOps strategies for Amazon QuickSight business intelligence assets

4:00 PM – 5:00 PM (PDT)

Ceasars Forum

ANT343 | Securely manage data across organizational boundaries

4:00 PM – 5:00 PM (PDT)

MGM Grand

TRV303 | Build a digital concierge for travelers and guests with generative AI

2:30 PM – 3:30 PM (PDT)

Mandalay Bay

ANT335 | Get the most out of your data warehousing workloads

5:30 PM – 6:30 PM (PDT)

Ceasars Forum

ANT349-R | Advanced real-time analytics and ML in your data warehouse [REPEAT]

Builders’ sessions

These are 1-hour small-group sessions with up to nine attendees per table and one AWS expert. Each builders’ session begins with a short explanation or demonstration of what you’re going to build. When the demonstration is complete, bring your laptop to experiment and build with the AWS expert.

Monday, Nov 27

Tuesday, Nov 28

Wednesday, Nov 29

2:30 PM – 3:30 PM (PDT)

Mandalay Bay

ANT313-R | Use data catalogs to improve self-service analytics [REPEAT]

11:00 AM – 12:00 PM (PDT)

Mandalay Bay

OPN401 | Apache Hudi on AWS: Tuning for cost and performance

4:00 PM – 5:00 PM (PDT)

Ceasars Forum

ANT308-R | Build large-scale transactional data lakes with Apache Iceberg on AWS [REPEAT]

11:00 AM – 12:00 PM (PDT)

MGM Grand

BSI206 | Scale enterprise BI securely with Amazon QuickSight

Workshops

Workshops are 2-hour interactive sessions where you work in teams or individually to solve problems using AWS services. Each workshop starts with a short lecture, and the rest of the time is spent working the problem. Bring your laptop to build along with AWS experts.

Monday, Nov 27	Tuesday, Nov 28	Wednesday, Nov 29
2:00 PM – 4:00 PM (PDT) Mandalay Bay BSI201 \| Build dashboards, reports, and explore generative BI in Amazon QuickSight	11:00 AM – 1:00 PM (PDT) MGM Grand ANT306 \| Build a data foundation to power your generative AI applications	11:30 AM – 1:30 PM (PDT) Ceasars Forum ANT312 \| Using Amazon OpenSearch Service as a vector database for gen AI apps
2:30 PM – 4:30 PM (PDT) Wynn ANT305-R \| Log analytics made easy with Amazon OpenSearch Serverless [REPEAT]	11:30 AM – 1:30 PM (PDT) Mandalay Bay ANT401-R \| Event detection with Amazon MSK and Amazon Managed Service for Apache Flink [REPEAT]	.
.	11:30 AM – 1:30 PM (PDT) MGM Grand BSI202 \| Quickly build predictive dashboards using no-code ML and generative BI	.
.	2:00 PM – 4:00 PM (PDT) Wynn ANT310 \| Share data across Regions and organizations for near-real-time insights	.
.	11:30 AM – 1:30 PM (PDT) Ceasars Forum ANT304-R \| A pragmatic approach to data governance on AWS [REPEAT]	.
.	11:30 AM – 1:30 PM (PDT) Venetian AIM350 \| Personalized marketing content with generative AI and Amazon Personalize	.
.	12:00 PM – 2:00 PM (PDT) Ceasars Forum ANT402 \| Protect and securely share the right data	.
.	12:00 PM – 2:00 PM (PDT) Wynn ANT307 \| Connect and analyze all your data with zero-ETL approaches	.

Code talks

Code talks are similar to our popular chalk talk format, but instead of focusing on an architecture solution with whiteboarding, the speakers lead an interactive discussion featuring live coding or code samples. These sessions focus on the actual code that goes into building a solution. Attendees will learn the “why” behind the solution and see it come to life, and even the errors that are bound to happen. Attendees are encouraged to ask questions and follow along.

Monday, Nov 27

Tuesday, Nov 28

Wednesday, Nov 29

Thursday, Nov 30

11:30 AM – 12:30 PM (PDT)

Wynn

ANT332-R | Customize Amazon Athena to integrate with new data sources [REPEAT]

1:00 PM – 2:00 PM (PDT)

Wynn

ANT345 | Simplify working with data across multicloud with AWS analytics

5:30 PM – 6:30 PM (PDT)

Wynn

ANT405 | Build a Flink application on Amazon Managed Service for Apache Flink

4:00 PM – 5:00 PM (PDT)

Wynn

AIM366 | Data preparation for ML at scale with Amazon SageMaker notebooks

Conclusion

We hope this post acts as your go-to resource for navigating the AWS analytics track at re:Invent 2023. For staying in the know about the most recent trends and advancements in AWS Analytics, follow our dedicated LinkedIn page. Visit the Amazon QuickSight guide to learn what’s new in Business Intelligence.

About the authors

Imtiaz (Taz) Sayed is the WW Tech Leader for Analytics at AWS. He enjoys engaging with the community on all things data and analytics. He can be reached via LinkedIn.

Navnit Shukla serves as an AWS Specialist Solution Architect with a focus on Analytics. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices. Notably, Navnit Shukla is the accomplished author of the book titled “Data Wrangling on AWS.” He can be reached via LinkedIn.

Real-time streaming data top picks you cannot miss at AWS re:Invent 2023

2023-11-08 Anna Montalat

Post Syndicated from Anna Montalat original https://aws.amazon.com/blogs/big-data/real-time-streaming-data-top-picks-you-cannot-miss-at-aws-reinvent-2023/

Save the date: AWS re:Invent 2023 is happening from November 27 to December 1 in Las Vegas, and you cannot miss it. re:Invent is a learning conference organized by AWS for the global cloud computing community. It’s a great opportunity to come together with cloud enthusiasts from around the world to hear about the latest cloud industry innovations, meet with AWS experts, have fun, and build connections.

Join us as we delve into the world of real-time streaming data at re:Invent 2023 and discover how you can use real-time streaming data to build new use cases, optimize existing projects and processes, and reimagine what’s possible. In today’s data-driven landscape, the quality of data is the foundation upon which the success of organizations and innovations stands. High-quality data is not just about accuracy; it’s also about timeliness. To derive meaningful insights and ensure the optimal performance of machine learning (ML) and generative AI models, data needs to be ingested and processed in real time. Real-time data empowers these models to adapt and respond instantaneously to changing scenarios, making them not just smarter but also more practical. With real-time streaming data, organizations can reimagine what’s possible. From enabling predictive maintenance in manufacturing to delivering hyper-personalized content in the media and entertainment industry, and from real-time fraud detection in finance to precision agriculture in farming, the potential applications are vast.

This post will help you plan your re:Invent experience by highlighting the essential sessions on real-time streaming data and beyond that you shouldn’t miss. To attend these sessions, be sure to register for re:Invent and access the session catalog. Register now!

Keynotes and Innovation Talk sessions you cannot miss!

Don’t miss your chance to hear from some of the leading voices in AWS. Here are some of our favorite keynotes and Innovation Talk sessions.

Adam Selipsky, Chief Executive Officer of Amazon Web Services – Keynote

Tuesday, November 28 | 8:30 AM – 10:30 AM PST | The Venetian

Reserve your seat now!

Swami Sivasubramanian, Vice President of AWS Data and AI– Keynote

Wednesday, November 29 | 8:30 AM – 10:30 AM PST | The Venetian

Join Swami Sivasubramanian, Vice President of Data and AI at AWS, to discover how you can use your company’s data to build differentiated generative AI applications and accelerate productivity for employees across your organization. Also hear from customer speakers with real-world examples of how they’ve used their data to support their generative AI use cases and create new experiences for their customers.

Reserve your seat now!

Putting your data to work with generative AI – Innovation Talk

Thursday, November 30 | 12:30 – 1:30 PM PST | The Venetian

Join Mai-Lan Tomsen Bukovec, Vice President, Technology at AWS to learn how you can turn your data lake into a business advantage with generative AI. Explore strategies for putting your proprietary datasets to work when building unique, differentiated generative AI solutions. Learn how to utilize your datasets using Amazon SageMaker and Amazon Bedrock and popular frameworks like PyTorch with AWS compute, storage, and analytics. Hear best practices for using unstructured (video, image, PDF), semi-structured (Parquet), and table-formatted (Iceberg) data for training, fine-tuning, checkpointing, and prompt engineering. Also hear different architectural patterns that customers use today to harness their business data for customized generative AI solutions.

Reserve your seat now!

Data drives transformation: Data foundations with AWS analytics – Innovation Talk

Thursday, November 30 | 2:00 – 3:00 PM PST | The Venetian

Join G2 Krishnamoorthy, Vice President of AWS Analytics, to discuss strategies for embedding analytics into your applications and ideas for building a data foundation that supports your business initiatives. With new capabilities for self-service and simpler builder experiences, you can democratize data access for line-of-business users, analysts, scientists, and engineers. Hear inspiring stories from adidas, GlobalFoundries, and the University of California, Irvine.

Reserve your seat now!

Breakout sessions

Deepen your learning with re:Invent breakout sessions. re:Invent breakout sessions are lecture-style and 1 hour long. These sessions take place across the re:Invent campus and cover all topics at all levels. The following are some of our favorites:

Accelerate innovation with real-time data: Join Mindy Ferguson, Vice President of Streaming and Messaging at AWS, and Arvinth Ravi, General Manager of Amazon Kinesis Data Streams, in this session that highlights the importance of implementing ubiquitous real-time data strategies to gain a competitive edge. Discover the latest launches in AWS streaming data services, gain insights into real-world applications, and explore how you can use them to solve a variety of use cases to make quick, real-time decisions to optimize costs, increase customer engagement, and drive growth. Reserve your seat now!
How Rivian builds real-time analytics from electric vehicles: Learn how Rivian’s vehicle data platform team uses Amazon MSK and other AWS analytics services to build a secure, scalable, cost-effective, and extensible data platform that powers different teams across the organization. Reserve your seat now!
Serverless data streaming: Amazon Kinesis Data Streams and AWS Lambda: Explore the intricacies of creating scalable, production-ready data streaming architectures using Amazon Kinesis Data Streams and AWS Lambda. Delve into tips and best practices essential to navigating the challenges and pitfalls inherent to distributed systems that arise along the way, and observe how AWS services work and interact. Reserve your seat now!
Break down data silos using real-time synchronization with Flink CDC: Learn how, by using Flink CDC on AWS, you can simplify real-time synchronization and ingestion across transactional databases and finally break down data silos. Reserve your seat now!
Scaling serverless data processing with Amazon Kinesis and Apache Kafka: Explore how to build scalable data processing applications using AWS Lambda. Learn practical insights into integrating Lambda with Amazon Kinesis and Apache Kafka using their event-driven models for real-time data streaming and processing. Design serverless data processing pipelines and extract valuable insights from real-time data streams. Reserve your seat now!

Chalk talks

Chalk talks are a highly interactive content format with a small audience. Each begins with a short lecture delivered by an AWS expert followed by a Q&A session with the audience.

Stream data and build transactional lakes with AWS Glue streaming: Learn how to create a low-code streaming pipeline with AWS Glue and Amazon Kinesis to load, clean, and transform sales data, and make it available for machine learning. Reserve your seat now!
Migrating self-managed Apache Flink to fully managed on AWS: Review key details to keep in mind when migrating self-managed Apache Flink applications to Amazon Managed Service for Apache Flink. Explore how using cloud-based services with the right patterns can reduce operational overhead. Hear about best practices that can help with migration while still maintaining costs, scale, and performance. Reserve your seat now!
How to migrate your Apache Kafka workloads to Amazon MSK: Amazon Managed Streaming for Apache Kafka (Amazon MSK) is an AWS streaming data service that manages Apache Kafka infrastructure and operations. Learn the benefits of migrating to Amazon MSK and how to lift and shift existing clusters using MirrorMaker 2.0 (MM2). Take a deep dive into the MM2 and Kafka Connect architecture, and explore the common bottlenecks and challenges of high-throughput data streams. Reserve your seat now!
Accelerating value from data: Migrate from batch to stream processing: Explore the competitive advantages of transitioning from traditional batch processing to stream processing. Look at key concepts and architectural patterns of streaming to address potential challenges and hiccups along this migration journey. Through an example use case, gain insights into how to successfully transition from batch to real-time data processing using AWS streaming data services. Reserve your seat now!
Fast-track streaming ETL with AWS streaming data services: Learn how to build streaming data pipelines across data lakes and data warehouses. Learn best practices for performance, scale, and cost control in Amazon Kinesis Data Streams, Amazon MSK, Amazon Redshift streaming ingestion, and AWS Glue streaming. Reserve your seat now!
Advanced real-time analytics and ML in your data warehouse: Learn how enterprise customers are rearchitecting their data environments to lean more heavily on streaming analytics to process throughput of millions of records per second. We will show you how to implement an ELT approach that uses Amazon Redshift’s native integration with streaming data services and Amazon Redshift ML to build near-real-time ML predictions in SQL for better decision-making. Reserve your seat now!
Building better with Apache Kafka and Amazon EventBridge: Explore the convergence of event-driven architectures and real-time event streaming to meet modern business requirements. Learn about Amazon EventBridge and Apache Kafka and gain insights into their roles for facilitating real-time decision-making, increased business velocity, and operational autonomy. Reserve your seat now!
Speed up data predictions with real-time inference at scale: It is crucial to analyze, learn, and make predictions from your data in a timely manner because the value of data diminishes over time. Dive into event-driven architecture and see how you can use machine learning inference with streaming data to detect patterns and anomalies to realize a faster path to decisions from your data. Reserve your seat now!

Code Talk

Code Talks are 60-minute sessions that focus on the actual code involved in building a solution. You’ll learn the “why” behind the solution and see it come to life—complete with the inevitable errors. Your questions are welcome and encouraged.

Build a Flink application on Amazon Managed Service for Apache Flink: Join this code talk to develop and host a streaming and production-ready Apache Flink application on Amazon Managed Service for Apache Flink using the DataStream and Table API programming models. Reserve your seat now!

Workshops

Workshops are 2-hour hands-on sessions where you work in teams to solve problems using AWS services. Workshops organize attendees into small groups and provide scenarios to encourage interaction, giving you the opportunity to learn from and teach each other. Don’t forget to bring your laptop!

Event detection with MSK and Amazon Managed Service for Apache Flink: Take on the role of a technology manager for a Las Vegas casino. Learn how to create a stream-processing application that identifies historically “high roller” casino guests entering the casino and that sends you an email when they sit down at a gambling table. Reserve your seat now!

Bootcamp

Bolster your confidence with AWS services and solutions in these eight-hour, deep-dive sessions. Learn by doing with expert guidance, immersive exercises, and self-paced labs.

The below is one you cannot miss:

Building streaming data analytics solutions on AWS: Learn to design and implement streaming data analytics solutions using AWS services with an expert AWS instructor. This 8-hour, deep-dive bootcamp is designed for data engineers, solutions architects, and developers who want to build and manage streaming data analytics pipelines. This bootcamp covers how to scale streaming applications using Amazon Kinesis; how to optimize data storage; how to select and deploy appropriate options to ingest, transform, store, and analyze data; and more. You can practice new skills in a real AWS environment with hands-on labs that cover data analytics solutions. Reserve your seat now!

Still time for FUN!

All work and no play … not at re:Invent! Sure, we’ll work hard and learn a lot, but we also plan to have a great time while we’re together. AWS Builder Labs will give you the opportunity to test your skills in sandbox settings while working alongside some of the leading minds from AWS!

Join the PeerTalk platform to easily find, connect, and network with others at re:Invent. Opt in to discover attendees who share similar interests—PeerTalk recommends connections for you. Browse recommendations, message people, and arrange meetups through the platform, then sit down for a face-to-face conversation in one of the four PeerTalk lounges located throughout the re:Invent campus.

Stop by the Analytics kiosk in the re:Invent expo hall. Connect with experts to dive deeper into AWS streaming data services such as Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Amazon Managed Service for Apache Flink, and Amazon Managed Streaming For Apache Kafka (Amazon MSK). Also, win prizes, show off your Analytics superhero, and learn all about the latest innovations from our AWS Analytics services.

Register today

It’s going to be an amazing event, and we can’t wait to see you at re:Invent 2023! Register now to secure your spot!

About the author

Anna Montalat is a Senior Product Marketing Manager for AWS streaming data services which includes Amazon Managed Streaming for Apache Kafka (MSK), Kinesis Data Streams, Kinesis Video Streams, Kinesis Data Firehose, and Kinesis Data Analytics. She is passionate about bringing new and emerging technologies to market, working closely with service teams and enterprise customers. Outside of work, Anna skis through winter time and sails through summer.

Monday, Nov 27	Tuesday, Nov 28	Wednesday, Nov 29
2:00 PM – 4:00 PM (PDT) Mandalay Bay BSI201 \| Build dashboards, reports, and explore generative BI in Amazon QuickSight	11:00 AM – 1:00 PM (PDT) MGM Grand ANT306 \| Build a data foundation to power your generative AI applications	11:30 AM – 1:30 PM (PDT) Ceasars Forum ANT312 \| Using Amazon OpenSearch Service as a vector database for gen AI apps
2:30 PM – 4:30 PM (PDT) Wynn ANT305-R \| Log analytics made easy with Amazon OpenSearch Serverless [REPEAT]	11:30 AM – 1:30 PM (PDT) Mandalay Bay ANT401-R \| Event detection with Amazon MSK and Amazon Managed Service for Apache Flink [REPEAT]	.
.	11:30 AM – 1:30 PM (PDT) MGM Grand BSI202 \| Quickly build predictive dashboards using no-code ML and generative BI	.
.	2:00 PM – 4:00 PM (PDT) Wynn ANT310 \| Share data across Regions and organizations for near-real-time insights	.
.	11:30 AM – 1:30 PM (PDT) Ceasars Forum ANT304-R \| A pragmatic approach to data governance on AWS [REPEAT]	.
.	11:30 AM – 1:30 PM (PDT) Venetian AIM350 \| Personalized marketing content with generative AI and Amazon Personalize	.
.	12:00 PM – 2:00 PM (PDT) Ceasars Forum ANT402 \| Protect and securely share the right data	.
.	12:00 PM – 2:00 PM (PDT) Wynn ANT307 \| Connect and analyze all your data with zero-ETL approaches	.

Your DevOps and Developer Productivity guide to re:Invent 2023

How to reserve a seat in the sessions

DevOps and Developer Productivity breakout sessions

What are breakout sessions?

Level 200 — Intermediate

DevOps and Developer Productivity chalk talks

What are chalk talks?

Level 300 — Advanced

DevOps and Developer Productivity workshops

What are workshops?

Level 300 — Advanced

Level 400 — Expert

DevOps and Developer Productivity builders’ sessions

What are builders’ sessions?

Level 300 — Advanced

Level 400 — Expert

DevOps and Developer Productivity lightning talks

What are lightning talks?

DevOps and Developer Productivity code talks

What are code talks?

Want to stay connected?

Explore the latest compute innovations

Customer experiences and applications with machine learning

Discover what powers AWS compute

Optimize your compute costs

Check out workload-specific sessions

Hear from AWS customers

Ready to unlock new possibilities?

Keynotes

Innovation Talks

Expo

Sessions

Plan your re:Invent agenda

About the author

The big stage

Innovation Talks

Sessions

Analytics kiosk in the AWS Village

AWS Analytics Superheroes

Get ready for re:Invent

About the author

How long have you been at AWS?

How do you explain your job to your non-tech friends?

How did you get started in data protection and cryptography? What about it piqued your interest?

What’s been the most dramatic change you’ve seen in the data protection landscape?

You are a frequent contributor to the AWS Security Blog. What were some of your recent posts about?

You are speaking in a couple of sessions at AWS re:Invent; what will your sessions focus on? What do you hope attendees will take away from your session?

What are you currently working on that you’re excited about?

Is there something you wish customers would ask you about more often?

How about outside of work, any hobbies?

2023 AWS Analytics Superheroes

Keynotes

Analytics Innovation Talk

Breakout sessions

Chalk talks

Builders’ sessions

Workshops

Code talks

Conclusion

About the authors

Keynotes and Innovation Talk sessions you cannot miss!

Breakout sessions

Chalk talks

Code Talk

Workshops

Bootcamp

Still time for FUN!

Register today

About the author

The collective thoughts of the interwebz