Tag Archives: AWS re:Invent

New Amazon CloudWatch and Amazon OpenSearch Service launch an integrated analytics experience

2024-12-01 Elizabeth Fuentes

Post Syndicated from Elizabeth Fuentes original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-and-amazon-opensearch-service-launch-an-integrated-analytics-experience/

Today, Amazon Web Services (AWS) announces a new integrated analytics experience and zero-ETL integration between Amazon CloudWatch and Amazon OpenSearch Service. This integration simplifies log data analysis and visualization without data duplication, streamlining log management while reducing technical overhead and operational costs. CloudWatch Logs customers now have access to two additional query languages beyond CloudWatch Logs Insights QL, while OpenSearch customers can query CloudWatch logs in place without creating separate extract, transform, and load (ETL) pipelines.

Organizations often need different analytics capabilities for their log data. Some teams prefer CloudWatch Logs for its scalability and simplicity in centralizing logs from all their systems, applications, and AWS services. Others require OpenSearch Service for advanced analytics and visualizations. Previously, integration between these services required maintaining separate ingestion pipelines or creating ETL processes. This new integration helps customers get the best of both services by eliminating this complexity by bringing the power of OpenSearch analytics directly to CloudWatch Logs, without any data copy.

Amazon CloudWatch Logs now supports OpenSearch Piped Processing Language (PPL) and OpenSearch SQL directly within the CloudWatch Logs Insights console. You can use SQL to analyze data and correlate logs using JOIN. You can use SQL functions (such as JSON, mathematical, datetime, and string functions) for intuitive log analytics. You can also use the OpenSearch PPL to filter, aggregate, and analyze data. With a few clicks, you can access pre-built, out-of-the-box dashboards for vended logs, such as Amazon Virtual Private Cloud (VPC), AWS CloudTrail, and AWS WAF. These dashboards enable faster monitoring and troubleshooting through visualizations, such as analyzing flows over time, top talkers, megabytes, and packets transferred over time, without having to configure individual widgets or build specific queries. You can analyze VPC flows over time, identify top talkers, track network traffic metrics, monitor web request trends in AWS WAF, or analyze API activity patterns in AWS CloudTrail.

Additionally, OpenSearch Service users can now analyze CloudWatch logs using OpenSearch Discover and run SQL and PPL, similar to how they analyze data in Amazon Simple Storage (Amazon S3), and build indexes and create dashboards directly without any ETL operations or separate ingestion pipelines.

Let’s explore how this integration works
To demonstrate the new OpenSearch SQL and PPL query capabilities in CloudWatch, I start in the CloudWatch console. In the navigation pane, I choose Logs then Logs Insights. After selecting log groups for the query, I can now use OpenSearch PPL or OpenSearch SQL query languages directly within CloudWatch Logs Insights, with no additional setup or integration required. Using this new capability, I can write complex queries using familiar SQL syntax or OpenSearch PPL, making log analysis more intuitive and efficient. In the Query commands menu, you can find sample queries to help you get started.

This example demonstrates how to use SQL JOIN to combine data from two log groups: pet adoptions and pet availability. By filtering for specific customer IDs, you can analyze related log records and trace IDs for troubleshooting purposes.

One of the powerful features of this integration for CloudWatch Logs customers is the ability to create pre-built dashboards for Amazon VPC Flows, AWS CloudTrail and AWS WAF logs. Let’s explore this by creating a dashboard for AWS WAF logs. In the Analyze with OpenSearch tab, I choose Settings and follow the steps.

After a few minutes, my integration is ready and I go to Create an OpenSearch dashboard. In the options Select automatic dashboard type, I choose AWS WAF logs.

In the Dashboard data configuration tab, I can select Data synchronization frequency to occur every 15 minutes. I Select the log groups and View log samples of the selected log groups. I finish by choosing Create a dashboard.

After creating my dashboard, I can explore my logs. The AWS WAF logs dashboard provides comprehensive visibility into web application firewall metrics and events, with automatically configured visualizations that help you monitor and analyze security patterns.

Similarly, the CloudTrail dashboard offers deep insights into API activity across your AWS environment. It’s useful for monitoring API activity, auditing actions, and identifying potential security or compliance issues.

The VPC Flow Logs dashboard provides detailed visualization of key metrics from your logs for network traffic analysis. You can analyze network traffic, detect unusual patterns, and monitor resource usage. The dashboard currently supports only VPC v2 fields (default format). Custom formatted fields are not supported.

With zero-ETL to access CloudWatch data from OpenSearch Services, I also can build an OpenSearch dashboard from the OpenSearch Service console without having to build and maintain an ETL process. For this, I go to Central management, then I select the new Connected data sources menu, click choose Connect to create a new connected data source, and choose CloudWatch Logs.

In the next step, I name my data source and choose to Create a new role, which must have the necessary permissions to execute actions on OpenSearch Service. You can see them in the Sample custom policy.

In the Set up OpenSearch step, configure a OpenSearch data connection for CloudWatch Logs by selecting Create a new collection. As part of setting up the CloudWatch Logs source, a new OpenSearch Service serverless collection and OpenSearch UI application is created to store the indexed views and provide a user interface to analyze your CloudWatch Logs data. I create a new collection, name it, and conﬁgure the OpenSearch application and workspace within the application. After setting the Data retention days, I choose Next and ﬁnish with Review and connect.

When the integration with CloudWatch is ready, I can choose between Explore logs without indexing data which will take me to a querying interface in Discover or Explore vended logs by creating a dashboard for Amazon VPC Flows, CloudTrail and AWS WAF logs.

After I select Explore logs, OpenSearch UI takes me to Discover in the application workspace I created during the data source setup. In Discover, I select the data picker and choose View all available data to access my CloudWatch Logs data source and log groups.

After I select the log groups, I can analyze my CloudWatch logs using OpenSearch SQL and PPL directly in Discover, without having to switch between applications.

To create a dashboard, I return to the Connected data sources overview page on the console. From there, I select Create dashboard, which allows me to visually analyze my CloudWatch data without having to define queries or build visualizations, as I previously did in the CloudWatch console

After the dashboard is created, I navigate to OpenSearch resources where I can see the newly created indexes being populated with data in my Collection. After I have the data, I can go to the dashboard with the data from the CloudWatch logs that I selected in the configuration, and as more data comes in, it will be displayed in near real-time on the OpenSearch dashboard.

With this zero-ETL integration you can ingest data directly into OpenSearch, using its powerful query capabilities and visualization features while maintaining data consistency and reducing operational overhead.

Integration Highlights
For CloudWatch customers:

Query capabilities – Streamline log investigation by using OpenSearch SQL and PPL queries directly within the CloudWatch Logs Insights console.
Analytics features – With a few clicks, access pre-built, out-of-the-box dashboards for vended logs, such as VPC, AWS WAF, and CloudTrail logs. These dashboards enable faster monitoring and troubleshooting through visualizations for analyzing flows over time, top talkers, megabytes, and packets transferred over time, without having to configure individual widgets or build specific queries.
Getting started for CloudWatch users – Configure integration from CloudWatch Logs to OpenSearch Service. For more information refer to the Amazon CloudWatch Logs query capabilities and Amazon CloudWatch Logs vended dashboard documentation.

For OpenSearch Service customers:

Zero-ETL integration – Access and analyze CloudWatch data directly from OpenSearch Service without building or maintaining ETL processes. This integration eliminates separate ingestion pipelines while reducing storage costs and operational overhead through simplified data management and zero data duplication.
Getting started for OpenSearch users – Create a data connection selecting CloudWatch as a data source from OpenSearch Service. For more information, refer to the Amazon OpenSearch Service Developer Guide.

Regional availability and pricing
This integration is now available in AWS Regions where Amazon OpenSearch Service direct query is available. For pricing details and free trial information, you can visit the Amazon CloudWatch Pricing and Amazon OpenSearch Service Pricing pages.

PS: Writing a blog post at AWS is always a team effort, even when you see only one name under the post title. In this case, I want to thank Joshua Bright, Ashok Swaminathan, Abeetha Bala, Calvin Weng, and Ronil Prasad for their generous help with screenshots, technical guidance, and sharing their expertise in both services, which made this integration overview possible and comprehensive.

— Eli

Your DevOps and Developer Productivity guide to 2024 re:Invent

2024-11-21 Artur Rodrigues

Post Syndicated from Artur Rodrigues original https://aws.amazon.com/blogs/devops/your-devops-and-developer-productivity-guide-to-2024-reinvent/

It’s that time of the year again. The annual AWS re:Invent conference is just around the corner. Still need to save your spot? You can register here.

This year’s DevOps and Developer Productivity (DOP) track features an impressive lineup, including 11 breakout sessions, 14 chalk talks, 2 code talks, 8 workshops, 3 builder sessions, and 2 lightning talks.

I have curated a list of the DOP sessions that you should pay attention. I also invite you to visit the re:Invent catalog to explore the full range of DOP offerings. There is a collection of GenAI related sessions, leveraging Amazon Q Developer and Amazon Bedrock, as well as the usual AWS DevOps tools that we all love, Infrastructure as Code (IaC), Continuous Integration and Continuous Deployment (CI/CD).

How to reserve a seat in the sessions

Reserved seating is available for registered attendees to secure seats in the sessions of their choice. Reserve a seat by signing in to the attendee portal and navigating to “Event”, then “Sessions”.

Do not miss the Innovation Talk led by VP of Developer Experience, Adam Seligman. In DOP220 – Reimagining the Developer Experience at AWS – Software development is undergoing a seismic shift driven by generative AI, transforming how developers work, what they build, and who can become a developer. AWS empowers developers to fearlessly embrace this evolution, integrating cutting-edge yet responsible generative AI solutions across the development lifecycle. Explore real-world use cases accelerating legacy modernization, elevating cloud-native innovation, and unlocking remarkable results. Gain insights into AWS’s pragmatic approach, fueling creativity and customer impact. Join the vibrant community on this transformative journey, where generative AI is redefining software development, opening new frontiers for innovation, and democratizing access to coding for diverse creators shaping technology’s future.

DevOps and Developer Productivity breakout sessions

What are breakout sessions?

AWS re:Invent breakout sessions are lecture-style and 60 minutes long. These sessions are delivered by AWS experts and typically reserve 10–15 minutes for Q&A at the end. Breakout sessions are recorded and made available on-demand after the event.

DOP201 – AWS infrastructure as code: A year in review – AWS provides services that enable the creation, deployment and maintenance of application infrastructure in a programmatic, descriptive, and declarative way. These services provide rigor, clarity, and reliability to application development. Join this session to learn about the new features and improvements for AWS infrastructure as code with AWS CloudFormation and AWS Cloud Development Kit (AWS CDK) and discover how they can benefit your team.

DOP202 – Continuous integration and continuous delivery (CI/CD) for AWS – AWS provides one place where you can plan work, collaborate on code, and build, test, and deploy applications with continuous integration and continuous delivery (CI/CD) tools. In this session, learn about creating complete CI/CD pipelines using infrastructure as code on AWS.

DOP204 – Amazon Q Developer: Your gen AI assistant for software development – In this session, learn how Amazon Q Developer is transforming the developer experience by speeding up a range of tasks that support you as you research how to get started, evaluate system design, build secure and scalable applications, upgrade existing applications, and optimize application performance. Learn firsthand how Amazon Q capabilities for building, troubleshooting, and transforming applications faster and more easily frees you up to focus on experimentation and innovation.

DOP209 – Accelerate application maintenance and upgrades with generative AI – Developers spend significant time completing the undifferentiated work of maintaining and upgrading legacy applications. Teams need to balance investments in building new features with mandatory patching and update work. Now, using the power of generative AI, the Amazon Q Developer agent for code transformation can expedite these critical upgrade tasks, transforming applications to use the latest language features and versions in hours or days and saving significant costs. Join the session to learn what’s new and how your team can automate Java application upgrades.

DOP214 – Unleashing generative AI: Amazon’s journey with Amazon Q Developer – Join us to discover how Amazon rolled out Amazon Q Developer to thousands of developers, trained them in prompt engineering, and measured its transformative impact on productivity. In this session, learn best practices for effectively adopting generative AI in your organization. Gain insights into training strategies, productivity metrics, and real-world use cases to empower your developers to harness the full potential of this game-changing technology. Don’t miss this opportunity to stay ahead of the curve and drive innovation within your team.

DevOps and Developer Productivity chalk talks

What are chalk talks?

Chalk Talks are highly interactive sessions with a small audience. Experts lead you through problems and solutions on a digital whiteboard as the discussion unfolds. Each begins with a short lecture (10–15 minutes) delivered by an AWS expert, followed by a 45- or 50-minute Q&A session with the audience.

DOP318 – Prompt engineering expertise: Unleashing code with Amazon Q Developer – Dive into the art of prompt engineering and discover how to harness the full potential of Amazon Q Developer, AWS’s cutting-edge generative AI service. Learn techniques to craft compelling prompts that yield remarkable code generation results. Explore strategies to provide contextual information beyond prompts, such as import statements, to enhance the accuracy and relevance of your AI-generated code. Elevate your software development workflow and unleash the transformative capabilities of generative AI.

DOP324 – Incorporating generative AI in the CI/CD pipeline – In this chalk talk, discover how generative AI can revolutionize your continuous integration and delivery (CI/CD) pipeline. Learn how AI models can analyze code changes and generate recommendations for safe deployments. Explore automated orchestration capabilities that trigger deployments, monitor metrics, and adapt strategies. Gain insights into using AI for continual monitoring and self-improving release cycles, streamlining your software delivery while minimizing risks and manual efforts.

DOP314 – Automate Java app upgrades & accelerate innovation with generative AI – Amazon Q Developer’s agent for code transformation automates the end-to-end process of upgrading and transforming code. Reduce the time and costs associated with modernizing applications, unlock previously cost-prohibitive and cumbersome modernization opportunities, and save customers months or even years of effort. By automating undifferentiated upgrade and modernization tasks, customers can enhance application performance and security and accelerate innovation. Join this chalk talk to learn how to take your application modernization to the next level.

DOP323 – From Windows to Linux: .NET application modernization – Porting and upgrades of .NET applications running on Windows servers to Linux can deliver cost savings and enhance security and compliance, but the modernization process can be long and laborious. This interactive chalk talk explores strategies for porting server-side components of a .NET application within days by refactoring. The session includes codebase analysis, code decomposition into buildable units, transformation plan creation, and execution of key transformation tasks with approval from the developer.

DevOps and Developer Productivity workshops

What are workshops?

Workshops are two-hour interactive learning sessions where you work in small group teams to solve problems using AWS services. Each workshop starts with a short lecture (10–15 minutes) by the main speaker, and the rest of the time is spent working as a group.

DOP304 – Develop AWS CDK resources to deploy your applications on AWS – In this workshop, learn how to build and deploy applications using infrastructure as code with AWS Cloud Development Kit (AWS CDK). Create resources using AWS CDK, and learn maintenance and operations tips. In addition, get an introduction to building your own constructs. You must bring your laptop to participate.

DOP305 – Modern CI/CD with GitHub and AWS CodePipeline – In this workshop, learn how to build modern continuous integration and continuous delivery (CI/CD) pipelines using GitHub and AWS CodePipeline through the AWS Management Console. Learn how to work with monorepos and branching strategies. Explore advanced features such as automatic rollbacks, pipeline parameters, stage level conditions, and concurrent execution modes to improve your pipeline performance. You must bring your laptop to participate.

DOP309 – The Amazon Q Developer coding challenge – Join this workshop to participate in 20 increasingly complex coding challenges aided by Amazon Q Developer, an AI-powered assistant for software development. Discover how Amazon Q Developer’s auto-generated code recommendations and chat explanations can help you develop code and understand complex algorithmic coding challenges more efficiently compared to manual coding alone. Learn about Amazon Q Developer capabilities and how it can help you improve productivity. You must bring your laptop to participate.

DOP325 – Boost code quality with generative AI – In this hands-on workshop, you unleash the power of generative AI to boost code quality using Amazon Q Developer. You learn to use Amazon Q Developer to generate unit tests and documentation automatically, addressing the challenge of balancing new feature development with writing unit tests and documentation. By the end of the workshop, you have firsthand experience streamlining your development process and freeing up time to focus on core feature development. Come follow along with step-by-step instructions and gain practical experience with this cutting-edge AWS service. You must bring your laptop to participate.

DevOps and Developer Productivity builders’ sessions

What are builders’ sessions?

These 60-minute group sessions are led by an AWS expert and provide an interactive learning experience for building on AWS. Builders’ sessions are designed to create a hands-on experience where questions are encouraged.

DOP205 – Learning new skills with Amazon Q Developer – Experience the power of Amazon Q Developer, your AI-powered assistant for software development. In this session, explore how Amazon Q Developer can streamline your daily workflow on AWS. Stuck in the console? Open the Amazon Q Developer panel for instant assistance. Can’t find your way through the documentation? Amazon Q Developer guides you effortlessly. Need help crafting CLI commands? Amazon Q Developer has you covered. Want assistance right in Slack or Microsoft Teams? Amazon Q Developer is by your side, helping you work smarter, faster, and more efficiently across your favorite tools. You must bring your laptop to participate.

DOP302 – Creating secure code with Amazon Q Developer – In this builders’ session, gain hands-on experience using Amazon Q Developer to create secure code. Write unit tests, optimize code, and scan for vulnerabilities, and discover how Amazon Q Developer suggests remediations that help fix your code instantaneously. Also, learn how you can use Amazon Q Developer security scanning to outperform other publicly benchmarkable tools on detection across popular programming languages. You must bring your laptop to participate.

DOP401 – Modernizing Java applications with Amazon Q Developer – In this builders’ session, use Amazon Q Developer Agent for code transformation to modernize a Java application. Learn how Amazon Q Developer can leverage generative AI to automate common language upgrade tasks like updating your code, conducting unit tests, and verifying deployment readiness starting with Java. Save days’ or even months’ worth of the undifferentiated work involved in moving from older language versions. You must bring your laptop to participate.

DevOps and Developer Productivity lightning talks

What are lightning talks?

Lightning talks are short, 20-minute demos led from a stage.

DOP217 – Best practices for customizing Amazon Q Developer – With Amazon Q Developer, you can securely connect to your private repositories to generate even more relevant code recommendations based on your internal code repositories, ask questions about your company code, and understand your internal code bases faster. In this session, learn how to set up customizations and generate code based on your internal repos. Use the Amazon Q Developer chat in your IDE to ask questions about how your internal code base is structured, where and how certain functions or libraries are used, and how to use specific functions, methods, or APIs.

DOP219 – How NAB uses Amazon Q Developer for increased productivity – Significantly accelerate development by customizing Amazon Q Developer to generate even more relevant inline code recommendations and chat responses (in preview) by making it aware of your internal libraries, APIs, best practices, and architectural patterns. In this lightning talk, you learn how National Australia Bank (NAB) is using Amazon Q Developer to help their development teams ship faster, and innovate more for their customers, by using customizations.

DevOps and Developer Productivity code talks

What are code talks?

Code talks are 60-minute, highly-interactive discussions featuring live coding. Attendees are encouraged to dig in and ask questions about the speaker’s approach.

DOP313 – Get tailored code insights with Amazon Q Developer and private repos – Unlock the full potential of Amazon Q Developer with customized code recommendations tailored to your organization’s code base. In this code talk, learn how to securely connect Amazon Q to your private repositories, enabling it to generate highly relevant code suggestions based on your internal coding practices. Discover how to create and utilize customizations, and witness firsthand the transformative impact on code comprehension and development efficiency by comparing suggestions with and without customization. Elevate your coding experience with this powerful feature.

DOP315 – Optimize your cloud environments in the AWS console with generative AI – Available in the AWS Management Console, Amazon Q Developer is the only AI assistant that is an expert on AWS, helping developers and IT pros optimize their AWS cloud environments. Proactively diagnose and resolve errors and networking issues, provide guidance on architectural best practices, analyze billing information and trends, and use natural language in chat to manage resources in your AWS account. Learn how Amazon Q Developer accelerates task completion with tailored recommendations based on your specific AWS workloads, shifting from a reactive review to proactive notifications and remediation.

Want to stay connected?

Get the latest updates for DevOps and Developer Productivity by following us on Twitter and visiting the AWS devops blog.

If you are unable to join us in-person, Breakout Sessions will be available via our YouTube channel after the event. Contact your AWS Account Team is you are interested in learning more about any of these sessions or how to bring our experts to you.

We look forward to seeing you at re:Invent 2024!

Know before you go – AWS re:Invent 2024 cloud resilience

2024-11-19 Shllomi Ezra

Post Syndicated from Shllomi Ezra original https://aws.amazon.com/blogs/architecture/know-before-you-go-aws-reinvent-2024-cloud-resilience/

With AWS re:Invent 2024 just weeks away, the excitement is building and we’re looking forward to seeing you all soon. If you’re attending re:Invent with the goal of improving your organization’s cloud resilience operations, we will be offering valuable insights, best practices, and fun activities to improve your cloud resilience expertise.

This year, we’re offering more than 100 resilience breakout sessions, workshops, chalk talks, builders’ sessions, and code talks. Find the complete list in the re:Invent 2024 session catalog and filter by “Resilience” in the area of interest field.

In this post, we highlight must-see sessions for those building resilient applications and architectures on AWS. Reserved seating is now open, so act quickly to claim your seat. Be sure to also check out the vertical-specific re:Invent guides.

Our recommendations are divided into three topics to help you choose the sessions most relevant to your business: resilience fundamentals, advanced resilience patterns, and resilience for customers operating in regulated industries.

What is cloud resilience all about?

Cloud resilience refers to the ability for an application to resist or recover from disruptions, including those related to infrastructure, dependent services, misconfigurations, transient network issues, and load spikes. Cloud resilience also plays a critical role in an organization’s broader business resilience strategy, including the ability to meet digital sovereignty requirements. Resilient applications are those built with high availability—the percentage of time the application is available for use—and also those with a disaster recovery or continuity of operations plan in place.

Resilience fundamentals

Join us as we explore the strategies, tools, and mindsets that enable organizations to thrive in the face of uncertainty. These sessions cover conceptual overviews and demos of AWS cloud resilience services.

Breakout sessions

Failing without flailing: Lessons we learned at AWS the hard way (ARC333)

At AWS, we’ve learned that building resilient services requires more than just designing for high availability. In this session, AWS operational leaders are back for more insights on how to mitigate impact when, not if, the unexpected happens. Hear a few short stories collected from 18 years of operational excellence, with practical advice on preparing for and mitigating failure.

Think big, build small: When to scale and when to simplify (ARC331)

Join this session to learn how to navigate the complexities of cloud architecture. Hear insights and guidance developed from working with successful AWS customers, including how to optimize for business value and agility. Discover the AWS approach to architectural tiers, engineering simplicity and reliability, and treating infrastructure as an investment.

Mastering resilience at every layer of the cake (ARC327)

Join this session to learn resilience at various levels, from platform to applications, using AWS services like AWS Resilience Hub, AWS Fault Injection Service, ARC, Amazon Elastic Disaster Recovery, and AWS Backup. You’ll leave with a mental model for resilience across these layers, and ready-to-use reference architectures and guidance. The session includes demos for a fun, lively experience.

Building resilient applications on AWS with Capital One (ARC334)

In this session, discover the patterns and principles of AWS resilience best practices. Then, hear Capital One showcase its next-generation design and deployment patterns that push the boundaries of resilient architectures and support its most critical business processes. Learn about the AWS services it uses, the trade-offs it must consider, and the decision matrix that guides developers to the right pattern for the right use case.

Data protection and resilience with AWS storage (STG301)

Join this session to dive deep on how AWS storage offers organizations defense-in-depth data protection and resilience for application data across recovery point and time objectives, helping mitigate risks with immutable solutions, restore testing, policy-based access controls, encryption, and auditing and reporting.

Workshops

Building, operating, and testing resilient Multi-AZ applications (ARC303)

Join this workshop to get hands-on experience building, operating, and testing a resilient Multi-AZ application.

Building resilient architectures with observability (COP308)

Explore how to use AWS services, including AWS Resilience Hub, Amazon CloudWatch, and AWS Fault Injection Service, to build resilient and reliable cloud-based applications.

Advanced resilience patterns

Building resilient and reliable applications in the cloud is critical for organizations running mission-critical workloads. Unexpected outages, latency spikes, or performance issues can have severe business impact. The sessions and workshops in this track explore advanced techniques and tools to help you proactively identify and address resilience weaknesses in your systems. Learn how to use chaos engineering, multi-Region architectures, and the latest AWS services and best practices to enhance the resilience and operational excellence of your cloud applications.

Breakout sessions

Chaos engineering: A proactive approach to system resilience (ARC326)

This session demonstrates the benefits of chaos engineering in action. Gain insights from BMW Group’s transformative journey, learning key lessons on scaling chaos engineering across the organization, and how BMW Group conducts large-scale chaos experiments in production, uncovering issues and fostering a culture of greater resilience and continuous improvement.

Try again: The tools and techniques behind resilient systems (ARC403)

Grand architectural theories are nice, but what makes systems resilient is in the details. Marc Brooker, VP and distinguished engineer, looks at some of the resiliency tools and techniques AWS uses in its systems. Marc rethinks, retries, breaks open circuit breakers, decodes erasure coding, and tackles the tail. Learn about formal methods and simulation, and how these tools help build faster code, faster.

Multi-Region or single Region? Considerations and architectures (ARC309)

Watch experts walk through and whiteboard architectures that take advantage of AWS services that support multi-Region capabilities, and discuss what a failover scenario would look like in real life. Leave with an understanding of what it takes to run a multi-Region architecture on AWS.

Best practices for creating multi-Region architectures on AWS (ARC323)

In this session, learn the two critical areas you’ll need to consider. First, explore different failover strategies and the trade-offs between them. Then, learn how to make the decision to initiate a cross-Region failover as well as what goes into the process. Lastly, hear from Samsung Account about their multi-Region application and how they think about these two critical areas.

Workshops

Chaos engineering workshop (ARC322)

This workshop introduces AWS Fault Injection Service for running controlled resilience experiments to improve application performance, observability, and resilience. You must bring your laptop to participate.

Gen AI resilience: Chaos engineering with AWS Fault Injection Service (ARC305)

Learn how to construct a useful hypothesis backlog for generative AI applications and how to use AWS Fault Injection Service to run those experiments. You must bring your laptop to participate.

Building operational resilience in workloads using generative AI (SUP401)

Building operational resilience requires proactive identification and mitigation of risks. In this workshop, use AWS managed generative AI services in real-world scenarios to learn how to assess readiness, proactively improve your architecture, react quickly to events, troubleshoot issues, and implement effective observability practices. Also use AWS Countdown and the AWS Well-Architected Framework as the entry point reference frameworks to use generative AI services for operation. Through hands-on activities, learn strategies for debugging issues, detecting anomalies and incidents, and optimizing architectures to improve the resilience of your workloads. You must bring your laptop to participate.

Resilience for customers operating in regulated industries

In regulated industries like finance, healthcare, and telecom, resilient architecture is critical for compliance, security, and operational continuity. These sectors face strict regulations that demand robust data protection, disaster recovery, and uptime guarantees. A resilient architecture helps organizations maintain service availability, minimize downtime, and recover quickly from disruptions, safeguarding sensitive data and avoiding regulatory breaches. It also enables businesses to adapt to evolving regulations while delivering secure, uninterrupted services.

Breakout sessions

Fidelity Investments: Building for mission-critical resilience (FSI318)

This session explores the transformation of Fidelity Investments’s trade processing platform on AWS and the critical role resiliency plays in preserving operational integrity.

Service event replay: Stress-testing your architecture’s resilience (FSI314)

Learn how to assess the resiliency of your own architectures and develop strategies to strengthen your response and recovery capabilities.

Workshops

Scaling multi-tenant SaaS with a cell-based architecture (ARC402)

In this workshop, see how cell-based architectures provide you with new ways to group, deploy, scale, and operate your multi-tenant workloads. Also see how this approach influences the tiering, scaling, and resilience profile of your SaaS architecture. You must bring your laptop to participate.

Advanced cross-Region DR patterns on AWS (ARC401)

Join this hands-on workshop to explore a resilient, cloud-centered architecture that surpasses the stringent availability and recovery regulations for financial markets utility providers. You must bring your laptop to participate.

Meet experts at the AWS Cloud Resilience kiosk

Throughout the re:Invent week, if you have any questions or suggestions for the AWS Cloud Resilience team, drop by the Cloud Resilience kiosk at the AWS Village in the 2024 re:Invent Expo (the Venetian).

This post was copyedited for grammar, spelling, capitalization, punctuation, terminology, and legal issues. Other important issues are noted in comments, and you should consider revising the content accordingly before publication.

Your guide to AWS Analytics at AWS re:Invent 2024

2024-11-14 Imtiaz Sayed

Post Syndicated from Imtiaz Sayed original https://aws.amazon.com/blogs/big-data/your-guide-to-aws-analytics-at-aws-reinvent-2024/

It’s AWS re:Invent time, where you turn your ideas into reality. Get a front row seat to hear real stories from AWS customers, experts and leaders about navigating pressing topics like generative AI and data analytics.

For data enthusiasts and data professionals alike, this blog is a curated and comprehensive guide to all analytics sessions, for you to efficiently plan your itinerary. Secure your spot early for must-attend sessions through the attendee portal. Can’t join in person? No worries – grab a free pass to stream live sessions online.

Join us at the AWS Analytics Kiosk in the AWS Village Expo to get your data questions answered by AWS experts, to dive deeper into re:Invent launches, participate in a data-centric quiz and AWS authored book giveaways.

Keynotes

Join AWS CEO Matt Garman to hear how AWS is innovating across every aspect of the world’s leading cloud. He explores how we are reinventing foundational building blocks as well as developing brand new experiences, all to empower customers and partners with what they need to build a better future.

Join Dr. Swami Sivasubramanian, VP of AI and Data at AWS, to discover how you can use a strong data foundation to create innovative and differentiated solutions for your customers. Hear from customer speakers with real-world examples of how they’ve used data to support a variety of use cases, including generative AI, to create unique customer experiences.

Join Dr. Werner Vogels, VP and CTO at Amazon.com, as he shares the critical lessons and strategies he’s learned for managing increasingly complex systems. The keynote explores the core principles for embracing complexity, drawing on Amazon experiences building distributed systems at massive scale.

Analytics Innovation Talk

The boundaries between data analytics and AI are blurring as data workers’ behaviors evolve, and previously distinct data roles and use cases converge. Getting to near real-time, trustworthy insights has become paramount, so data workers are seeking seamless collaboration and interoperability across tools and data sources. In this talk, join Sirish Chandrasekaran, Director for Data Warehousing at AWS, and Rick Sears, Director for Data Processing at AWS, to envision a future with AWS where your data workers can effortlessly move between analyzing historical patterns, predicting future scenarios, and automating decision flows at scale, breaking through disparate tools and siloed workflows.

Breakout sessions

Dive into cutting-edge topics with re:Invent breakout sessions. These immersive, hour-long lectures are led by AWS experts, customers, and partners, offering you unparalleled insights and knowledge in a concise format. Whether you’re exploring the latest in cloud technology, AWS Analytics advancements, or industry-specific solutions, these sessions are designed to expand your horizon and inspire your next big idea.

Monday, Dec 2

Tuesday, Dec 3

Wednesday, Dec 4

Thursday, Dec 5

8:30 AM – 9:30 AM PST | MGM Grand

ANT324 | Accelerate value from data: Migrating from batch to stream processing

12:00 PM – 1:00 PM PST | Caesars Forum

ANT341 | Enhance performance with observability, security, and log analytics

8:30 AM – 9:30 AM PST | Mandalay Bay

ANT343 | Monitor and manage data quality

11:00 AM – 12:00 PM PST |Mandalay Bay

ANT325 | Achieve seamless and secure data sharing

10:00 AM – 11:00 AM PST| Mandalay Bay

ANT335 | Build highly performant data solutions with serverless analytics

12:00 PM – 1:00 PM PST | MGM Grand

ANT327 | What’s new: Data streaming on AWS

9:00 AM – 10:00 AM PST | MGM Grand

BSI201|Supercharge your apps with embedded Amazon QuickSight and Amazon Q

11:00 AM – 12:00 PM PST | MGM Grand

ANT340 | Revolutionize your search applications for generative AI

1:00 PM – 2:00 PM PST | Mandalay Bay

ANT342 | Operate and scale managed Apache Kafka and Apache Flink clusters

1:30 PM – 2:30 PM PST | MGM Grand

ANT349| Innovations in AWS analytics: Data warehousing and SQL analytics

10:00 AM – 11:00 AM PST | Mandalay Bay

ANT336 | Build large-scale transactional data lakes with open table formats

11:30 AM – 12:30 PM PST | Caesars Forum

ANT344 | Cost-effective data processing with Amazon EMR

1:00 PM – 2:00 PM PST | Wynn

BSI101 | Reimagine business intelligence with generative AI

1:30 PM – 2:30 PM PST | Wynn

ANT334 | Scale with self-service analytics on AWS

11:30 AM – 12:30 PM PST | Mandalay Bay

ANT329 | What’s new in search, observability & vectors in Amazon OpenSearch Service

5:30 PM – 6:30 PM PST | Mandalay Bay

ANT347 | Maximize efficiency and reduce costs with Amazon OpenSearch Service

2:30 PM – 3:30 PM PST | Mandalay Bay

ANT328 | AI-powered analytics with Amazon Redshift Serverless & data sharing

11:30 AM – 12:30 PM PST | Mandalay Bay

BSI102 | What’s new with Amazon QuickSight

3:00 PM – 4:00 PM PST | Caesars Forum

ANT346 | Innovations in AWS analytics: Data processing

12:00 PM – 1:00 PM PST| Mandalay Bay

ANT339 | Scaling to new heights with Amazon Redshift multi-cluster architecture

4:00 PM – 5:00 PM PST | Venetian

ANT302 | Data foundation in the age of generative AI

1:00 PM – 2:00 PM PST | Mandalay Bay

ANT303 | Explore what’s new in data governance with AWS analytics

4:00 PM – 5:00 PM PST | Mandalay Bay

ANT202 | Demystify and democratize access to your data with a business catalog

1:00 PM – 2:00 PM PST | Mandalay Bay

ANT330 | Solving different data ingestion use cases with AWS

4:00 PM – 5:00 PM PST | Wynn

ANT348 | Innovations in AWS analytics: Zero-ETL and data integrations

4:00 PM – 5:00 PM PST | Mandalay Bay

BSI206 | Scale BI to all your users with Amazon Q in QuickSight

5:30 PM – 6:30 PM PST | MGM Grand

BSI205 | Migrate to QuickSight: Reduce costs and increase productivity

5:30 PM – 6:30 PM PST | MGM Grand

ANT345 | Modernize your data warehouse by moving to Amazon Redshift

Chalk talks

These hour-long, highly engaging sessions offer a unique blend of expert insight and collaborative learning. An AWS specialist kicks off with a concise, informative lecture, setting the stage for an in-depth, interactive Q&A. With a limited audience size, you’ll have the opportunity to dive deep into topics, ask pressing questions, and engage in meaningful discussions with both the presenter and fellow attendees.

Monday, Dec 2

Tuesday, Dec 3

Wednesday, Dec 4

Thursday, Dec 5

10:00 AM – 11:00 AM PST | MGM Grand

ANT316-R | Architectural patterns for near real-time data analytics on AWS

11:30 AM – 12:30 PM PST | Caesars Forum

ANT315 | Amazon OpenSearch Service cost optimizations

10:00 AM – 11:00 AM PST | Mandalay Bay

ANT305 | Strategies for efficient zero-ETL integrations

11:00 AM – 12:00 PM PST | Mandalay Bay

ANT304-R1 | Accelerating the shift from batch to stream processing

11:30 AM – 12:30 PM PST | MGM Grand

ANT337-R | Cost optimization for data analytics on AWS

11:30 AM – 12:30 PM PST | Wynn

ANT410 | Maximize your data performance with Amazon EMR on Amazon EKS

10:30 AM – 11:30 AM PST | MGM Grand

BSI304 | Security and governance: Safeguarding your data with Amazon QuickSight

12:30 PM – 1:30 PM PST | Mandalay Bay

ANT411 | Data integration with AWS Glue and Amazon MWAA

11:30 AM – 12:30 PM PST | MGM Grand

ANT337-R | Cost optimization for data analytics on AWS

12:00 PM – 1:00 PM PST | MGM Grand

ANT332-R | Democratize generative AI data access without compromising on security

11:30 AM – 12:30 PM PST | Mandalay Bay

ANT331-R1 | Build your data strategy for generative AI with Amazon Redshift

12:30 PM – 1:30 PM PST | MGM Grand

ANT409-R1 | Optimize Apache Spark workloads with Amazon EMR Serverless

1:00 PM – 2:00 PM PST | Mandalay Bay

ANT318 | Build serverless streaming data pipelines for real-time analytics

1:00 PM – 2:00 PM PST | MGM Grand

ANT326 | Build multi-tenant data processing architectures

1:00 PM – 2:00 PM PST | Wynn

ANT317-R1 | Best practices for migrating to Amazon OpenSearch Service

2:00 PM – 3:00 PM PST | Mandalay Bay

ANT320 | Data preparation authoring with AWS Glue Studio

2:30 PM – 3:30 PM PST | Mandalay Bay

ANT412 | Ingest streaming data into Apache Iceberg tables with AWS streaming

1:00 PM – 2:00 PM PST | Caesars Forum

ANT413-R | Data governance with AWS analytics

1:00 PM – 2:00 PM PST | Caesars Forum

ANT323-R1 | Catalog and govern your data for generative AI

2:00 PM – 3:00 PM PST | MGM Grand

ANT338 | Using natural language to author data integration applications

4:00 PM – 5:00 PM PST | Caesars Forum

ANT317-R | Best practices for migrating to Amazon OpenSearch Service

1:30 PM – 2:30 PM PST | Wynn

ANT323-R | Catalog and govern your data for generative AI

1:00 PM – 2:00 PM PST | Mandalay Bay

ANT414-R1 | Scalable design patterns for Apache Iceberg–based data lakes

3:30 PM – 4:30 PM PST | MGM Grand

ANT413-R1 | Data governance with AWS analytics

5:30 PM – 6:30 PM PST | Caesars Forum

ANT304-R | Accelerating the shift from batch to stream processing

2:30 PM – 3:30 PM PST | Wynn

ANT314 | Add search to your existing databases with Amazon OpenSearch Ingestion

2:30 PM – 3:30 PM PST | Caesars Forum

ANT333 | Streamline data access management with trusted identity propagation

2:30 PM – 3:30 PM PST | Caesars Forum

ANT331-R | Build your data strategy for generative AI with Amazon Redshift

4:00 PM – 5:00 PM PST | Caesars Forum

ANT321 | Model your business structure with Amazon DataZone

3:00 PM – 4:00 PM PST | MGM Grand

ANT319 | Create a data marketplace with Amazon DataZone

4:00 PM – 5:00 PM PST | MGM Grand

ANT322 | Modernize and simplify ETL with AWS Glue

4:00 PM – 5:00 PM PST | Caesars Forum

ANT414-R | Scalable design patterns for Apache Iceberg–based data lakes

4:30 PM – 5:30 PM PST | Caesars Forum

ANT337-R1 | Cost optimization for data analytics on AWS

4:30 PM – 5:30 PM PST | MGM Grand

ANT409-R | Optimize Apache Spark workloads with Amazon EMR Serverless

5:30 PM – 6:30 PM PST | MGM Grand

ANT332-R1 | Democratize generative AI data access without compromising on security

5:30 PM – 6:30 PM PST | Caesars Forum

ANT316-R1 | Architectural patterns for near real-time data analytics on AWS

Builders’ sessions

Immerse yourself in our builders’ sessions – a hands-on learning experience designed to elevate your AWS skills. These focused, hour-long workshops bring together a small group of up to ten attendees with a dedicated AWS expert at each table.

Monday, Dec 2	Tuesday, Dec 3	Wednesday, Dec 4
8:30 AM – 9:30 AM PST \| Caesars Forum ANT306-R \| Orchestrate data and ML workflows with managed Apache Airflow	12:00 PM – 1:00 PM PST \| Caesars Forum ANT307-R \| Seamless data sharing with Amazon Redshift	8:30 AM – 9:30 AM PST \| Caesars Forum ANT307-R3 \| Seamless data sharing with Amazon Redshift
5:30 PM – 6:30 PM PST \| Caesars Forum ANT306-R1 \| Orchestrate data and ML workflows with managed Apache Airflow	12:00 PM – 1:00 PM PST \| Wynn ANT401 \| Vector search with Amazon OpenSearch Service	.
.	1:30 PM – 2:30 PM PST \| Wynn ANT306-R2 \| Orchestrate data and ML workflows with managed Apache Airflow	.
.	1:30 PM – 2:30 PM PST \| Caesars Forum ANT307-R1 \| Seamless data sharing with Amazon Redshift	.
.	3:00 PM – 4:00 PM PST \| Caesars Forum ANT307-R2 \| Seamless data sharing with Amazon Redshift	.
.	4:30 PM – 5:30 PM PST \| Caesars Forum ANT306-R3 \| Orchestrate data and ML workflows with managed Apache Airflow	.

Workshops

Roll your sleeves in our dynamic 2-hour workshops, where you’ll tackle real-world challenges using AWS services. These interactive sessions kick off with a brief, informative lecture to set the stage, then quickly transition into hands-on problem-solving. Bring your laptop and prepare to build alongside AWS experts, who will guide you through practical applications of cloud computing concepts. Whether you’re new to AWS or looking to sharpen your skills, these workshops offer a unique opportunity to learn by doing, enabling you to leave with confidence and applicable knowledge in AWS technologies.

Mon, Dec 2

Tuesday, Dec 3

Wednesday, Dec 4

Thursday, Dec 5

12:00 PM – 2:00 PM PST | Mandalay Bay

ANT404 | Migrating from self-managed Apache Kafka to Amazon MSK

11:30 AM – 1:30 PM PST | Venetian

ANT309 | Enhance insights for your data warehouse with zero-ETL & generative AI

9:00 AM – 11:00 AM PST | MGM Grand

ANT310 | Low-cost logging and observability with Amazon OpenSearch Service

12:00 PM – 2:00 PM PST | Mandalay Bay

ANT350-R1 | End-to-end data integration and data engineering on AWS

3:00 PM – 5:00 PM PST | Mandalay Bay

ANT312 | Unlock your enterprise data with intelligent document search

11:30 AM – 1:30 PM PST | MGM Grand

BSI204-R | Hands-on with Amazon Q in QuickSight: A step-by-step workshop

1:00 PM – 3:00 PM PST | MGM Grand

ANT350-R | End-to-end data integration and data engineering on AWS

3:00 PM – 5:00 PM PST | Mandalay Bay

ANT402-R1 | Build open table data lakes for real-time insights with Apache Iceberg

12:30 PM – 2:30 PM PST | Caesars Forum

ANT402-R | Build open table data lakes for real-time insights with Apache Iceberg

4:00 PM – 6:00 PM PST | MGM Grand

ANT308-R1 | Build and govern your data mesh with Amazon DataZone

3:30 PM – 5:30 PM PST | Venetian

BSI204-R1 | Hands-on with Amazon Q in QuickSight: A step-by-step workshop

3:30 PM – 5:30 PM PST | Wynn

ANT308-R | Build and govern your data mesh with Amazon DataZone

4:30 PM – 6:30 PM PST | MGM Grand

ANT311 | Prepare your data for generative AI

Code talks

Dive into the world of practical AWS development with our engaging Code Talks. These sessions elevate the popular chalk talk format by shifting focus from architectural concepts to hands-on coding. Watch as expert speakers guide you through live coding demonstrations, showcasing real-world solutions in action. You’ll gain insights into the reasoning behind each implementation choice and witness the development process unfold in real-time. Whether you’re a seasoned developer or just starting your AWS journey, Code Talks offer a unique opportunity to enhance your skills and deepen your understanding of AWS solutions through practical, code-centric discussions.

Mon, Dec 2

Tuesday, Dec 3

Wednesday, Dec 4

Thursday, Dec 5

1:00 PM – 2:00 PM PST | Wynn

ANT407 | Predictive maintenance with Amazon Managed Service for Apache Flink

3:00 PM – 4:00 PM PST | Wynn

ANT406-R | Generative AI–powered search with Amazon OpenSearch Service

12:00 PM – 1:00 PM PST | Wynn

ANT408 | Working with UDFs in Amazon Redshift and Amazon Athena

2:30 PM – 3:30 PM PST | Wynn

ANT406-R1 | Generative AI–powered search with Amazon OpenSearch Service

Session IDs for chalk talks, builders’ sessions, workshops, and code talks that end with R (for example, ANT406-R), indicate repeat sessions.

Conclusion

We hope this post acts as your go-to resource for navigating the AWS analytics track at re:Invent 2024. For staying in the know about the most recent trends and advancements in AWS Analytics, follow our LinkedIn page.

About the Authors

Imtiaz (Taz) Sayed is the WW Tech Leader for Analytics at AWS. He enjoys engaging with the community on all things data and analytics. He can be reached through LinkedIn.

Navnit Shukla serves as an AWS Specialist Solutions Architect with a focus on Analytics. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices. Notably, Navnit Shukla is the accomplished author of the book titled Data Wrangling on AWS. He can be reached through LinkedIn.

Maximize your cloud security experience at AWS re:Invent 2024: A comprehensive guide to security sessions

2024-11-11 Apurva More

Post Syndicated from Apurva More original https://aws.amazon.com/blogs/security/maximize-your-cloud-security-experience-at-aws-reinvent-2024-a-comprehensive-guide-to-security-sessions/

AWS re:Invent 2024, which takes place December 2–6 in Las Vegas, will be packed with invaluable sessions for security professionals, cloud architects, and compliance leaders who are eager to learn about the latest security innovations. This year’s event puts best practices for zero trust, generative AI–driven security, identity and access management (IAM), DevSecOps, network and infrastructure security, data protection, and threat detection and incident response at the forefront. The event will provide invaluable learning and networking opportunities for professionals focused on cloud security.

To help you navigate the extensive list of sessions and maximize your learning, we’ve curated a list of must-attend security sessions at re:Invent 2024. To join us, register today, and we’ll see you in Vegas!

Keynotes and innovation talks

The re:Invent 2024 keynote and innovation talks offer the opportunity to gain direct, transformative insights from senior AWS leaders. Delve into the latest breakthroughs in generative AI, cloud security, and cutting-edge architectural innovations that are redefining the future of application development and the AWS Cloud.

KEY002 – CEO Keynote with Matt Garman. Discover how AWS is innovating across the cloud, from reinventing core services to creating new experiences, empowering customers and partners to build a secure and better future.
SEC203-INT – Security insights and innovation from AWS with Chris Betz. Discover how groundbreaking security innovations and generative AI empower your organization to accelerate innovation securely, as AWS CISO Chris Betz reveals transformative strategies to integrate and automate security, freeing your team to focus on high-value initiatives.

Check out the full list of innovation talks. Not attending live this year? The keynote and innovation talks will be live streamed.

Sessions

To add sessions to your re:Invent 2024 agenda and find time and location information, choose the session title link.

Accelerating least privilege with advanced access analysis

Explore identity management and access control best practices to minimize your attack surface and enable a zero-trust architecture.

SEC325 | Chalk talk | A least privilege journey made easier by IAM Access Analyzer: Learn how centralized security teams and IAM policy authors are using IAM Access Analyzer to gain visibility into unused, overly permissive access and use actionable recommendations to achieve least privilege at scale.
SEC337 | Chalk talk | Scaling IAM: advanced administration and delegation patterns: Discover innovative strategies for effective access management, balancing security and agility as your organization expands. Learn from real-world scenarios, best practices, and cutting-edge techniques to optimize your IAM infrastructure for scalability, performance, and future growth.
SEC202 | Builders’ session | API Authorization with Amazon Cognito and Verified Permissions: In this session, gain hands-on experience with modern authorization in a microservices-based architecture on AWS and learn how to externalize and customize authentication with Amazon Cognito, apply fine-grained authorization with policy-based access controls using Amazon Verified Permissions, and integrate with APIs that are protected by Amazon API Gateway. You must bring your laptop to participate.
SEC334 | Chalk talk | Building zero trust architectures with AWS practical guidance: This chalk talk delves into building zero-trust network architectures using AWS services. Learn how to secure user-to-application, application-to-application, and other access scenarios through a zero-trust lens.
SEC232 | Breakout session | Secure by design: Enhancing the posture of root with central control: This session explores how to manage root access securely across your AWS environment, while maintaining centralized control and governance. Additionally, discover the latest tools and initiatives AWS offers to enforce multi-factor authentication (MFA), align with industry initiatives, and help your environment to remain secure.

Fortifying your security posture with threat detection and incident response

Use AWS security services to help you enhance your security posture and streamline security operations by continuously identifying and prioritizing security risks.

SEC321 | Breakout session | Innovations in AWS detection and response: This session focuses on practical use cases, such as threat detection, workload and data protection, automated and continual vulnerability management, centralized monitoring, continuous cloud security posture management, unified security data management, investigation and response, and generative AI. Gain a deeper understanding of how you can seamlessly integrate AWS detection and response services to help protect your workloads at scale, enhance your security posture, and streamline security operations across your entire AWS environment.
SEC332 | Chalk talk | Anatomy of a ransomware event targeting data within AWS: In this chalk talk, learn the anatomy of a ransomware event that targets data within AWS, including detection, response, and recovery. Leave with a deeper understanding of the AWS services and features you can use to protect against ransomware events in your environment and the knowledge to investigate possible ransomware events if they occur.
SEC301 | Workshop | Threat detection and response using AWS security services: This workshop simulates several security events across different resources and behaviors. Get hands-on in a provided sandbox environment to review and respond to findings from the simulated events. You must bring your laptop to participate.
SEC219 | Breakout session |Uncovering sophisticated cloud threats with Amazon GuardDuty: Learn how Amazon GuardDuty offers fully managed threat detection that gives you end-to-end visibility across your AWS environment. The unique detection capabilities of GuardDuty are guided by AWS visibility into the cloud threat landscape and can help responders address issues faster, minimizing the mean time to repair (MTTR) and optimizing security resources—so your teams can spend more time innovating and less time chasing down security risks.
SEC343 | Chalk talk | Identify a prioritization strategy for security response & remediation: Join this chalk talk to learn about a framework for automating your response and remediation to security findings for your accounts. With AWS Security Hub as the foundation, explore the decision-making process regarding which findings could be auto-remediated, the implications of an auto-remediation approach, and how to achieve a quick and thorough response.
SEC401 | Code talk| Inspect and secure your application with generative AI: Explore how to use generative AI to improve the security of your applications. Learn how AI-powered tools can help rapidly identify and then recommend remediations for security issues. Learn about how Amazon Inspector detects software and code vulnerabilities in your applications, and discover how to scan for issues and remediate them using generative AI in your integrated development environment (IDE).

Securing the edge against evolving risks with confidence

Use AWS edge security services to help protect against distributed denial of service (DDoS) and exploits directed against applications and achieve a more consistent security posture.

SEC322 | Breakout session | Reduce your risk exposure with least privilege egress controls: Join this session to learn how to align your egress control strategy with the principle of least privilege. Learn how the recent launches for AWS Network Firewall, Amazon Route 53 Resolver DNS Firewall, and other security services can help you to reduce your exposure to a variety of risks. Simplify implementation and make rule recommendations that are specific to your use cases. Gain the confidence that your security policies meet their intended needs.
SEC344 | Chalk talk | Lessons learned from DDoS mitigation: Insights from AWS Shield Response Team (AWS SRT) escalations: In this chalk talk, dive into past DDoS events and find out how the AWS SRT helped to mitigate security escalations. Gain insights into this type of intrusion and how you can apply mitigation strategies to make your application more DDoS-resilient.
SEC327| Chalk talk | Building secure network designs for generative AI applications: In this chalk talk, learn how to build layered network security controls to protect, detect, and respond to issues faster and to accelerate your generative AI applications securely on AWS. Discover key considerations, best practices, and reference architectures to achieve your defense-in-depth network design objectives.
SEC304 | Workshop | Mitigate zero-day events and ransomware risks with VPC egress controls: In this network security workshop, learn how to implement AWS best practices for egress controls to mitigate risks from software supply chain dependencies, zero-day events, cryptocurrency mining, and ransomware. You must bring your laptop to participate.
SEC317 | Breakout session | How Amazon threat intelligence helps protect your infrastructure: Explore AWS threat intelligence capabilities and learn how they power managed firewall rules and security findings in security services such as AWS WAF, AWS Network Firewall, and Amazon Route 53 Resolver DNS Firewall. Learn about the threat intelligence AWS uses to protect AWS infrastructure, build new security features, and empower customers to enhance their application protection on AWS.

Safeguarding sensitive data in the age of generative AI

Discover advanced techniques and AWS services to help you protect the confidentiality and privacy of your data when you implement emerging AI technologies.

SEC323 | Breakout session | The AWS approach to secure generative AI: Join this session to learn how AWS thinks about security across the three layers of our generative AI stack, from the bottom infrastructure layer to the middle layer (which provides access to the models and tools customers need to build and scale generative AI applications) to the top layer (which includes applications that make use of large language models (LLMs) and other foundation models (FMs) to make work easier).
SEC310 | Workshop | Persona-based access to data for generative AI applications: In this workshop, manage document access in a chatbot application tailored to various user roles within an organization. Learn how to address challenges around secure information distribution, enhancing efficiency and compliance by aligning access rights with job functions. You must bring your laptop to participate.
SEC336 | Chalk talk |Security and compliance considerations using Amazon Q Business: This chalk talk covers best practices for securing your Amazon Q Business application, including access control, data protection, and compliance considerations.

To find more generative AI–focused sessions, see this blog post.

Empowering developers with a security-minded culture

Integrate security seamlessly within your DevSecOps practices to accelerate time to market and reduce risk.

SEC216 | Breakout session | Build trust in your CI/CD pipeline: Codify container security at scale: In this session, learn how to automate container security and compliance at scale. Explore how Amazon Q Developer, Amazon Inspector, and Amazon Elastic Compute Cloud (Amazon EC2) Image Builder complement each other, helping to automate the creation of secured container images and their eventual storage in Amazon Elastic Container Registry (Amazon ECR). Leave knowing how to support your builders and allow them to develop quickly without compromising on security.
SEC217 | Breakout session | Building a resilient and effective culture of security: This talk offers guidance on cultivating a resilient, empowered culture of security, including gaining leadership support, distributing security ownership, and embedding psychological safety to build trust, transparency, and a proactive security-first mindset.
SEC218 | Breakout session | Emotionally intelligent security leadership to drive business impact: Elevate your leadership and learn to align security needs with strategic business outcomes, spearhead impactful transformations, and cultivate a sustainable security culture. Get an inside look how AWS and its customers lead security with empathy, translate security purpose into results, inspire innovation, and foster connections to improve positive escalation culture. Become empowered to lead with precision, acquire the art of connecting security objectives to meaningful business impact, and steer your organization toward a future where security is a catalyst for success and resilience.
SEC314 | Code talk | Accelerate your DevOps pipeline and remain secure with policy as code: In this code talk, learn how to define compliance rules for your AWS infrastructure and evaluate them using AWS CloudFormation Guard, which is an open source, general-purpose, policy-as-code evaluation tool. Explore how to seamlessly integrate automated policy validation into your existing deployment pipeline, empowering DevOps engineers to build policy assessment steps into their CI/CD pipelines. Security assessors will experience streamlined review processes while maintaining a robust security posture.
SEC302 | | Breakout session | Better together: Protecting data through culture and technology: This session examines the full range of data protection capabilities that are available with AWS and how best practices and culture can complement these capabilities to improve security outcomes. Learn more about the defense-in-depth perspective, which details how organizations can protect their data and bolster their security culture by consistently building security into every layer.

Expo

Want to talk directly with an AWS expert on cloud security? Then don’t miss this opportunity to have one-on-one conversations with leading AWS security experts in the Security Activation area of the expo floor to help you take your organization’s security posture to new heights.

Delve into key security domains such as:

Detection and response: Explore techniques for detecting and responding to security risks to help protect your workloads at scale.
Network and infrastructure security: Learn how to build and manage a secure global network with AWS services.
Application security: Discover strategies to ship secure software and address the challenges of application security.
Identity and access management: Adopt modern cloud-native identity solutions and apply least-privilege access controls.
Digital sovereignty and data protection: Maintain control over your data and choose how to secure and manage it in the AWS Cloud.

Still time for fun!

After an inspiring week of transformative insights and deep learning, join us for the world renowned re:Play party—the ultimate re:Invent sendoff! Immerse yourself in live entertainment from headlining musical artists, scrumptious cuisine, and flowing refreshments as we come together to unwind, connect, and toast to a future of limitless possibilities.

Register today

It’s going to be an amazing event, and we can’t wait to see you at re:Invent 2024! Register now to secure your spot.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

The attendee’s guide to the AWS re:Invent 2024 Compute track

2024-11-05 aostan

Post Syndicated from aostan original https://aws.amazon.com/blogs/compute/the-attendees-guide-to-the-aws-reinvent-2024-compute-track/

From December 2^nd to December 6^th, AWS will hold its annual premier learning event: re:Invent. At this event, attendees can become stronger and more proficient in any area of AWS technology through a variety of experiences: large keynotes given by AWS leaders, smaller innovation talks and interactive working sessions given by AWS experts, and fun activities such as live music and games at re:Play.

There are over 2000+ learning sessions that focus on specific topics at various skill levels, and the compute team have created 72 unique sessions for you to choose. There are many sessions you can choose from, and we are here to help you choose the sessions that best fits your needs. Even if you are not able to join in person, you can catch-up with many of the sessions on-demand and even watch the keynote and innovation sessions live.

The Basic: Session types

If you’re able to join us, just a reminder that we offer several types of sessions which can help maximize your learning in a variety of AWS topics.

re:Invent attendees can also choose to attend chalk-talks, builder sessions, workshops, or code talk sessions. Each of these are live non-recorded interactive sessions.

Breakout sessions: Attendees will be in a lecture-style 60-minute informative sessions presented by AWS experts, customers, or partners. These sessions are recorded and uploaded a few days after to the AWS Events YouTube channel.
Chalk-talk sessions: Attendees will interact with presenters, asking questions and using a whiteboard in session.
Builder Sessions: Attendees participate in a one-hour session and build something.
Workshops sessions: Attendees join a two-hour interactive session where they work in a small team to solve a real problem using AWS services.
Code talk sessions: Attendees participate in engaging code-focused sessions where an expert leads a live coding session.
Lightning talk sessions: Attendees watch a 20-minute demo dedicated to either a specific service or customer story (located in the Expo Hall).

Getting started with Amazon EC2

The foundation of compute in AWS is Amazon Elastic Compute Cloud (Amazon EC2). Amazon EC2 offers the broadest and deepest compute platform, with over 800 instances and choice of the latest processor, storage, networking, operating system, and purchase model to help you best match the needs of your workload. We’ve created the following sessions to help you implement and manage your workloads in Amazon EC2.

CMP101 | What’s new with Amazon EC2
Learn about the latest compute innovations from AWS. This session helps you better understand Amazon EC2 instances and how organizations like yours can use them to run any workload while meeting your cost, performance, and sustainability goals.
CMP343 | Select and launch the right instance for your workload and budget
With more than 800 instances for various use cases, including instances best for common workloads and for workloads with specific requirements, how do you choose instances? Learn how to determine which instance is best for your specific use case and budget.
CMP319 | Managing Amazon EC2 capacity and availability
Amazon EC2 offers a variety of capacity usage and reservation models, so you can choose the right combination for your workload and budget. Learn how to combine these models in a way that’s best for your business and manage your capacity to improve utilization and availability.
CMP207 | AWS-accelerated computing enables customer success with generative AI
Discover how AWS provides the most performant, low-cost infrastructure for building and scaling large-scale generative AI models. Come learn what’s new in the accelerated computing portfolio including our GPU-based and AWS AI chips-powered instances.
CMP318 | Choose the optimal compute environment for your AI/ML workloads
If you’re trying to decide between accelerators such as AWS Inferentia and AWS Trainium, GPUs from NVIDIA and AMD, processors such as AWS Graviton, or managed services such as Amazon Bedrock and Amazon SageMaker, this chalk covers the different options available on AWS.

Learn about AWS compute innovations

AWS has invested years designing custom silicon optimized for the cloud to deliver the best price performance for a wide range of applications and workloads using AWS services. Learn more about the AWS Nitro System, processors at AWS, and ML chips.

CMP301 | Dive deep into the AWS Nitro System

The AWS Nitro System is a rich collection of building block technologies that are powering the recent and future generations of Amazon EC2 instances. Dive deep into the Nitro System and see how it made the seemingly impossible possible.

CMP320 | AWS Graviton: The best price performance for your AWS workloads
AWS Graviton-based Amazon EC2 instances provide the best price performance for workloads in Amazon EC2. Learn about common use cases, best practices to optimize your workloads across various applications, customer success stories, and how to accelerate your Graviton journey.
CMP209 | Conquer AI performance, cost, and scale with AWS AI chips

Generative AI promises to revolutionize industries, but its immense computational demands and escalating costs pose significant challenges. To overcome these hurdles, AWS designed and purpose-built AI chips including AWS Trainium2 and AWS Inferentia2.

CMP334 | Deep dive into third generation AWS Nitro SSDs
Learn about AWS Nitro SSDs. Discover how AWS Nitro SSDs are different than other commercially available SSDs and see how AWS Nitro SSDs can deliver performance to benefit your workloads.

Optimize your compute costs

At AWS, we focus on delivering the best possible cost structure for our customers. Frugality is one of our founding leadership principles. Cost effective design continues to shape everything we do, from how we develop products to how we run our operations. Come learn of new ways to optimize your compute costs through AWS services, tools, and optimization strategies in the following sessions:

CMP214 | Win-win: Maximize Amazon EC2 savings while improving performance
Let this session be your guide to building cost-effective, sustainable infrastructure without sacrificing application performance on AWS. Learn both technical and non-technical best practices for building efficient compute architectures on AWS.
CMP408 | Amazon EC2 flex instances: Deliver performance at lower cost
Amazon EC2 flex instances provide the easiest way to save costs and achieve better price performance for a majority of your workloads. In this session, dive deep into flex instances, explore how they deliver performance at lower cost, and identify the suitable workloads.
CMP312 | Spot the savings: Optimize deployments with Amazon EC2 Spot Instances
Amazon EC2 Spot Instances use spare Amazon EC2 capacity available to you at steep discounts compared to on-demand prices. In this workshop, learn about the solutions, tools, and best practices to help you maximize your savings with Spot instances.
CMP346 | Uncover compute efficiency with AWS Graviton Savings Dashboard
The Graviton Savings Dashboard offers a comprehensive analysis of your compute usage, identifying prime candidates for Graviton migration. Learn how to implement and use the Graviton Savings Dashboard to quantify the potential TCO reduction from Graviton adoption.
CMP311 | Proactively scale for optimal cost and availability in Amazon EC2
Amazon EC2 Auto Scaling groups help you take advantage of the elasticity benefits that are built in to AWS. With more responsive and proactive scaling, you run only the required number of instances at any time of the day, reducing the cost of overprovisioned EC2 instancess

Maximize you workload’s performance

Your workload’s performance matters beyond just cost because it directly impacts the quality, efficiency, and effectiveness of your compute solution. It can significantly influence customer satisfaction, business growth, and overall productivity. Even if a cheaper option exists, a low-cost option with poor performance can lead to long-term financial losses due to issues such as lost customers, engineering rework, and negative reputation. We have a number of sessions that help you optimize your workload’s performance.

CMP411 | Everything you’ve wanted to know about performance on EC2 instances
This session covers all the details you’ve always wanted to know to optimize your compute performance such as memory topology, accessing hardware counters, accounting for the side-effects of hyperthreading, properly running performance tests, and optimizing your latency.
CMP413 | Moving from naive benchmarking to application performance engineering
Most of the time, benchmarks aren’t representative of their applications’ behaviors. In this session, learn the tools and best practices that will help you understand your applications’ performance behaviors on Amazon EC2 instances so that you can maximize your performance.
CMP405 | How to optimize latency and throughput
The availability of processors with and without hyperthreading makes performance evaluation a tricky game. In this code talk, study a web application and evaluate its performance in various scenarios, and discover how to optimize throughput and latency along the way.

Customer experiences and applications with machine learning

Machine learning (ML) has been evolving for decades and has an inflection point with generative AI applications capturing widespread attention and imagination. More customers, across a diverse set of industries, choose AWS compared to any other major cloud provider to build, train, and deploy their ML applications. Learn about generative AI infrastructure at Amazon or get hands-on experience building ML applications through our ML focused sessions, such as the following:

CMP208 | Customer stories: Optimizing AI performance and cost with AWS AI chips
AWS Trainium and AWS Inferentia deliver high-performance AI training and inference while reducing costs by up to 50%. Attend this session to hear from four AWS customers and how they realized these benefits to grow their businesses while delivering innovative experiences.
CMP321 | Explore the many ways to train foundation models on AWS
This session unravels the complexities of building and scaling large scale foundation models. From selecting the optimal compute resources to optimizing data pipelines and maximizing network performance.
CMP331 | Build and accelerate LLMs on AWS Trainium and AWS Inferentia using Ray
Learn how to accelerate the development and deployment of large language models (LLMs) with Ray, AWS Trainium, and AWS Inferentia. This session delves into how Ray’s unified compute framework integrates with powerful AWS AI chips to optimize performance and cost efficiency.
CMP304 | Fine-tune Hugging Face LLMs using Amazon SageMaker and AWS Trainium
You can improve the performance of a pretrained LLM by fine-tuning the model using a smaller task-specific or domain-specific dataset. In this builders’ session, learn how to use Amazon SageMaker to fine-tune a pretrained Hugging Face LLM using AWS Trainium for inference use.
CMP337 | Fine-tune and deploy Llama 3.1 models on AWS Trainium and Inferentia
This session provides an overview of Neuron SDK and the various capabilities that maximize performance and deliver ease-of-use when training and deploying Llama 3.1 models on AWS AI chips.
CMP314 | Keeping it small: Agentic workflows with SLMs on AWS Inferentia
While LLMs offer versatility, smaller language models (SLMs) provide resource efficiency, speed, and simplicity. This session explores task simplification and decomposition techniques to harness multiple specialized SLMs, surpassing a single large model’s accuracy at fraction of cost.
CMP329 | Beyond text: Unlock multi-modal AI with AWS AI chips
Revolutionize your applications with multi-modal AI. Learn how to harness the power of AWS AI chips to create intelligent systems that understand and process text, images, and video.
CMP323 | Optimize your AI/ML workloads with Amazon EC2 Graviton
Join this session to explore performance, cost, and sustainability optimizations of your AI/ML solutions with services powered by AWS Graviton, accelerated computing instances, and Amazon EC2 Spot Instances.
CMP407 | Optimized RAG pipelines using AWS Graviton: VectorDB and LLM endpoints
Learn from AWS experts about efficient RAG deployment options for use cases that optimize cloud resource usage and costs. Learn what’s possible when using AWS Graviton to power your RAG-optimized generative AI inference workloads.

Accelerate your AWS Graviton adoption journey

The AWS Graviton Processors are custom designed server processors designed by AWS. They deliver the best price performance for your cloud workloads running in AWS, and help you reduce your carbon footprint. Ready to realize up to 40% better price performance for your workloads? We have curated the following session to help you accelerate your Graviton adoption:

CMP305 | Learnings from developers adopting AWS Graviton at scale
In this chalk talk, engage directly with AWS specialists that help customers on a daily basis with their adoption journey—from workload selection to running at scale in production. Explore AWS Graviton use cases, best practices, performance, and customer success stories.
CMP310 | Migrating applications to AWS Graviton on Amazon EKS
During this hands-on workshop, walk through the steps for migrating a workload running on x86 to AWS Graviton-based instances including performing tests locally and modifying the CI/CD pipeline to build and deploy the application in Amazon EKS using Karpenter.
CMP316 | AWS Graviton GameDay: Optimize your Amazon EC2 workload with Graviton
Ready to learn more about AWS Graviton in an immersive environment? In this team-based gamified learning setting, perform a live migration of your workload to Graviton. You learn how to unlock Graviton’s full price-performance potential and optimize the size of an Amazon EC2 fleet.
CMP404 | Exploring performance analysis with AWS Graviton instances
In this session, AWS experts open a shell on an Amazon EC2 instance and dig into the system to see which tools and resources you can use, including the Amazon Aperf tool. Learn as they write some mini-applications to study their performance behavior and how to improve them.

Check out workload-specific sessions

Amazon EC2 offers the broadest and deepest compute platform to help you best match the needs of your workload. More SAP, high performance computing (HPC), ML, and Windows workloads run on AWS than any other cloud. Join sessions focused around your specific workload to learn about how you can leverage AWS solutions to accelerate your innovations.

CMP205 | Launch a secure WordPress site on Amazon Lightsail in minutes
Join this session to learn how to set up a secure and highly available WordPress website on Amazon Lightsail – an easy-to-use virtual private server (VPS). Discover what Amazon Lightsail is, the resources available to create a website, and how to set one up in minutes, all at a predictable monthly cost.
CMP341 | Migrate and modernize your web applications with AWS Elastic Beanstalk
Moving classic web applications to the cloud can be a complex task for customers. In this chalk talk, operators and developers can learn the benefits of using AWS Elastic Beanstalk to upload and deploy web applications in a simplified, fast way and integrate with your existing CI/CD.
CMP213 – Run workloads efficiently on EKS with Karpenter and EC2 Spot Instances
This session covers how Karpenter can help you reduce complexities and improve efficiency in Kubernetes clusters. Explore how to leverage Amazon EC2 Spot Instances as a purchase option, and learn how AWS Graviton-based Amazon EC2 instances help further optimize your workloads while improving sustainability.
CMP322 | Amazon EC2 High Memory portfolio for SAP HANA
This session showcases how customers leverage the agility, flexibility and resiliency that AWS High Memory instances provide to help them deploy and scale the infrastructure for SAP HANA deployments while meeting performance and high availability goals.
CMP324 | Protect sensitive data in use with AWS Confidential compute
Confidential computing enables customers to protect code and data from unauthorized access during processing. This session dives into how AWS delivers a combination of hardware- and software-based solutions to deliver confidential computing capabilities.
CMP203 | Drive innovation and results with high performance computing on AWS
In this session, explore how customers across healthcare and life sciences (HCLS) and manufacturing industries are harnessing the convergence of HPC, cloud, and AI on AWS to accelerate time to insights, optimize performance, and drive innovation.
CMP210 | Modernize Apple platform development with AWS and EC2 Mac
Learn about Apple application development in the cloud using EC2 Mac instances and hear firsthand how an AWS customer optimized its Apple development workflow and benefited from Apple application development in the cloud.
CMP302 | Run containerized workloads efficiently on AWS
Containers offer scalability and flexibility, enabling seamless deployment, management, and scaling of applications in any environment. Using containers on AWS can help you improve your efficiency and achieve your price performance goals.
CMP326 | Accelerate AI innovation for health care and life sciences on AWS
Biopharma researchers are looking to build and deploy models such as AlphaFold2, ProtGPT2, and ESM-2 for generative biology and chemistry. In this chalk talk we cover how to deploy NVIDIA BioNeMo on NVIDIA GPU-powered Amazon EC2 instances, AWS ParallelCluster, and Amazon SageMaker.
CMP342 | Scaling 3D content creation with open source technologies
This chalk talk explores 3D Gaussian Splatting as an emergent 3D reconstruction technique and how AWS can accelerate and scale the generation, management, and consumption of digitalized real-world assets in enterprise contexts, from virtual production to immersive commerce.
CMP315 | Creating immersive 3D digital twins from photos, videos, and LiDAR
Join spatial computing specialists as they show you how to build a digital twin in this interactive workshop using NVIDIA Omniverse.

Ready to unlock new possibilities?

The AWS Compute team looks forward to seeing you in Las Vegas. Come meet us at the Compute Booth in the Expo and check out our various Amazon EC2 demos. And if you’re looking for more session recommendations, check-out additional re:Invent attendee guides curated by experts.

Exploring digital sovereignty: learning opportunities at re:Invent 2024

2024-10-24 Marta Taggart

Post Syndicated from Marta Taggart original https://aws.amazon.com/blogs/security/exploring-digital-sovereignty-learning-opportunities-at-reinvent-2024/

AWS re:Invent 2024, a learning conference hosted by Amazon Web Services (AWS) for the global cloud computing community, will take place December 2–6, 2024, in Las Vegas, Nevada, across multiple venues. At re:Invent, you can join cloud enthusiasts from around the world to hear the latest cloud industry innovations, meet with AWS experts, and build connections. Whether you want to build deep technical expertise, understand how to prioritize your investments, learn more about the infrastructure offerings of the sovereign-by-design AWS Cloud, or see how the AWS Nitro System enables enhanced security for your workloads, re:Invent is a great opportunity to explore our digital sovereignty solutions.

This year, there will be many ways that you can learn about our advanced sovereignty controls, security features, and infrastructure options that can help meet your unique digital sovereignty needs, including sessions and hands-on activities with AWS hybrid and edge services including AWS Local Zones, AWS Dedicated Local Zones, and AWS Outposts. In the Expo, you can visit the Digital Sovereignty & Data Protection kiosk in the AWS Village to watch demos, learn about the upcoming AWS European Sovereign Cloud, and get your questions answered by AWS team members. To see AWS designed chips and Outposts devices, check out the AWS Next Gen Infrastructure Hub in the AWS Village. You can also visit the AWS Partner Network (APN) booth to connect with AWS Digital Sovereignty Partners to learn about the benefits of partner programs.

Breakout sessions and lightning talks

To add sessions to your AWS re:Invent agenda and find time and location information, choose the session title link.

SEC229 | Breakout | Digital sovereignty: overcome complexity and enable future-readiness
Max Peterson, VP, Sovereign Cloud, AWS
Organizations are facing increasing complexity in an evolving sovereignty landscape. Building a strong digital foundation can help simplify efforts to meet requirements today and prepare your organization for the future, without slowing innovation. Join this session to learn how AWS sovereign cloud offerings, ranging from encryption services to the announced AWS European Sovereign Cloud, provides more control and choice to help meet your unique needs. Discover how customers are keeping critical workloads secure and protected when leveraging new technologies on AWS, including generative AI, and learn about new digital sovereignty solutions offered by AWS Partners.

HYB201 | Breakout | AWS wherever you need it: From the cloud to the edge
Jan Hofmeyr, VP, EC2 Networking and Hybrid Edge, AWS, and Jeff Feist, Executive Director – Hosting Solutions, Merck & Co., Inc.
While most workloads can be migrated to the cloud, some remain on premises or at the edge due to low latency, local data processing, or digital sovereignty needs. In this session, learn how AWS services like AWS Outposts, AWS Local Zones, AWS Dedicated Local Zones, and AWS IoT Core support hybrid cloud and edge computing workloads such as multiplayer gaming, high-frequency trading, medical imaging, smart manufacturing, and generative AI applications with data residency requirements.

HYB309 | Breakout | Well-architected for data residency with hybrid cloud services
Sherry Lin, Principal Product Manager, AWS; Lakshmi VP, Specialist SA – Hybrid Edge, AWS; and Kevin Ng, Senior Director, Core Engineering Products, GovTech
With concerns over data privacy, security, and digital sovereignty, many countries across the world are strengthening data residency laws to keep personal and sensitive data within their borders. For organizations operating across multiple geographies, it can be challenging to meet the evolving data residency laws. In this session, following the AWS Well-Architected Framework, explore the best practices around data residency when using hybrid cloud services, including AWS Local Zones, AWS Dedicated Local Zones, and AWS Outposts.

IOT202 | Breakout | AWS IoT for edge LLM deployment and execution
Nikit Pednekar, Principal Product Manager, AWS, and Stefano Marzani, WW Tech Leader, SDX, AWS
With the advent of generative AI and large language models (LLMs), you must be wondering, how can these technologies be applied at the IoT edge? After all, there are many benefits of running LLMs at the edge—from network bandwidth efficiencies, offline processing, lower latency, and data sovereignty to cost savings, security, and differentiation. In this session, learn how using AWS IoT services and LLMs at the edge can uplift your solutions with actionable outcomes and innovative capabilities, such as gesture recognition, natural language processing for voice control, real-time predictive maintenance, energy optimization, anomaly detection, and more.

KUB310 | Breakout | Amazon EKS for edge and hybrid use cases
Chris Splinter, Product Manager, AWS, and Gokul Chandra Purnachandra Reddy, Senior Solutions Architect, AWS
There are some workloads that may need to run on-premises, at the edge, or in a hybrid scenario due to low-latency, data dependencies, data sovereignty, or other regulatory reasons, especially in industries such as manufacturing, healthcare, telco, and financial services. Data dependent workloads may have to wait for the data to be on AWS services before fully migrating. In this session, we will share production-ready architectures leveraging services like Amazon EKS Anywhere to run container workloads on-premises and support modernizing VMware-based workloads. Also learn best practices on migration of on-premises Kubernetes deployments to AWS Cloud.

PEX110 | Lightening Talk | Supercharge your growth and capabilities with partner programs
Mike Cannady, Director, Partner Core Public Sector, AWS
Discover the latest AWS Partner program updates that propel your public sector business forward. Join this lightning talk to explore innovations tailored to partners: generative AI programs, digital sovereignty, solution building, and managed services. Whether you’re starting out or seasoned, glean insights and use cases to elevate your journey. Don’t miss this opportunity to supercharge your development and stay ahead in this ever-evolving landscape.

Interactive sessions (chalk talks and workshops)

HYB304 | Workshop | Implement RAG without compromising on digital sovereignty
Aditya Lolla, Senior Solutions Architect, Hybrid Edge, AWS, and Robert Belson, Senior Developer Advocate, AWS
As governments and standards bodies develop data protection and privacy regulations, organizations increasingly need to combine the use of generative AI tooling in the cloud with regulated data that need to remain on premises to meet data sovereignty requirements. In this workshop, learn how to extend Agents for Amazon Bedrock to hybrid and edge services like AWS Outposts and AWS Local Zones to build distributed Retrieval Augmented Generation (RAG) applications with on-premises data for improved model outcomes. Get hands-on with Amazon Bedrock, AWS Lambda, and AWS hybrid and edge services, and build Amazon Simple Storage Service (Amazon S3) compliant workflows using a hybrid S3 compatible solution. You must bring your laptop to participate.

WPS207 | Chalk Talk | How AWS can help you meet your digital sovereignty requirements
Mehmet Bakkaloglu, Principal Solutions Architect, AWS, and Addy Upreti, Principal Technical Product Manager – Digital Sovereignty, AWS
Customers in the public sector and regulated industries such as healthcare, financial services and telecom have shared how they face digital sovereignty concerns in their cloud journey. In this talk, you can learn about how AWS is sovereign-by-design and the range of capabilities that can enable you to meet your digital sovereignty needs. Plus, discover how the AWS European Sovereign Cloud is being built to provide further choice to meet these needs. We’ll talk through how AWS can help accelerate your cloud journey while meeting your requirements.

HYB310 | Chalk Talk | Addressing data residency requirements with hybrid and edge services
Sedji Gaouaou, Senior Solutions Architect, AWS, and Fabio Rodriguez, Head of Hybrid Cloud Solutions Architect, AWS
Data residency is a critical consideration for organizations that collect and store sensitive information, including personal identifiable information (PII), financial data, healthcare data, or information pertaining to national security. To help organizations operating across multiple geographies drive innovation while meeting data residency requirements, AWS offers multiple global infrastructure offerings like AWS Regions, AWS Dedicated Local Zones, AWS Local Zones, and AWS Outposts. In this interactive chalk talk, learn how these infrastructure offerings can help you accelerate digital transformation while meeting data residency needs.

For a full view of digital sovereignty content, including sessions with partners, explore the AWS re:Invent catalog and filter on the Digital Sovereignty area of interest. Not able to attend in-person? Register for free for the virtual-only pass to livestream keynotes and innovation talks, and access on-demand breakout sessions today. See you in Las Vegas or on the livestream!

If you have feedback about this post, submit comments in the Comments section below.

Strengthening security in the era of generative AI: Must-attend sessions at re:Invent 2024

2024-10-10 Anna Montalat

Post Syndicated from Anna Montalat original https://aws.amazon.com/blogs/security/strengthening-security-in-the-era-of-generative-ai-must-attend-sessions-at-reinvent-2024/

Generative AI is transforming industries in new and exciting ways every single day. At Amazon Web Services (AWS), security is our top priority, and we see security as a foundational enabler for organizations looking to innovate. As you prepare for AWS re:Invent 2024, make sure that these essential sessions are on your schedule to learn how security can help your organization innovate with generative AI solutions quickly and securely. Leading experts will provide deep insights into how you can secure generative AI workloads in order to protect data and navigate governance, risk, and compliance.

In this post, we’ve highlighted some of our must-attend sessions and favorite activities for security leaders and practitioners, technical decision-makers, and artificial intelligence and machine learning (AI/ML) builders. To join in on the fun, register here, and we’ll see you in Vegas!

Keynotes and innovation talks

The AWS re:Invent 2024 keynote and innovation talks offer the opportunity to gain direct, transformative insights from senior AWS leaders. Delve into the latest breakthroughs in generative AI, cloud security, and cutting-edge architectural innovations that are redefining the future of application development and the AWS Cloud.

KEY002 – CEO Keynote with Matt Garman. Discover how AWS is innovating across the cloud, from reinventing core services to creating new experiences, empowering customers and partners to build a secure and better future.
SEC203-INT – Security insights and innovation from AWS with Chris Betz. Discover how groundbreaking security innovations and generative AI empower your organization to accelerate innovation securely, as AWS CISO Chris Betz reveals transformative strategies to integrate and automate security, freeing your team to focus on high-value initiatives.
ARC203-INT – Architectural methods & breakthroughs in innovative apps in the cloud with Shaown Nandi and Ben Cabanas. This talk showcases how generative AI and cutting-edge architectural advancements are transforming application design, empowering AWS customers to modernize their systems, develop robust data strategies, and securely navigate the evolving cloud landscape.
Check out the full list of innovation talks. Not attending live this year? The keynote and all innovation talks will be live streamed.

Sessions

Discover a range of learning opportunities designed to deepen your expertise in securing generative AI. This year’s sessions put a strong focus on providing customers with impactful real-world, practical prescriptions for securing your AI workloads and the data that powers them. Whether you prefer lecture-style breakout sessions, interactive chalk talks, hands-on workshops, or code-driven discussions, there’s a session tailored to help meet your needs. Explore the following options and reserve your spot to enhance your understanding and practical skills in this rapidly evolving field.

You can find more details and descriptions for session levels (100–400) and session types on the re:Invent website.

Breakout sessions

Breakout sessions are lecture-style, 1-hour sessions delivered by AWS experts, customers, and partners—perfect for deepening your knowledge on important topics, gaining actionable insights, and connecting with industry leaders.

SEC214 –Elevating client experiences with secure AI: Rocket Mortgage’s approach. Discover how Rocket Mortgage implemented AWS generative AI services to enhance customer experiences while navigating security challenges. Register for this session
SEC315 – Bring your workforce identities to AWS for generative AI and analytics. This session will demonstrate the power of integrating your workforce identity provider to gain easier access to generative AI applications and tools. AWS and NVIDIA will demonstrate a full end-to-end identity-aware experience. Register for this session
SEC323 –The AWS approach to secure generative AI. Learn how AWS secures generative AI across the infrastructure, model, and application layers, giving customers control over their data with built-in security features. Register for this session
SEC403 –Generative AI for security in the real world. Explore practical generative AI applications for security teams, including incident response, red team/blue team enablement, and security operations center (SOC) use cases, to boost your security operations. Register for this session

Chalk talks

Chalk talks are 1-hour long, highly interactive sessions with a small audience. This format is ideal for diving deep into specific topics, engaging directly with AWS experts, and getting your questions answered in real time.

SEC303 – Protecting data within your generative AI architectures. Mitigate risks when training large language models (LLMs) on sensitive data. Explore techniques like sanitization, anonymization, and differential privacy. Register for this talk
SEC327 – Building secure network designs for gen AI applications. Optimize your cloud network design to power transformative generative AI applications more securely, as we share proven best practices, proactive controls, and reference architectures to build resilient, defense-in-depth architectures and accelerate innovation on AWS. Register for this talk
SEC335 –Harness generative AI for business growth amidst the regulatory landscape. Explore how AWS AI/ML solutions can drive business growth while helping you align to responsible practices. Learn from your peers about their strategies to navigate evolving regulatory landscapes, from the European Union’s General Data Protection Regulation (GDPR) to industry-specific mandates. Register for this talk
SEC336 –Security and compliance considerations using Amazon Q Business. Discover best practices for securing your Amazon Q Business application, focusing on access control, data protection, and compliance considerations, so that you can keep your AI assistant secure and compliant. Register for this talk
SEC338 –Safeguard your generative AI apps from prompt injections. Learn to protect your generative AI applications from prompt injection attacks by understanding input validation, secure prompt engineering, and content moderation. Register for this talk
PEX308 – Securing generative AI on AWS. Explore generative AI security considerations through a partner lens, including how partners can enhance data security and the value-adds that partners bring to a customer’s generative AI workloads. Register for this talk
AIM344 – Understanding the deep security controls within Amazon Q Business. Learn about the security-related capabilities and controls within Amazon Q that allow you to confidently use your business data safely. Register for this talk
AIM407 – Understand the deep security controls within Amazon Bedrock. Dive deep into the security nuances of Amazon Bedrock, as we unpack the architectures, data flows, and lifecycle management of complex features like Guardrails, Agents, and Knowledge Bases, empowering you to use this generative AI service with uncompromising data privacy and control. Register for this talk
DEV323 – OWASP Top 10 for LLMs. Strengthen your skills in securing generative AI applications by exploring real-world vulnerabilities and proven mitigation strategies against the OWASP Top 10 risks for large language models (LLMs), through interactive demos and hands-on exercises. Register for this talk

Code talks

Code talks are similar to our popular chalk talk format, but with a focus on live coding or code samples rather than whiteboarding. These sessions look into the actual code used to build a solution, allowing attendees to understand the “why” behind the approach and witness the development process, including any errors that may arise. Participants are encouraged to ask questions and follow along for a deeper, hands-on learning experience.

SEC401 – Inspect and secure your application with generative AI. Harness the power of generative AI to bolster your application security, as we unveil how AI-driven tools can rapidly detect vulnerabilities and recommend remediation strategies, empowering you to build more secure software with ease. Register for this talk
SEC405 – Consolidated data protection insights with generative AI. Discover how to secure your AWS KMS keys across your accounts by using Amazon Q in QuickSight for quick, actionable insights. Register for this talk

Builders’ sessions

Interact with small groups, led by an AWS expert providing interactive learning about how to build on AWS. Each builders’ session begins with a short explanation or demonstration of what attendees are building, then it’s your turn to build! The expert will guide you end-to-end through this hands-on experience.

Note: You must bring your own laptop to participate in these sessions.

DOP302 – Creating secure code with Amazon Q Developer. Supercharge your coding prowess with Amazon Q Developer, as you gain hands-on experience using its AI-powered capabilities to write more secure, optimized code, detect vulnerabilities, and implement instant remediations—transforming your development workflow. Register for this session
SMB302 – Empower your business with defense-in-depth architecture. Empower your small-to-medium business to innovate more securely with generative AI by exploring practical, cost-effective defense-in-depth strategies, layered security architectures, and AI-specific safeguards to build resilient, trusted AI-powered solutions in the AWS Cloud. Register for this session

Workshops

Workshops are 2-hour interactive sessions where you collaborate in teams or work individually to solve real-world challenges by using AWS services, making them perfect for hands-on learning. Each workshop begins with a brief lecture, followed by dedicated time to work through the problem.

Note: Don’t forget to bring your laptop to build alongside AWS experts.

SEC305 – Generative AI-based code remediations and patch management at scale. Experience hands-on how to use generative AI to assist in automating vulnerability detection and remediation across AWS Lambda, containers, and Amazon Elastic Compute Cloud (Amazon EC2) at scale, empowering your team to proactively secure your applications. Register for this workshop
SEC306 – Securing your generative AI applications on AWS. Gain hands-on experience securing generative AI applications by using AWS services and features. Deploy a vulnerable sample AI app, then implement layered security controls to protect, detect, and respond to issues. Use these best practices to secure your own AI apps when you return home! Register for this workshop
SEC309 – AWS IAM Identity Center: Secure access to generative AI applications. You’ll learn how to build an identity-aware chat experience, train it on a sample dataset, and connect it to an external workforce identity provider by using native integration between Amazon Q Business and AWS Identity and Access Management (IAM) Identity Center. Register for this workshop
SEC310 – Persona-based access to enterprise data for generative AI applications. Learn how to secure document access in generative AI applications by using retrieval augmented generation (RAG), metadata filtering, and Amazon Cognito in this interactive workshop. Register for this workshop

Expo

Want to talk directly with an AWS security expert on generative AI security, or a variety of other security topics? Then don’t miss this opportunity to have one-on-one conversations with leading AWS security experts in the Security Activation area of the expo floor to help you take your organization’s security posture to new heights.

Delve into key security domains such as:

Detection and Response: Explore techniques for detecting and responding to security risks to help protect your workloads at scale.
Network and Infrastructure Security: Learn how to build and manage a secure global network with AWS services.
Application Security: Discover strategies to ship secure software and address the challenges of application security.
Identity and Access Management: Adopt modern cloud-native identity solutions and apply least-privilege access controls.
Digital Sovereignty & Data Protection: Maintain control over your data and choose how to secure and manage it in the AWS Cloud.

Still time for fun!

Register today

It’s going to be an amazing event, and we can’t wait to see you at re:Invent 2024! Register now to secure your spot.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Achieving Frugal Architecture using the AWS Well-Architected Framework guidance

2024-08-14 Ashley DeLoach

Post Syndicated from Ashley DeLoach original https://aws.amazon.com/blogs/architecture/achieving-frugal-architecture-using-the-aws-well-architected-framework-guidance/

As part of the re:Invent 2023 keynote, Dr. Werner Vogels introduced the Frugal Architect mindset. This mindset emphasizes the importance of continuous learning, curiosity, and regular revision of architectural choices with a focus on cost and sustainability. Cost and sustainability should be treated as critical non-functional requirements, alongside factors like security, compliance, and performance. The Frugal Architect approach involves measuring and optimizing cost at every stage of the development process, which allows for innovation in parallel with promoting responsible resource usage. In the rapidly-evolving technology landscape, builders should adopt the Frugal Architect mindset to balance innovation with cost efficiency and environmental sustainability.

This blog discusses how the six pillars of the AWS Well-Architected Framework (operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability) align with the seven Frugal Architect laws. It demonstrates how adhering to the principles and best practices outlined in these pillars can help architects and builders effectively implement the Frugal Architect laws in their projects. The Well-Architected Framework provides a comprehensive set of guidelines that embed the concepts of frugality, efficiency, and cost effectiveness, which are the core tenets of the Frugal Architect laws. By following the Framework’s pillars, architects can build secure, reliable, efficient, and cost-optimized systems and promote sustainability.

Make Cost a Non-functional Requirement (Law 1)

Non-functional requirements are criteria that evaluate a system’s operation instead of its specific features or functionality. This includes aspects like accessibility, availability, scalability, security, portability, maintainability, and compliance. However, one crucial non-functional requirement that is often overlooked is cost. Consider implications early on and throughout the design, development, and operation of your systems. Organizations can strike a balance between desired features, time-to-market, and operational efficiency through early prioritization of cost considerations. The Frugal Architect argues that you should treat cost as a fundamental non-functional requirement that should be given upfront consideration when planning and initiating system development projects.

The Cost Optimization Pillar of the AWS Well-Architected Framework provides guidance on how to optimize costs when using AWS Cloud services. It emphasizes treating cost as a key requirement, not an afterthought. The main principles focus on the importance of a robust financial management processes, adoption of a cloud consumption model that allows for flexible scaling and pay-per-use billing, continual measurement of outputs against costs to optimize efficiency, use of managed services to minimize operational overhead, and implementation of transparent cost attribution to tie cloud spending to revenue sources and workloads. Organizations that follow these practices can effectively manage and optimize their costs and benefit from the scalability and agility of cloud computing.

These cost optimization principles can help organizations maximize the financial benefits of using the AWS Cloud and avoid wasteful spending. Cost optimization is an ongoing process that includes rightsizing, higher output for the same cost, and use of the most cost-effective AWS services. The pillar promotes a disciplined approach to evaluate trade-offs between cost and other optimization areas like performance or reliability. Overall, you can use this pillar to make informed decisions to provision and operate AWS services cost-effectively.

Systems that Last Align Cost to Business (Law 2)

The durability and longevity of a system are closely tied to how well its costs align with the underlying business model. During the creation of a system, consider revenue sources and profit drivers. The key is to identify the primary dimension or aspect that generates revenue, and then verify that the system architecture supports and optimizes for that revenue-generating dimension. Essentially, revenue and profitability considerations should be the primary forces behind cost decisions in system design.

The AWS Well-Architected Cost Optimization Pillar provides practices and guidance for organizations to accurately monitor their AWS costs and usage. This visibility helps users understand the profitability of different business units and products, which facilitates informed decisions on resource allocation across the organization. Organizations can implement these practices to gain insights into their AWS spending patterns, which aids in development of effective cost optimization strategies. Overall, accurate expenditure analysis and attribution are crucial for organizations to optimize cloud costs, measure ROI, and make data-driven resource allocation decisions.

It’s important to accurately identify and attribute cloud costs to specific workloads. The cloud allows for transparent cost attribution, which helps organizations link costs to individual revenue streams and workload owners. This granular cost attribution data empowers workload owners to measure return on investment (ROI) for their workloads. With detailed cost information, workload owners can optimize resource utilization and reduce costs by rightsizing resources, eliminating waste, and making informed decisions. Organizations must use accurate cost attribution to understand where their cloud spending is going and verify that resources are being used efficiently across different workloads and revenue streams.

Architecting is a Series of Trade-Offs (Law 3)

Architectural decisions involve trade-offs, particularly between cost, resilience, and performance. Systems will inevitably fail, so investment in resilience is important but may impact performance. It’s important to find the right balance between technical requirements and business needs and align with risk tolerance and budget constraints. Frugality is about maximizing value, not just minimizing spend. Frugality means that you determine what you’re can pay for based on your priorities and make informed trade-off decisions. Ultimately, architectural choices require careful consideration of the tensions between different non-functional requirements.

The AWS Well-Architected Framework helps you make architectural trade-offs through its design principles and practices across its six pillars with your business requirements in mind. As you architect workloads, you make trade-offs between pillars based on your business context. You might optimize to improve the sustainability impact and reduce cost at the expense of reliability in development environments, or for mission-critical solutions. You might optimize reliability with increased costs and sustainability impact. In ecommerce solutions, performance can affect revenue and customer propensity to buy. Security generally is not a viable trade-off against the other pillars.

Rather than optimizing for any single pillar, the Framework guides a holistic evaluation across all pillars to determine the right architectural approach. Organizations can use AWS best practices while they find the optimal balance that aligns with their unique requirements. The key is making intentional trade-off decisions instead of following any uniform approach.

Unobserved Systems Lead to Unknown Costs (Law 4)

Without proper observation and measurement, the true operational costs of a system remain hidden, and wasteful practices can persist unnoticed. Just as exposing a utility meter prompts more mindful usage, visibility increases into costs can drive more sustainable behaviors. While implementing comprehensive monitoring requires upfront investment, the long-term benefits of conserving resources and optimizing efficiency make it a worthwhile endeavor. Ultimately, you should maintain cost awareness to foster a culture of responsible, sustainable practices.

The Operational Excellence Pillar of the AWS Well-Architected Framework emphasizes the importance of observability to gain actionable insights into workloads. This involves creation of key performance indicators (KPIs) and use of observability data telemetry to comprehensively understand workload behavior, performance, reliability, cost, and health. Organizations can implement observability best practices to make informed decisions and take prompt action when business outcomes are at risk due to issues with workload operation. Observability data provides visibility into the current state and helps identify areas for improvement. This means that organizations can be proactive in performance optimization, reliability enhancement, and cost reduction based on the actionable insights derived from observability telemetry data. Overall, observability is crucial for maintenance of operational excellence through the use of data-driven decision-making and continuous improvement of workloads.

Overall, monitoring guidance is a core component across multiple pillars of the Well-Architected Framework, as it helps organizations effectively manage and optimize their cloud workloads. For more detail on the monitoring principles of the AWS Well-Architected Framework, see Cost-Aware Architectures Implement Cost Controls (Law 5).

Cost-Aware Architectures Implement Cost Controls (Law 5)

The key aspects of frugal architecture combine granular controls with robust monitoring to identify areas for optimization. This helps you optimize costs and maintain a good user experience. With a robust monitoring system, you can take action where improvements are needed.

The AWS Well-Architected Framework aligns with the concept of frugality, which focuses on maximizing value rather than just minimizing spending. The Framework helps businesses achieve maximum value by making architectural choices that meet their specific requirements.

The Cost Optimization Pillar emphasizes the continual monitoring of usage and costs to identify opportunities for efficiency improvements and cost savings. This includes expenditure analysis, adoption of consumption-based models, and implementation of cloud financial management practices.

The Security Pillar, Reliability Pillar, and Performance Efficiency Pillar reinforce the importance of monitoring systems, workloads, and costs in real-time to maintain security, automatically recover from failures, and optimize performance relative to cost.

The Sustainability Pillar focuses on measurement of a workload’s current and forecasted environmental impact. It recommends continual evaluation of new hardware and software offerings that can reduce the environmental footprint.

Overall, monitoring guidance spans multiple Well-Architected pillars to maximize value through optimization of cost, performance, security, reliability, and sustainability.

Cost Optimization is Incremental (Law 6)

Cost efficiency is a continuous process, not a one-time goal. Regularly monitor your systems to identify inefficient patterns and areas for optimization. Revisit and refine systems periodically to find additional opportunities for improvement and further reduce costs over time.

The Cost Optimization Pillar covers principles like analysis and attribution of expenditure, measurement of overall efficiency, adoption of a consumption model, and implementation of cloud financial management practices.

Additionally, the Operational Excellence Pillar provides principles that apply not just to cost optimization but all pillars. These include observability for actionable insights, safe automation where possible, frequent small reversible changes, frequent refinement of operations procedures, anticipation of failure, and documentation and distribution of learning from operational events and metrics.

Organizations can follow these AWS Well-Architected Framework principles and their practices to continuously improve their cloud architectures and operations and optimize costs effectively.

Unchallenged Success Leads to Assumptions (Law 7)

We should continue to reevaluate past approaches, even those that were previously successful. Just because something worked before does not mean that it is still the best method. Grace Hopper, a computer scientist, mathematician, and United States Navy rear admiral, cautioned against blind adherence to tradition, saying that “we’ve always done it this way” is a dangerous mindset. We must be willing to question the old ways and explore new and potentially better methods.

The AWS Well-Architected Framework advocates for an evolutionary architecture approach to system design. Traditional architectures are often designed as static, with only a few major version updates during the system’s lifetime. However, as businesses and requirements change over time, initial architectural decisions can limit the ability to adapt and evolve the system. Cloud computing enables capabilities like automated testing and lower-risk design changes, which allows systems to evolve continually rather than being constrained by the original design. An evolutionary architecture positions businesses to take advantage of new innovations and changes as part of standard practice. Rather than being locked into original architectural choices, an evolutionary approach fosters ongoing adaptation and modernization as requirements shift. This contrasts with traditional fixed architectures that make it difficult to evolve over time and provides greater flexibility to evolve systems iteratively.

The Operational Excellence Pillar includes implementation of observability to understand system behavior, safe automation of processes, frequent but reversible changes, regular refinement of operations procedures, proactive anticipation potential failures proactively, and distribution of learnings from operational events and metrics to drive continuous improvement.

Overall, the Well-Architected Framework provides guidance on evolutionary architecture and operations processes to effectively manage increasing software complexity over time.

Conclusion

Frugality is about maximizing value, rather than just minimizing costs. Following AWS Well-Architected Framework best practices regarding security, reliability, and operational excellence can help realize frugal yet robust architectures. True frugality involves optimizing costs by aligning spending with areas that deliver the highest business value and impact. The Well-Architected Framework provides guidance for making architectural decisions that increase efficiency, lower risks, and maximize return on cloud investments. This involves determining priorities, understanding sources of value, and making informed trade-off decisions based on those priorities. It’s important to avoid indiscriminate cost-cutting and instead focus on resources on what matters most to drive value for the organization. By following Well-Architected best practices, companies can practice frugality in a strategic way that balances optimization with business goals.

Start your Frugal Architecture journey with AWS Well-Architected today by reading the documentation or visiting the AWS Well-Architected Tool in the console.

AWS re:Invent 2023: Security, identity, and compliance recap

2024-01-26 Nisha Amthul

Post Syndicated from Nisha Amthul original https://aws.amazon.com/blogs/security/aws-reinvent-2023-security-identity-and-compliance-recap/

In this post, we share the key announcements related to security, identity, and compliance at AWS re:Invent 2023, and offer details on how you can learn more through on-demand video of sessions and relevant blog posts. AWS re:Invent returned to Las Vegas in November 2023. The conference featured over 2,250 sessions and hands-on labs, with over 52,000 attendees over five days. If you couldn’t join us in person or want to revisit the security, identity, and compliance announcements and on-demand sessions, this post is for you.

At re:Invent 2023, and throughout the AWS security service announcements, there are key themes that underscore the security challenges that we help customers address through the sharing of knowledge and continuous development in our native security services. The key themes include helping you architect for zero trust, scalable identity and access management, early integration of security in the development cycle, container security enhancement, and using generative artificial intelligence (AI) to help improve security services and mean time to remediation.

Key announcements

To help you more efficiently manage identity and access at scale, we introduced several new features:

A week before re:Invent, we announced two new features of Amazon Verified Permissions:
- Batch authorization — Batch authorization is a new way for you to process authorization decisions within your application. Using this new API, you can process 30 authorization decisions for a single principal or resource in a single API call. This can help you optimize multiple requests in your user experience (UX) permissions.
- Visual schema editor — This new visual schema editor offers an alternative to editing policies directly in the JSON editor. View relationships between entity types, manage principals and resources visually, and review the actions that apply to principal and resources types for your application schema.
We launched two new features for AWS Identity and Access Management (IAM) Access Analyzer:
- Unused access — The new analyzer continuously monitors IAM roles and users in your organization in AWS Organizations or within AWS accounts, identifying unused permissions, access keys, and passwords. Using this new capability, you can benefit from a dashboard to help prioritize which accounts need attention based on the volume of excessive permissions and unused access findings. You can set up automated notification workflows by integrating IAM Access Analyzer with Amazon EventBridge. In addition, you can aggregate these new findings about unused access with your existing AWS Security Hub findings.
- Custom policy checks — This feature helps you validate that IAM policies adhere to your security standards ahead of deployments. Custom policy checks use the power of automated reasoning—security assurance backed by mathematical proof—to empower security teams to detect non-conformant updates to policies proactively. You can move AWS applications from development to production more quickly by automating policy reviews within your continuous integration and continuous delivery (CI/CD) pipelines. Security teams automate policy reviews before deployments by collaborating with developers to configure custom policy checks within AWS CodePipeline pipelines, AWS CloudFormation hooks, GitHub Actions, and Jenkins jobs.
We announced AWS IAM Identity Center trusted identity propagation to manage and audit access to AWS Analytics services, including Amazon QuickSight, Amazon Redshift, Amazon EMR, AWS Lake Formation, and Amazon Simple Storage Service (Amazon S3) through S3 Access Grants. This feature of IAM Identity Center simplifies data access management for users, enhances auditing granularity, and improves the sign-in experience for analytics users across multiple AWS analytics applications.

To help you improve your security outcomes with generative AI and automated reasoning, we introduced the following new features:

Amazon Inspector expands AWS Lambda code scanning with generative AI-powered remediation — Now you can assess your custom proprietary AWS Lambda code for security issues such as injection flaws and data leaks. This update provides you with actionable security findings, including affected code snippets and remediation suggestions, simplifying updates to vulnerable code.
Amazon CodeWhisperer provides code suggestions to help remediate identified security and code quality issues tailored to your application code. You can use this new capability to help review and accept fixes quickly with confidence. Security scanning is available for Java, Python, and JavaScript, and is now also available for TypeScript, C#, CloudFormation (YAML, JSON), AWS CDK (TypeScript, Python), and HashiCorp Terraform.
Amazon Detective introduces finding group summaries using generative AI — With Amazon Detective finding group summaries, you can more quickly locate and review key insights on suspicious activity identified in finding groups in natural language. This makes it simpler to investigate and understand unusual or suspicious activities.
AWS Config launches generative AI-powered natural language querying (Preview) — With this feature, you can simplify your investigation and search of AWS resource configurations and compliance metadata.

AWS Control Tower launched a set of 65 purpose-built controls designed to help you meet your digital sovereignty needs. In November 2022, we launched AWS Digital Sovereignty Pledge, our commitment to offering all AWS customers the most advanced set of sovereignty controls and features available in the cloud. Introducing AWS Control Tower controls that support digital sovereignty is an additional step in our roadmap of capabilities for data residency, granular access restriction, encryption, and resilience. AWS Control Tower offers you a consolidated view of the controls enabled, your compliance status, and controls evidence across multiple accounts.

We announced two new feature expansions for Amazon GuardDuty to provide the broadest threat detection coverage:

GuardDuty ECS Runtime Monitoring — This expansion introduces runtime threat detection for Amazon Elastic Container Service (Amazon ECS) workloads, including serverless container workloads running on AWS Fargate. GuardDuty now provides the broadest container security coverage available when you run container workloads on AWS.
GuardDuty runtime monitoring for Amazon EC2 (Preview) — This expansion introduces runtime threat detection for Amazon Elastic Compute Cloud (Amazon EC2) workloads. GuardDuty EC2 Runtime Monitoring deepens threat detection coverage for Amazon EC2 workloads.

We launched two new capabilities for Amazon Inspector in addition to Amazon Inspector code remediation for Lambda function to help you detect software vulnerabilities at scale:

Enhanced container image security — Amazon Inspector now integrates with developer tools, introducing a new set of open source plugins and an API. You can use this new capability to assess your container images for software vulnerabilities at build time directly from your CI/CD pipelines wherever they are running.
Amazon Inspector agentless vulnerability assessments for Amazon EC2 (Preview) — mazon Inspector can continuously monitor your Amazon EC2 instances without installing an agent or additional software.

We introduced four new capabilities in AWS Security Hub to help you address security gaps across your organization and enhance the user experience for security teams, providing increased visibility:

Central configuration — Streamline and simplify how you set up and administer Security Hub in your multi-account, multi-Region organizations. With central configuration, you can use the delegated administrator account as a single pane of glass for your security findings—and also for your organization’s configurations in Security Hub.
Customize security controls — You can now refine the best practices monitored by Security Hub controls to meet more specific security requirements. There is support for customer-specific inputs in Security Hub controls, so you can customize your security posture monitoring on AWS.
Metadata enrichment for findings — This enrichment adds resource tags, a new AWS application tag, and account name information to every finding ingested into Security Hub. This includes findings from AWS security services such as GuardDuty, Amazon Inspector, and IAM Access Analyzer, in addition to a large and growing list of AWS Partner Network (APN) solutions. Using this enhancement, you can better contextualize, prioritize, and act on your security findings.
Dashboard enhancements — You can now filter and customize your dashboard views, and access a new set of widgets that we carefully chose to help reflect the modern cloud security threat landscape and relate to potential threats and vulnerabilities in your AWS cloud environment. This improvement makes it simpler for you to focus on risks that require your attention, providing a more comprehensive view of your cloud security.

We added three new capabilities for Amazon Detective in addition to Amazon Detective finding group summaries to simplify the security investigation process:

Detective investigations for IAM — Using this feature, you can investigate IAM objects, such as users and roles, for indicators of compromise (IoCs). This feature helps to determine potential involvement in known tactics from the MITRE ATT&CK framework.
Security investigations for GuardDuty ECS Runtime Monitoring — You can now use the new runtime threat detections from GuardDuty, along with the enhanced visualizations and additional context for detections in Amazon ECS from Detective, to help improve your detection and response to potential threats in your container workloads.
Log retrieval from Amazon Security Lake — Detective integrates with Security Lake, so you can query and retrieve logs stored in Security Lake. Use this integration to get additional information from AWS CloudTrail logs and Amazon Virtual Private Cloud (Amazon VPC) Flow Logs stored in Security Lake when you conduct security investigations in Detective.

We introduced AWS Secrets Manager batch retrieval of secrets to identify and retrieve a group of secrets for your application at once with a single API call. The new API, BatchGetSecretValue, provides greater simplicity for common developer workflows, especially when you need to incorporate multiple secrets into your application.

We worked closely with AWS Partners to create offerings that make it simpler for you to protect your cloud workloads:

AWS Built-in Competency — AWS Built-in Competency Partner solutions help minimize the time it takes for you to figure out the best AWS services to adopt, regardless of use case or category.
AWS Cyber Insurance Competency — AWS has worked with leading cyber insurance partners to help simplify the process of obtaining cyber insurance. This makes it simpler for you to find affordable insurance policies from AWS Partners that integrate their security posture assessment through a user-friendly customer experience with Security Hub.

Experience content on demand

If you weren’t able to join in person or you want to watch a session again, you can see the many sessions that are available on demand.

Keynotes, innovation talks, and leadership sessions

Catch the AWS re:Invent 2023 keynote where AWS chief executive officer Adam Selipsky shares his perspective on cloud transformation and provides an exclusive first look at AWS innovations in generative AI, machine learning, data, and infrastructure advancements. You can also replay the other AWS re:Invent 2023 keynotes.

The security landscape is evolving as organizations adapt and embrace new technologies. In this talk, discover the AWS vision for security that drives business agility. Stream the innovation talk from Amazon chief security officer, Steve Schmidt, and AWS chief information security officer, Chris Betz, to learn their insights on key topics such as Zero Trust, builder security experience, and generative AI.

At AWS, we work closely with customers to understand their requirements for their critical workloads. Our work with the Singapore Government’s Smart Nation and Digital Government Group (SNDGG) to build a Smart Nation for their citizens and businesses illustrates this approach. Watch the leadership session with Max Peterson, vice president of Sovereign Cloud at AWS, and Chan Cheow Hoe, government chief digital technology officer of Singapore, as they share how AWS is helping Singapore advance on its cloud journey to build a Smart Nation.

Breakout sessions and new launch talks

Stream breakout sessions and new launch talks on demand to learn about the following topics:

Discover how AWS, customers, and partners work together to raise their security posture with AWS infrastructure and services.
Learn about trends in identity and access management, detection and response, network and infrastructure security, data protection and privacy, and governance, risk, and compliance.
Dive into our launches! Learn about the latest announcements from security experts, and uncover how new services and solutions can help you meet core security and compliance requirements.

Consider joining us for more in-person security learning opportunities by saving the date for AWS re:Inforce 2024, which will occur June 10-12 in Philadelphia, Pennsylvania. We look forward to seeing you there!

If you’d like to discuss how these new announcements can help your organization improve its security posture, AWS is here to help. Contact your AWS account team today.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

AWS re:Invent 2023 Amazon Redshift Sessions Recap

2023-12-18 Mia Heard

Post Syndicated from Mia Heard original https://aws.amazon.com/blogs/big-data/aws-reinvent-2023-amazon-redshift-sessions-recap/

Amazon Redshift powers data-driven decisions for tens of thousands of customers every day with a fully managed, AI-powered cloud data warehouse, delivering the best price-performance for your analytics workloads. Customers use Amazon Redshift as a key component of their data architecture to drive use cases from typical dashboarding to self-service analytics, real-time analytics, machine learning (ML), data sharing and monetization, and more.

This year’s AWS re:Invent conference, held in Las Vegas from November 27 through December 1, showcased the advancements of Amazon Redshift to help you further accelerate your journey towards modernizing your cloud analytics environments. To learn more about the latest and greatest advancements and how customers are powering data-driven decision-making using Amazon Redshift, watch the re:Invent sessions available on demand listed in this post.

Keynotes

Adam Selipsky, Chief Executive Officer of Amazon Web Services

Watch Adam Selipsky, CEO of Amazon Web Services, as he shares his perspective on cloud transformation. He highlights innovations in data, infrastructure, and artificial intelligence and machine learning that are helping AWS customers achieve their goals faster, mine untapped potential, and create a better future. Learn more about the AWS zero-ETL future with newly launched AWS databases integrations with Amazon Redshift.

Swami Sivasubramanian, Vice President of AWS Data and Machine Learning

Watch Swami Sivasubramanian, Vice President of Data and AI at AWS, to discover how you can use your company data to build differentiated generative AI applications and accelerate productivity for employees across your organization. Learn more about these new generative AI features to increase productivity including Amazon Q generative SQL in Amazon Redshift.

Peter DeSantis, Senior Vice President of AWS Utility Computing

Watch Peter DeSantis, Senior Vice President of AWS Utility Computing, as he deep dives into the engineering that powers AWS services. Get a closer look at how scaling for data warehousing works in AWS with the latest introduction of AI driven scaling and optimizations in Amazon Redshift Serverless to enable better price-performance for your workloads.

Innovation Talks

Data drives transformation: Data foundations with AWS analytics with G2 Krishnamoorthy, Vice President of AWS Analytics

G2’s session discusses strategies for embedding analytics into your applications and ideas for building a data foundation that supports your business initiatives. With new capabilities for self-service and straightforward builder experiences, you can democratize data access for line of business users, analysts, scientists, and engineers. Hear also from Adidas, GlobalFoundries, and University of California, Irvine.

Sessions

ANT203 | What’s new in Amazon Redshift

Watch this session to learn about the newest innovations within Amazon Redshift—the petabyte-scale AWS Cloud data warehousing solution. Amazon Redshift empowers users to extract powerful insights by securely and cost-effectively analyzing data across data warehouses, operational databases, data lakes, third-party data stores, and streaming sources using zero-ETL approaches. Easily build and train machine learning models using SQL within Amazon Redshift to generate predictive analytics and propel data-driven decision-making. Learn about Amazon Redshift’s newest functionality to increase reliability and speed to insights through near-real-time data access, ML, and more—all with impressive price-performance.

ANT322 | Modernize analytics by moving your data warehouse to Amazon Redshift

Watch this session to hear from AWS customers as they share their journeys moving to a modern cloud data warehouse and analytics with Amazon Redshift. Learn best practices for building powerful analytics and ML applications and operating at scale while keeping costs low.

ANT211 | Powering self-service & near real-time analytics with Amazon Redshift

To stay competitive, allowing data citizens across your organization to see near-real-time analytics without worrying about data infrastructure management is crucial for your business. In this session, learn how your data users can get to near-real-time insights on streaming data with Amazon Redshift and AWS streaming data services. Explore a solution using flexible querying tools and a serverless architecture, which brings intelligent automation and scaling capabilities, and maintains consistently high performance for even the most demanding and volatile workloads.

ANT325 | Amazon Redshift: A decade of innovation in cloud data warehousing

Exponential data growth has created unique challenges for data practitioners to manage data warehouses that can support high performance workloads at scale within cost constraints. Amazon Redshift has been constantly innovating over the last decade to give you a modern, massively parallel processing cloud data warehouse that delivers the best price-performance, ease of use, scalability, and reliability. In this session, learn about Amazon Redshift’s technical innovations including serverless, AI/ML-powered autonomics, and zero-ETL data integrations. Discover how you can use Amazon Redshift to build a data mesh architecture to analyze your data.

ANT326 | Set up a zero-ETL-based analytics architecture for your organizations

ETL (extract, transform, and load data) can be challenging, time-consuming, and costly. AWS is building a zero-ETL future with capabilities like streaming ingestion into the data warehouse, federated querying, and connectors that access data in place across databases, data lakes, and third-party data sources without data movement. In this session, learn how zero-ETL investments such as Amazon Aurora zero-ETL integration with Amazon Redshift drive direct integration between AWS data services to allow data engineers to focus on creating value from data instead of spending time and resources building pipelines.

ANT351 | [NEW LAUNCH] Multi-data warehouse writes through Amazon Redshift data sharing

Organizations want simple and secure ways for their teams to meet their ETL SLAs, optimize costs, and collaborate on live data. With multi-data warehouse writes available through Amazon Redshift data sharing, you can write to the same databases with multiple warehouses at the same time. Join this session to learn how you can keep your ETL jobs completing predictably and on time, collaborate on live data with multiple teams, and better optimize your costs with this newly launched capability.

ANT 352 | [NEW LAUNCH] Amazon Q generative SQL in Amazon Redshift Query Editor

SQL, the industry standard language for data analytics, often requires users to spend a lot of time understanding an organization’s complex metadata in order to write and carry out complex SQL queries for data insights. Join this session to learn how you can help SQL users of all skill levels within your organization derive insights faster with the new Amazon Q generative SQL capability in Amazon Redshift Query Editor. This session demonstrates how this functionality works and how to use text prompts in plain English to build effective queries, including complex multi-table join or nested queries.

ANT 354 | [NEW LAUNCH] AI-powered scaling and optimization for Amazon Redshift Serverless

Amazon Redshift Serverless makes it easier to run analytics workloads of any size without having to manage data warehouse infrastructure. Redshift Serverless helps developers, data scientists, and analysts work across various data sources to build reports, applications, machine learning models, and more. In this session, learn about Redshift Serverless new AI-driven scaling and optimization functionality. This new functionality proactively adapts to workload changes and applies tailored performance optimizations by intelligently predicting query patterns and using machine learning, increasing consistent price-performance.

SEC245 | Simplify and improve access control for your AWS analytics services

As organizations adopt new AWS services, end users need more access to data across a full range of AWS analytics services to extract value and insights. Data end users are accustomed to seamless authentication to their AWS applications, and cloud administrators want more granular, user-based access control over their data. Join this session to learn how to simplify and improve access control using a new AWS IAM Identity Center feature, known as trusted identity propagation, along with supported AWS analytics services. Also learn how to audit user and group-based access activity across interconnected AWS managed applications so that you can align better with regulatory and sovereignty requirements.

What’s new with Amazon Redshift

Want to learn more about the most recent features launched in Amazon Redshift? Refer to Amazon Redshift announcements at AWS re:Invent 2023 to enable analytics on all your data to learn about all of the Amazon Redshift launch announcements made at 2023 AWS re:Invent

_______________________________________________________________________

About the Author

Mia Heard is a product marketing manager for Amazon Redshift, a fully managed, AI-powered cloud data warehouse with the best price-performance for analytic workloads.

Use AWS Fault Injection Service to demonstrate multi-region and multi-AZ application resilience

2023-11-30 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-aws-fault-injection-service-to-demonstrate-multi-region-and-multi-az-application-resilience/

AWS Fault Injection Service (FIS) helps you to put chaos engineering into practice at scale. Today we are launching new scenarios that will let you demonstrate that your applications perform as intended if an AWS Availability Zone experiences a full power interruption or connectivity from one AWS region to another is lost.

You can use the scenarios to conduct experiments that will build confidence that your application (whether single-region or multi-region) works as expected when something goes wrong, help you to gain a better understanding of direct and indirect dependencies, and test recovery time. After you have put your application through its paces and know that it works as expected, you can use the results of the experiment for compliance purposes. When used in conjunction with other parts of AWS Resilience Hub, FIS can help you to fully understand the overall resilience posture of your applications.

Intro to Scenarios
We launched FIS in 2021 to help you perform controlled experiments on your AWS applications. In the post that I wrote to announce that launch, I showed you how to create experiment templates and to use them to conduct experiments. The experiments are built using powerful, low-level actions that affect specified groups of AWS resources of a particular type. For example, the following actions operate on EC2 instances and Auto Scaling Groups:

With these actions as building blocks, we recently launched the AWS FIS Scenario Library. Each scenario in the library defines events or conditions that you can use to test the resilience of your applications:

Each scenario is used to create an experiment template. You can use the scenarios as-is, or you can take any template as a starting point and customize or enhance it as desired.

The scenarios can target resources in the same AWS account or in other AWS accounts:

New Scenarios
With all of that as background, let’s take a look at the new scenarios.

AZ Availability: Power Interruption – This scenario temporarily “pulls the plug” on a targeted set of your resources in a single Availability Zone including EC2 instances (including those in EKS and ECS clusters), EBS volumes, Auto Scaling Groups, VPC subnets, Amazon ElastiCache for Redis clusters, and Amazon Relational Database Service (RDS) clusters. In most cases you will run it on an application that has resources in more than one Availability Zone, but you can run it on a single-AZ app with an outage as the expected outcome. It targets a single AZ, and also allows you to disallow a specified set of IAM roles or Auto Scaling Groups from being able to launch fresh instances or start stopped instances during the experiment.

The New actions and targets experience makes it easy to see everything at a glance — the actions in the scenario and the types of AWS resources that they affect:

The scenarios include parameters that are used to customize the experiment template:

The Advanced parameters – targeting tags lets you control the tag keys and values that will be used to locate the resources targeted by experiments:

Cross-Region: Connectivity – This scenario prevents your application in a test region from being able to access resources in a target region. This includes traffic from EC2 instances, ECS tasks, EKS pods, and Lambda functions attached to a VPC. It also includes traffic flowing across Transit Gateways and VPC peering connections, as well as cross-region S3 and DynamoDB replication. The scenario looks like this out of the box:

This scenario runs for 3 hours (unless you change the disruptionDuration parameter), and isolates the test region from the target region in the specified ways, with advanced parameters to control the tags that are used to select the affected AWS resources in the isolated region:

You might also find that the Disrupt and Pause actions used in this scenario useful on their own:

For example, the aws:s3:bucket-pause-replication action can be used to pause replication within a region.

Things to Know
Here are a couple of things to know about the new scenarios:

Regions – The new scenarios are available in all commercial AWS Regions where FIS is available, at no additional cost.

Pricing – You pay for the action-minutes consumed by the experiments that you run; see the AWS Fault Injection Service Pricing Page for more info.

Naming – This service was formerly called AWS Fault Injection Simulator.

— Jeff;

Zonal autoshift – Automatically shift your traffic away from Availability Zones when we detect potential issues

2023-11-30 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/zonal-autoshift-automatically-shift-your-traffic-away-from-availability-zones-when-we-detect-potential-issues/

Today we’re launching zonal autoshift, a new capability of Amazon Route 53 Application Recovery Controller that you can enable to automatically and safely shift your workload’s traffic away from an Availability Zone when AWS identifies a potential failure affecting that Availability Zone and shift it back once the failure is resolved.

When deploying resilient applications, you typically deploy your resources across multiple Availability Zones in a Region. Availability Zones are distinct groups of physical data centers at a meaningful distance apart (typically miles) to make sure that they have diverse power, connectivity, network devices, and flood plains.

To help you protect against an application’s errors, like a failed deployment, an error of configuration, or an operator error, we introduced last year the ability to manually or programmatically trigger a zonal shift. This enables you to shift the traffic away from one Availability Zone when you observe degraded metrics in that zone. It does so by configuring your load balancer to direct all new connections to infrastructure in healthy Availability Zones only. This allows you to preserve your application’s availability for your customers while you investigate the root cause of the failure. Once fixed, you stop the zonal shift to ensure the traffic is distributed across all zones again.

Zonal shift works at the Application Load Balancer (ALB) or Network Load Balancer (NLB) level only when cross-zone load balancing is turned off, which is the default for NLB. In a nutshell, load balancers offer two levels of load balancing. The first level is configured in the DNS. Load balancers expose one or more IP addresses for each Availability Zone, offering a client-side load balancing between zones. Once the traffic hits an Availability Zone, the load balancer sends traffic to registered healthy targets, typically an Amazon Elastic Compute Cloud (Amazon EC2) instance. By default, ALBs send traffic to targets across all Availability Zones. For zonal shift to properly work, you must configure your load balancers to disable cross-zone load balancing.

When zonal shift starts, the DNS sends all traffic away from one Availability Zone, as illustrated by the following diagram.

Manual zonal shift helps to protect your workload against errors originating from your side. But when there is a potential failure in an Availability Zone, it is sometimes difficult for you to identify or detect the failure. Detecting an issue in an Availability Zone using application metrics is difficult because, most of the time, you don’t track metrics per Availability Zone. Moreover, your services often call dependencies across Availability Zone boundaries, resulting in errors seen in all Availability Zones. With modern microservice architectures, these detection and recovery steps must often be performed across tens or hundreds of discrete microservices, leading to recovery times of multiple hours.

Customers asked us if we could take the burden off their shoulders to detect a potential failure in an Availability Zone. After all, we might know about potential issues through our internal monitoring tools before you do.

With this launch, you can now configure zonal autoshift to protect your workloads against potential failure in an Availability Zone. We use our own AWS internal monitoring tools and metrics to decide when to trigger a network traffic shift. The shift starts automatically; there is no API to call. When we detect that a zone has a potential failure, such as a power or network disruption, we automatically trigger an autoshift of your infrastructure’s NLB or ALB traffic, and we shift the traffic back when the failure is resolved.

Obviously, shifting traffic away from an Availability Zone is a delicate operation that must be carefully prepared. We built a series of safeguards to ensure we don’t degrade your application availability by accident.

First, we have internal controls to ensure we shift traffic away from no more than one Availability Zone at a time. Second, we practice the shift on your infrastructure for 30 minutes every week. You can define blocks of time when you don’t want the practice to happen, for example, 08:00–18:00, Monday through Friday. Third, you can define two Amazon CloudWatch alarms to act as a circuit breaker during the practice run: one alarm to prevent starting the practice run at all and one alarm to monitor your application health during a practice run. When either alarm triggers during the practice run, we stop it and restore traffic to all Availability Zones. The state of application health alarm at the end of the practice run indicates its outcome: success or failure.

According to the principle of shared responsibility, you have two responsibilities as well.

First you must ensure there is enough capacity deployed in all Availability Zones to sustain the increase of traffic in remaining Availability Zones after traffic has shifted. We strongly recommend having enough capacity in remaining Availability Zones at all times and not relying on scaling mechanisms that could delay your application recovery or impact its availability. When zonal autoshift triggers, AWS Auto Scaling might take more time than usual to scale your resources. Pre-scaling your resource ensures a predictable recovery time for your most demanding applications.

Let’s imagine that to absorb regular user traffic, your application needs six EC2 instances across three Availability Zones (2×3 instances). Before configuring zonal autoshift, you should ensure you have enough capacity in the remaining Availability Zones to absorb the traffic when one Availability Zone is not available. In this example, it means three instances per Availability Zone (3×3 = 9 instances with three Availability Zones in order to keep 2×3 = 6 instances to handle the load when traffic is shifted to two Availability Zones).

In practice, when operating a service that requires high reliability, it’s normal to operate with some redundant capacity online for eventualities such as customer-driven load spikes, occasional host failures, etc. Topping up your existing redundancy in this way both ensures you can recover rapidly during an Availability Zone issue but can also give you greater robustness to other events.

Second, you must explicitly enable zonal autoshift for the resources you choose. AWS applies zonal autoshift only on the resources you chose. Applying a zonal autoshift will affect the total capacity allocated to your application. As I just described, your application must be prepared for that by having enough capacity deployed in the remaining Availability Zones.

Of course, deploying this extra capacity in all Availability Zones has a cost. When we talk about resilience, there is a business tradeoff to decide between your application availability and its cost. This is another reason why we apply zonal autoshift only on the resources you select.

Let’s see how to configure zonal autoshift
To show you how to configure zonal autoshift, I deploy my now-famous TicTacToe web application using a CDK script. I open the Route 53 Application Recovery Controller page of the AWS Management Console. On the left pane, I select Zonal autoshift. Then, on the welcome page, I select Configure zonal autoshift for a resource.

I select the load balancer of my demo application. Remember that currently, only load balancers with cross-zone load balancing turned off are eligible for zonal autoshift. As the warning on the console reminds me, I also make sure my application has enough capacity to continue to operate with the loss of one Availability Zone.

I scroll down the page and configure the times and days I don’t want AWS to run the 30-minute practice. At first, and until I’m comfortable with autoshift, I block the practice 08:00–18:00, Monday through Friday. Pay attention that hours are expressed in UTC, and they don’t vary with daylight saving time. You may use a UTC time converter application for help. While it is safe for you to exclude business hours at the start, we recommend configuring the practice run also during your business hours to ensure capturing issues that might not be visible when there is low or no traffic on your application. You probably most need zonal autoshift to work without impact at your peak time, but if you have never tested it, how confident are you? Ideally, you don’t want to block any time at all, but we recognize that’s not always practical.

Further down on the same page, I enter the two circuit breaker alarms. The first one prevents the practice from starting. You use this alarm to tell us this is not a good time to start a practice run. For example, when there is an issue ongoing with your application or when you’re deploying a new version of your application to production. The second CloudWatch alarm gives the outcome of the practice run. It enables zonal autoshift to judge how your application is responding to the practice run. If the alarm stays green, we know all went well.

If either of these two alarms triggers during the practice run, zonal autoshift stops the practice and restores the traffic to all Availability Zones.

Finally, I acknowledge that a 30-minute practice run will run weekly and that it might reduce the availability of my application.

Then, I select Create.

And that’s it.

After a few days, I see the history of the practice runs on the Zonal shift history for resource tab of the console. I monitor the history of my two circuit breaker alarms to stay confident everything is correctly monitored and configured.

It’s not possible to test an autoshift itself. It triggers automatically when we detect a potential issue in an Availability Zone. I asked the service team if we could shut down an Availability Zone to test the instructions I shared in this post; they politely declined my request :-).

To test your configuration, you can trigger a manual shift, which behaves identically to an autoshift.

A few more things to know
Zonal autoshift is now available at no additional cost in all AWS Regions, except for China and GovCloud.

We recommend applying the crawl, walk, run methodology. First, you get started with manual zonal shifts to acquire confidence in your application. Then, you turn on zonal autoshift configured with practice runs outside of your business hours. Finally, you modify the schedule to include practice zonal shifts during your business hours. You want to test your application response to an event when you least want it to occur.

We also recommend that you think holistically about how all parts of your application will recover when we move traffic away from one Availability Zone and then back. The list that comes to mind (although certainly not complete) is the following.

First, plan for extra capacity as I discussed already. Second, think about possible single points of failure in each Availability Zone, such as a self-managed database running on a single EC2 instance or a microservice that leaves in a single Availability Zone, and so on. I strongly recommend using managed databases, such as Amazon DynamoDB or Amazon Aurora for applications requiring zonal shifts. These have built-in replication and fail-over mechanisms in place. Third, plan the switch back when the Availability Zone will be available again. How much time do you need to scale your resources? Do you need to rehydrate caches?

You can learn more about resilient architectures and methodologies with this great series of articles from my colleague Adrian.

Finally, remember that only load balancers with cross-zone load balancing turned off are currently eligible for zonal autoshift. To turn off cross-zone load balancing from a CDK script, you need to remove stickinessCookieDuration and add load_balancing.cross_zone.enabled=false on the target group. Here is an example with CDK and Typescript:

    // Add the auto scaling group as a load balancing
    // target to the listener.
    const targetGroup = listener.addTargets('MyApplicationFleet', {
      port: 8080,
      // for zonal shift, stickiness & cross-zones load balancing must be disabled
      // stickinessCookieDuration: Duration.hours(1),
      targets: [asg]
    });    
    // disable cross zone load balancing
    targetGroup.setAttribute("load_balancing.cross_zone.enabled", "false");

Now it’s time for you to select your applications that would benefit from zonal autoshift. Start by reviewing your infrastructure capacity in each Availability Zone and then define the circuit breaker alarms. Once you are confident your monitoring is correctly configured, go and enable zonal autoshift.

— seb

IDE extension for AWS Application Composer enhances visual modern applications development with AI-generated IaC

2023-11-30 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/

Today, I’m happy to share the integrated development environment (IDE) extension for AWS Application Composer. Now you can use AWS Application Composer directly in your IDE to visually build modern applications and iteratively develop your infrastructure as code templates with Amazon CodeWhisperer.

Announced as preview at AWS re:Invent 2022 and generally available in March 2023, Application Composer is a visual builder that makes it easier for developers to visualize, design, and iterate on an application architecture by dragging, grouping, and connecting AWS services on a visual canvas. Application Composer simplifies building modern applications by providing an easy-to-use visual drag-and-drop interface and generates IaC templates in real time.

AWS Application Composer also lets you work with AWS CloudFormation resources. In September, AWS Application Composer announced support for 1000+ AWS CloudFormation resources. This provides you the flexibility to define configuration for your AWS resources at a granular level.

Building modern applications with modern tools
The IDE extension for AWS Application Composer provides you with the same visual drag-and-drop experience and functionality as what it offers you in the console. Utilizing the visual canvas in your IDE means you can quickly prototype your ideas and focus on your application code.

With Application Composer running in your IDE, you can also use the various tools available in your IDE. For example, you can seamlessly integrate IaC templates generated real-time by Application Composer with AWS Serverless Application Model (AWS SAM) to manage and deploy your serverless applications.

In addition to making Application Composer available in your IDE, you can create generative AI powered code suggestions in the CloudFormation template in real time while visualizing the application architecture in split view. You can pair and synchronize Application Composer’s visualization and CloudFormation template editing side by side in the IDE without context switching between consoles to iterate on their designs. This minimizes hand coding and increase your productivity.

Using AWS Application Composer in Visual Studio Code
First, I need to install the latest AWS Toolkit for Visual Studio Code plugin. If you already have the AWS Toolkit plugin installed, you only need to update the plugin to start using Application Composer.

To start using Application Composer, I don’t need to authenticate into my AWS account. With Application Composer available on my IDE, I can open my existing AWS CloudFormation or AWS SAM templates.

Another method is to create a new blank file, then right-click on the file and select Open with Application Composer to start designing my application visually.

This will provide me with a blank canvas. Here I have both code and visual editors at the same time to build a simple serverless API using Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. Any changes that I make on the canvas will also be reflected in real time on my IaC template.

I get consistent experiences, such as when I use the Application Composer console. For example, if I make some modifications to my AWS Lambda function, it will also create relevant files in my local folder.

With IaC templates available in my local folder, it’s easier for me to manage my applications with AWS SAM CLI. I can create continuous integration and continuous delivery (CI/CD) with sam pipeline or deploy my stack with sam deploy.

One of the features that accelerates my development workflow is the built-in Sync feature that seamlessly integrates with AWS SAM command sam sync. This feature syncs my local application changes to my AWS account, which is helpful for me to do testing and validation before I deploy my applications into a production environment.

Developing IaC templates with generative AI
With this new capability, I can use generative AI code suggestions to quickly get started with any of CloudFormation’s 1000+ resources. This also means that it’s now even easier to include standard IaC resources to extend my architecture.

For example, I need to use Amazon MQ, which is a standard IaC resource, and I need to modify some configurations for its AWS CloudFormation resource using Application Composer. In the Resource configuration section, change some values if needed, then choose Generate. Application Composer provides code suggestions that I can accept and incorporate into my IaC template.

This capability helps me to improve my development velocity by eliminating context switching. I can design my modern applications using AWS Application Composer canvas and use various tools such as Amazon CodeWhisperer and AWS SAM to accelerate my development workflow.

Things to know
Here are a couple of things to note:

Supported IDE – At launch, this new capability is available for Visual Studio Code.

Pricing – The IDE extension for AWS Application Composer is available at no charge.

Get started with IDE extension for AWS Application Composer by installing the latest AWS Toolkit for Visual Studio Code.

Happy coding!
— Donnie

Amazon SageMaker Studio adds web-based interface, Code Editor, flexible workspaces, and streamlines user onboarding

2023-11-30 Antje Barth

Post Syndicated from Antje Barth original https://aws.amazon.com/blogs/aws/amazon-sagemaker-studio-adds-web-based-interface-code-editor-flexible-workspaces-and-streamlines-user-onboarding/

Today, we are announcing an improved Amazon SageMaker Studio experience! The new SageMaker Studio web-based interface loads faster and provides consistent access to your preferred integrated development environment (IDE) and SageMaker resources and tooling, irrespective of your IDE choice. In addition to JupyterLab and RStudio, SageMaker Studio now includes a fully managed Code Editor based on Code-OSS (Visual Studio Code Open Source).

Both Code Editor and JupyterLab can be launched using a flexible workspace. With spaces, you can scale the compute and storage for your IDE up and down as you go, customize runtime environments, and pause-and-resume coding anytime from anywhere. You can spin up multiple such spaces, each configured with a different combination of compute, storage, and runtimes.

SageMaker Studio now also comes with a streamlined onboarding and administration experience to help both individual users and enterprise administrators get started in minutes. Let me give you a quick tour of some of these highlights.

New SageMaker Studio web-based interface
The new SageMaker Studio web-based interface acts as a command center for launching your preferred IDE and accessing your SageMaker tools to build, train, tune, and deploy models. You can now view SageMaker training jobs and endpoints in SageMaker Studio and access foundation models (FMs) via SageMaker JumpStart. Also, you no longer need to manually upgrade SageMaker Studio.

New Code Editor based on Code-OSS (Visual Studio Code Open Source)
As a data scientist or machine learning (ML) practitioner, you can now sign in to SageMaker Studio and launch Code Editor directly from your browser. With Code Editor, you have access to thousands of VS Code compatible extensions from Open VSX registry and the preconfigured AWS toolkit for Visual Studio Code for developing and deploying applications on AWS. You can also use the artificial intelligence (AI)-powered coding companion and security scanning tool powered by Amazon CodeWhisperer and Amazon CodeGuru.

Launch Code Editor and JupyterLab in a flexible workspace
You can launch both Code Editor and JupyterLab using private spaces that only the user creating the space has access to. This flexible workspace is designed to provide a faster and more efficient coding environment.

The spaces come preconfigured with a SageMaker distribution that contains popular ML frameworks and Python packages. With the help of the AI-powered coding companions and security tools, you can quickly generate, debug, explain, and refactor your code.

In addition, SageMaker Studio comes with an improved collaboration experience. You can use the built-in Git integration to share and version code or bring your own shared file storage using Amazon EFS to access a collaborative filesystem across different users or teams.

Streamlined user onboarding and administration
With redesigned setup and onboarding workflows, you can now set up SageMaker Studio domains within minutes. As an individual user, you can now use a one-click experience to launch SageMaker Studio using default presets and without the need to learn about domains or AWS IAM roles.

As an enterprise administrator, step-by-step instructions help you choose the right authentication method, connect to your third-party identity providers, integrate networking and security configurations, configure fine-grained access policies, and choose the right applications to enable in SageMaker Studio. You can also update settings at any time.

To get started, navigate to the SageMaker console and select either Set up for single user or Set up for organization.

The single-user setup will start deploying a SageMaker Studio domain using default presets and will be ready within a few minutes. The setup for organizations will guide you through the configuration step-by-step. Note that you can choose to keep working with the classic SageMaker Studio experience or start exploring the new experience.

Now available
The new Amazon SageMaker Studio experience is available today in all AWS Regions where SageMaker Studio is available. Starting today, new SageMaker Studio domains will default to the new web-based interface. If you have an existing setup and want to start using the new experience, check out the SageMaker Developer Guide for instructions on how to migrate your existing domains.

Give it a try, and let us know what you think. You can send feedback to AWS re:Post for Amazon SageMaker Studio or through your usual AWS contacts.

Start building your ML projects with Amazon SageMaker Studio today!

— Antje

Three new capabilities for Amazon Inspector broaden the realm of vulnerability scanning for workloads

2023-11-30 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/three-new-capabilities-for-amazon-inspector-broaden-the-realm-of-vulnerability-scanning-for-workloads/

Today, Amazon Inspector adds three new capabilities to increase the realm of possibilities when scanning your workloads for software vulnerabilities:

Amazon Inspector introduces a new set of open source plugins and an API allowing you to assess your container images for software vulnerabilities at build time directly from your continuous integration and continuous delivery (CI/CD) pipelines wherever they are running.
Amazon Inspector can now continuously monitor your Amazon Elastic Compute Cloud (Amazon EC2) instances without installing an agent or additional software (in preview).
Amazon Inspector uses generative artificial intelligence (AI) and automated reasoning to provide assisted code remediation for your AWS Lambda functions.

Amazon Inspector is a vulnerability management service that continually scans your AWS workloads for known software vulnerabilities and unintended network exposure. Amazon Inspector automatically discovers and scans running EC2 instances, container images in Amazon Elastic Container Registry (Amazon ECR) and within your CI/CD tools, and Lambda functions.

We all know engineering teams often face challenges when it comes to promptly addressing vulnerabilities. This is because of the tight release deadlines that force teams to prioritize development over tackling issues in their vulnerability backlog. But it’s also due to the complex and ever-evolving nature of the security landscape. As a result, a study showed that organizations take 250 days on average to resolve critical vulnerabilities. It is therefore crucial to identify potential security issues early in the development lifecycle to prevent their deployment into production.

Detecting vulnerabilities in your AWS Lambda functions code
Let’s start close to the developer with Lambda functions code.

In November 2022 and June 2023, Amazon Inspector added the capability to scan your function’s dependencies and code. Today, we’re adding generative AI and automated reasoning to analyze your code and automatically create remediation as code patches.

Amazon Inspector can now provide in-context code patches for multiple classes of vulnerabilities detected during security scans. Amazon Inspector extends the assessment of your code for security issues like injection flaws, data leaks, weak cryptography, or missing encryption. Thanks to generative AI, Amazon Inspector now provides suggestions how to fix it. It shows affected code snippets in context with suggested remediation.

Here is an example. I wrote a short snippet of Python code with a hardcoded AWS secret key. Never do that!

def create_session_noncompliant():
    import boto3
    # Noncompliant: uses hardcoded secret access key.
    sample_key = "AjWnyxxxxx45xxxxZxxxX7ZQxxxxYxxx1xYxxxxx"
    boto3.session.Session(aws_secret_access_key=sample_key)
    return response

I deploy the code. This triggers the assessment. I open the AWS Management Console and navigate to the Amazon Inspector page. In the Findings section, I find the vulnerability. It gives me the Vulnerability location and the Suggested remediation in a plain natural language explanation but also in diff text and graphical formats.

Detecting vulnerabilities in your container CI/CD pipeline
Now, let’s move to your CI/CD pipelines when building containers.

Until today, Amazon Inspector was able to assess container images once they were built and stored in Amazon Elastic Container Registry (Amazon ECR). Starting today, Amazon Inspector can detect security issues much sooner in the development process by assessing container images during their build within CI/CD tools. Assessment results are returned in near real-time directly to the CI/CD tool’s dashboard. There is no need to enable Amazon Inspector to use this new capability.

We provide ready-to-use CI/CD plugins for Jenkins and JetBrain’s TeamCity, with more to come. There is also a new API (inspector-scan) and command (inspector-sbomgen) available from our AWS SDKs and AWS Command Line Interface (AWS CLI). This new API allows you to integrate Amazon Inspector in the CI/CD tool of your choice.

Upon execution, the plugin runs a container extraction engine on the configured resource and generates a CycloneDX-compatible software bill of materials (SBOM). Then, the plugin sends the SBOM to Amazon Inspector for analysis. The plugin receives the result of the scan in near real-time. It parses the response and generates outputs that Jenkins or TeamCity uses to pass or fail the execution of the pipeline.

To use the plugin with Jenkins, I first make sure there is a role attached to the EC2 instance where Jenkins is installed, or I have an AWS access key and secret access key with permissions to call the Amazon Inspector API.

I install the plugin directly from Jenkins (Jenkins Dashboard > Manage Jenkins > Plugins)

Then, I add an Amazon Inspector Scan step in my pipeline.

I configure the step with the IAM Role I created (or an AWS access key and secret access key when running on premises), my Docker Credentials, the AWS Region, and the Image Id.

When Amazon Inspector detects vulnerabilities, it reports them to the plugin. The build fails, and I can view the details directly in Jenkins.

The SBOM generation understands packages or applications for popular operating systems, such as Alpine, Amazon Linux, Debian, Ubuntu, and Red Hat packages. It also detects packages for Go, Java, NodeJS, C#, PHP, Python, Ruby, and Rust programming languages.

Detecting vulnerabilities on Amazon EC2 without installing agents (in preview)
Finally, let’s talk about agentless inspection of your EC2 instances.

Currently, Amazon Inspector uses AWS Systems Manager and the AWS Systems Manager Agent (SSM Agent) to collect information about the inventory of your EC2 instances. To ensure Amazon Inspector can communicate with your instances, you have to ensure three conditions. First, a recent version of the SSM Agent is installed on the instance. Second, the SSM Agent is started. And third, you attached an IAM role to the instance to allow the SSM Agent to communicate back to the SSM service. This seems fair and simple. But it is not when considering large deployments across multiple OS versions, AWS Regions, and accounts, or when you manage legacy applications. Each instance launched that doesn’t satisfy these three conditions is a potential security gap in your infrastructure.

With agentless scanning (in preview), Amazon Inspector doesn’t require the SSM Agent to scan your instances. It automatically discovers existing and new instances and schedules a vulnerability assessment for them. It does so by taking a snapshot of the instance’s EBS volumes and analyzing the snapshot. This technique has the extra advantage of not consuming any CPU cycle or memory on your instances, leaving 100 percent of the (virtual) hardware available for your workloads. After the analysis, Amazon Inspector deletes the snapshot.

To get started, enable hybrid scanning under EC2 scanning settings in the Amazon Inspector section of the AWS Management Console. Hybrid mode means Amazon Inspector continues to use the SSM Agent–based scanning for instances managed by SSM and automatically switches to agentless for instances that are not managed by SSM.

Under Account management, I can verify the list of scanned instances. I can see which instances are scanned with the SSM Agent and which are not.

Under Findings, I can filter by vulnerability, by account, by instance, and so on. I select by instance and select the agentless instance I want to review.

For that specific instance, Amazon Inspector lists more than 200 findings, sorted by severity.

As usual, I can see the details of a finding to understand what the risk is and how to mitigate it.

Pricing and availability
Amazon Inspector code remediation for Lambda functions is available in ten Regions: US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, London, Stockholm). It is available at no additional cost.

Amazon Inspector agentless vulnerability scanning for Amazon EC2 is available in preview in three AWS Regions: US East (N. Virginia), US West (Oregon), and Europe (Ireland).

The new API to scan containers at build time is available in the 21 AWS Regions where Amazon Inspector is available today.

There are no upfront or subscription costs. We charge on-demand based on the volume of activity. There is a price per EC2 instance or container image scan. As usual, the Amazon Inspector pricing page has the details.

Start today by adding the Jenkins or TeamCity agent to your containerized application CI/CD pipelines or activate the agentless Amazon EC2 inspection.

Now go build!

— seb

Amazon CloudWatch Application Signals for automatic instrumentation of your applications (preview)

2023-11-30 Veliswa Boya

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/amazon-cloudwatch-application-signals-for-automatic-instrumentation-of-your-applications-preview/

One of the challenges with distributed systems is that they are made up of many interdependent services, which add a degree of complexity when you are trying to monitor their performance. Determining which services and APIs are experiencing high latencies or degraded availability requires manually putting together telemetry signals. This can result in time and effort establishing the root cause of any issues with the system due to the inconsistent experiences across metrics, traces, logs, real user monitoring, and synthetic monitoring.

You want to provide your customers with continuously available and high-performing applications. At the same time, the monitoring that assures this must be efficient, cost-effective, and without undiﬀerentiated heavy lifting.

Amazon CloudWatch Application Signals helps you automatically instrument applications based on best practices for application performance. There is no manual effort, no custom code, and no custom dashboards. You get a pre-built, standardized dashboard showing the most important metrics, such as volume of requests, availability, latency, and more, for the performance of your applications. In addition, you can define Service Level Objectives (SLOs) on your applications to monitor specific operations that matter most to your business. An example of an SLO could be to set a goal that a webpage should render within 2000 ms 99.9 percent of the time in a rolling 28-day interval.

Application Signals automatically correlates telemetry across metrics, traces, logs, real user monitoring, and synthetic monitoring to speed up troubleshooting and reduce application disruption. By providing an integrated experience for analyzing performance in the context of your applications, Application Signals gives you improved productivity with a focus on the applications that support your most critical business functions.

My personal favorite is the collaboration between teams that’s made possible by Application Signals. I started this post by mentioning that distributed systems are made up of many interdependent services. On the Service Map, which we will look at later in this post, if you, as a service owner, identify an issue that’s caused by another service, you can send a link to the owner of the other service to efficiently collaborate on the triage tasks.

Getting started with Application Signals
You can easily collect application and container telemetry when creating new Amazon EKS clusters in the Amazon EKS console by enabling the new Amazon CloudWatch Observability EKS add-on. Another option is to enable for existing Amazon EKS Clusters or other compute types directly in the Amazon CloudWatch console.

After enabling Application Signals via the Amazon EKS add-on or Custom option for other compute types, Application Signals automatically discovers services and generates a standard set of application metrics such as volume of requests and latency spikes or availability drops for APIs and dependencies, to name a few.

All of the services discovered and their golden metrics (volume of requests, latency, faults and errors) are then automatically displayed on the Services page and the Service Map. The Service Map gives you a visual deep dive to evaluate the health of a service, its operations, dependencies, and all the call paths between an operation and a dependency.

The list of services that are enabled in Application Signals will also show in the services dashboard, along with operational metrics across all of your services and dependencies to easily spot anomalies. The Application column is auto-populated if the EKS cluster belongs to an application that’s tagged in AppRegistry. The Hosted In column automatically detects which EKS pod, cluster, or namespace combination the service requests are running in, and you can select one to go directly to Container Insights for detailed container metrics such as CPU or memory utilization, to name a few.

Team collaboration with Application Signals
Now, to expand on the team collaboration that I mentioned at the beginning of this post. Let’s say you consult the services dashboard to do sanity checks and you notice two SLO issues for one of your services named pet-clinic-frontend. Your company maintains a set of SLOs, and this is the view that you use to understand how the applications are performing against the objectives. For the services that are tagged in AppRegistry all teams have a central view of the definition and ownership of the application. Further navigation to the service map gives you even more details on the health of this service.

At this point you make the decision to send the link to thepet-clinic-frontendservice to Sarah whose details you found in the AppRegistry. Sarah is the person on-call for this service. The link allows you to efficiently collaborate with Sarah because it’s been curated to land directly on the triage view that is contextualized based on your discovery of the issue. Sarah notices that the POST /api/customer/owners latency has increased to 2k ms for a number of requests and as the service owner, dives deep to arrive at the root cause.

Clicking into the latency graph returns a correlated list of traces that correspond directly to the operation, metric, and moment in time, which helps Sarah to find the exact traces that may have led to the increase in latency.

Sarah uses Amazon CloudWatch Synthetics and Amazon CloudWatch RUM and has enabled the X-Ray active tracing integration to automatically see the list of relevant canaries and pages correlated to the service. This integrated view now helps Sarah gain multiple perspectives in the performance of the application and quickly troubleshoot anomalies in a single view.

Available now
Amazon CloudWatch Application Signals is available in preview and you can start using it today in the following AWS Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Asia Pacific (Sydney), and Asia Pacific (Tokyo).

To learn more, visit the Amazon CloudWatch user guide. You can submit your questions to AWS re:Post for Amazon CloudWatch, or through your usual AWS Support contacts.

– Veliswa

New myApplications in the AWS Management Console simplifies managing your application resources

2023-11-30 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/new-myapplications-in-the-aws-management-console-simplifies-managing-your-application-resources/

Today, we are announcing the general availability of myApplications supporting application operations, a new set of capabilities that help you get started with your applications on AWS, operate them with less effort, and move faster at scale. With myApplication in the AWS Management Console, you can more easily manage and monitor the cost, health, security posture, and performance of your applications on AWS.

The myApplications experience is available in the Console Home, where you can access an Applications widget that lists the applications in an account. Now, you can create your applications more easily using the Create application wizard, connecting resources in your AWS account from one view in the console. The created application will automatically display in myApplications, and you can take action on your applications.

When you choose your application in the Applications widget in the console, you can see an at-a-glance view of key application metrics widgets in the applications dashboard. Here you can find, debug operational issues, and optimize your applications.

With a single action on the applications dashboard, you can dive deeper to act on specific resources in the relevant services, such as Amazon CloudWatch for application performance, AWS Cost Explorer for cost and usage, and AWS Security Hub for security findings.

Getting started with myApplications
To get started, on the AWS Management Console Home, choose Create application in the Applications widget. In the first step, input your application name and description.

In the next step, you can add your resources. Before you can search and add resources, you should turn on and set up AWS Resource Explorer, a managed capability that simplifies the search and discovery of your AWS resources across AWS Regions.

Choose Add resources and select the resources to add to your applications. You can also search by keyword, tag, or AWS CloudFormation stack to integrate groups of resources to manage the full lifecycle of your application.

After confirming, your resources are added, new awsApplication tags applied, and the myApplications dashboard will be automatically generated.

Now, let’s see which widgets can be useful.

The Application summary widget displays the name, description, and tag so you know which application you are working on. The Cost and usage widget visualizes your AWS resource costs and usage from AWS Cost Explorer, including the application’s current and forecasted month-end costs, top five billed services, and a monthly application resource cost trend chart. You can monitor spend, look for anomalies, and click to take action where needed.

The Compute widget summarizes of application compute resources, information about which are in alarm, and trend charts from CloudWatch showing basic metrics such as Amazon EC2 instance CPU utilization and AWS Lambda invocations. You also can assess application operations, look for anomalies, and take action.

The Monitoring and Operations widget displays alarms and alerts for resources associated with your application, service level objectives (SLOs), and standardized application performance metrics from CloudWatch Application Signals. You can monitor ongoing issues, assess trends, and quickly identify and drill down on any issues that might impact your application.

The Security widget shows the highest priority security findings identified by AWS Security Hub. Findings are listed by severity and service, so you can monitor their security posture and click to take action where needed.

The DevOps widget summarizes operational insights from AWS System Manager Application Manager, such as fleet management, state management, patch management, and configuration management status so you can assess compliance and take action.

You can also use the Tagging widget to assist you in reviewing and applying tags to your application.

Now available
You can enjoy this new myApplications capability, a new application-centric experience to easily manage and monitor applications on AWS. myApplications capability is available in the following AWS Regions: US East (Ohio, N. Virginia), US West (N. California, Oregon), South America (São Paulo), Asia Pacific (Hyderabad, Jakarta, Mumbai, Osaka, Seoul, Singapore, Sydney, Tokyo), Europe (Frankfurt, Ireland, London, Paris, Stockholm), Middle East (Bahrain) Regions.

AWS Premier Tier Services Partners— Escala24x7, IBM, Tech Mahindra, and Xebia will support application operations with complementary features and services.

Give it a try now in the AWS Management Console and send feedback to AWS re:Post for AWS Management Console, using the feedback link on the myApplications dashboard, or through your usual AWS Support contacts.

— Channy

Easily deploy SaaS products with new Quick Launch in AWS Marketplace

2023-11-30 Marcia Villalba

Post Syndicated from Marcia Villalba original https://aws.amazon.com/blogs/aws/easily-deploy-saas-products-with-new-quick-launch-in-aws-marketplace/

Today we are excited to announce the general availability of SaaS Quick Launch, a new feature in AWS Marketplace that makes it easy and secure to deploy SaaS products.

Before SaaS Quick Launch, configuring and launching third-party SaaS products could be time-consuming and costly, especially in certain categories like security and monitoring. Some products require hours of engineering time to manually set up permissions policies and cloud infrastructure. Manual multistep configuration processes also introduce risks when buyers rely on unvetted deployment templates and instructions from third-party resources.

SaaS Quick Launch helps buyers make the deployment process easy, fast, and secure by offering step-by-step instructions and resource deployment using preconfigured AWS CloudFormation templates. The software vendor and AWS validate these templates to ensure that the configuration adheres to the latest AWS security standards.

Getting started with SaaS Quick Launch
It’s easy to find which SaaS products have Quick Launch enabled when you are browsing in AWS Marketplace. Products that have this feature configured have a Quick Launch tag in their description.

After completing the purchase process for a Quick Launch–enabled product, you will see a button to set up your account. That button will take you to the Configure and launch page, where you can complete the registration to set up your SaaS account, deploy any required AWS resources, and launch the SaaS product.

The first step ensures that your account has the required AWS permissions to configure the software.

The second step involves configuring the vendor account, either to sign in to an existing account or to create a new account on the vendor website. After signing in, the vendor site may pass essential keys and parameters that are needed in the next step to configure the integration.

The third step allows you to configure the software and AWS integration. In this step, the vendor provides one or more CloudFormation templates that provision the required AWS resources to configure and use the product.

The final step is to launch the software once everything is configured.

Availability
Sellers can enable this feature in their SaaS product. If you are a seller and want to learn how to set this up in your product, check the Seller Guide for detailed instructions.

To learn more about SaaS in AWS Marketplace, visit the service page and view all the available SaaS products currently in AWS Marketplace.

— Marcia

Package and deploy models faster with new tools and guided workflows in Amazon SageMaker

2023-11-29 Antje Barth

Post Syndicated from Antje Barth original https://aws.amazon.com/blogs/aws/package-and-deploy-models-faster-with-new-tools-and-guided-workflows-in-amazon-sagemaker/

I’m happy to share that Amazon SageMaker now comes with an improved model deployment experience to help you deploy traditional machine learning (ML) models and foundation models (FMs) faster.

As a data scientist or ML practitioner, you can now use the new ModelBuilder class in the SageMaker Python SDK to package models, perform local inference to validate runtime errors, and deploy to SageMaker from your local IDE or SageMaker Studio notebooks.

In SageMaker Studio, new interactive model deployment workflows give you step-by-step guidance on which instance type to choose to find the most optimal endpoint configuration. SageMaker Studio also provides additional interfaces to add models, test inference, and enable auto scaling policies on the deployed endpoints.

New tools in SageMaker Python SDK
The SageMaker Python SDK has been updated with new tools, including ModelBuilder and SchemaBuilder classes that unify the experience of converting models into SageMaker deployable models across ML frameworks and model servers. Model builder automates the model deployment by selecting a compatible SageMaker container and capturing dependencies from your development environment. Schema builder helps to manage serialization and deserialization tasks of model inputs and outputs. You can use the tools to deploy the model in your local development environment to experiment with it, fix any runtime errors, and when ready, transition from local testing to deploy the model on SageMaker with a single line of code.

Let me show you how this works. In the following example, I choose the Falcon-7B model from the Hugging Face model hub. I first deploy the model locally, run a sample inference, perform local benchmarking to find the optimal configuration, and finally deploy the model with the suggested configuration to SageMaker.

First, import the updated SageMaker Python SDK and define a sample model input and output that matches the prompt format for the selected model.

import sagemaker
from sagemaker.serve.builder.model_builder import ModelBuilder
from sagemaker.serve.builder.schema_builder import SchemaBuilder
from sagemaker.serve import Mode

prompt = "Falcons are"
response = "Falcons are small to medium-sized birds of prey related to hawks and eagles."

sample_input = {
    "inputs": prompt,
    "parameters": {"max_new_tokens": 32}
}

sample_output = [{"generated_text": response}]

Then, create a ModelBuilder instance with the Hugging Face model ID, a SchemaBuilder instance with the sample model input and output, define a local model path, and set the mode to LOCAL_CONTAINER to deploy the model locally. The schema builder generates the required functions for serializing and deserializing the model inputs and outputs.

model_builder = ModelBuilder(
    model="tiiuae/falcon-7b",
    schema_builder=SchemaBuilder(sample_input, sample_output),
    model_path="/path/to/falcon-7b",
    mode=Mode.LOCAL_CONTAINER,
	env_vars={"HF_TRUST_REMOTE_CODE": "True"}
)

Next, call build() to convert the PyTorch model into a SageMaker deployable model. The build function generates the required artifacts for the model server, including the inferency.py and serving.properties files.

local_mode_model = model_builder.build()

For FMs, such as Falcon, you can optionally run tune() in local container mode that performs local benchmarking to find the optimal model serving configuration. This includes the tensor parallel degree that specifies the number of GPUs to use if your environment has multiple GPUs available. Once ready, call deploy() to deploy the model in your local development environment.

tuned_model = local_mode_model.tune()
tuned_model.deploy()

Let’s test the model.

updated_sample_input = model_builder.schema_builder.sample_input
print(updated_sample_input)

{'inputs': 'Falcons are',
 'parameters': {'max_new_tokens': 32}}
 
local_tuned_predictor.predict(updated_sample_input)[0]["generated_text"]

In my demo, the model returns the following response:

a type of bird that are known for their sharp talons and powerful beaks. They are also known for their ability to fly at high speeds […]

When you’re ready to deploy the model on SageMaker, call deploy() again, set the mode to SAGEMAKLER_ENDPOINT, and provide an AWS Identity and Access Management (IAM) role with appropriate permissions.

sm_predictor = tuned_model.deploy(
    mode=Mode.SAGEMAKER_ENDPOINT, 
	role="arn:aws:iam::012345678910:role/role_name"
)

This starts deploying your model on a SageMaker endpoint. Once the endpoint is ready, you can run predictions.

new_input = {'inputs': 'Eagles are','parameters': {'max_new_tokens': 32}}
sm_predictor.predict(new_input)[0]["generated_text"])

New SageMaker Studio model deployment experience
You can start the new interactive model deployment workflows by selecting one or more models to deploy from the models landing page or SageMaker JumpStart model details page or by creating a new endpoint from the endpoints details page.

The new workflows help you quickly deploy the selected model(s) with minimal inputs. If you used SageMaker Inference Recommender to benchmark your model, the dropdown will show instance recommendations from that benchmarking.

Without benchmarking your model, the dropdown will display prospective instances that SageMaker predicts could be a good fit based on its own heuristics. For some of the most popular SageMaker JumpStart models, you’ll see an AWS pretested optimal instance type. For other models, you’ll see generally recommended instance types. For example, if I select the Falcon 40B Instruct model in SageMaker JumpStart, I can see the recommended instance types.

However, if I want to optimize the deployment for cost or performance to meet my specific use cases, I could open the Alternate configurations panel to view more options based on data from before benchmarking.

Once deployed, you can test inference or manage auto scaling policies.

Things to know
Here are a couple of important things to know:

Supported ML models and frameworks – At launch, the new SageMaker Python SDK tools support model deployment for XGBoost and PyTorch models. You can deploy FMs by specifying the Hugging Face model ID or SageMaker JumpStart model ID using the SageMaker LMI container or Hugging Face TGI-based container. You can also bring your own container (BYOC) or deploy models using the Triton model server in ONNX format.

Now available
The new set of tools is available today in all AWS Regions where Amazon SageMaker real-time inference is available. There is no cost to use the new set of tools; you pay only for any underlying SageMaker resources that get created.

Learn more

Get started
Explore the new SageMaker model deployment experience in the AWS Management Console today!

— Antje

Monday, Dec 2	Tuesday, Dec 3	Wednesday, Dec 4
8:30 AM – 9:30 AM PST \| Caesars Forum ANT306-R \| Orchestrate data and ML workflows with managed Apache Airflow	12:00 PM – 1:00 PM PST \| Caesars Forum ANT307-R \| Seamless data sharing with Amazon Redshift	8:30 AM – 9:30 AM PST \| Caesars Forum ANT307-R3 \| Seamless data sharing with Amazon Redshift
5:30 PM – 6:30 PM PST \| Caesars Forum ANT306-R1 \| Orchestrate data and ML workflows with managed Apache Airflow	12:00 PM – 1:00 PM PST \| Wynn ANT401 \| Vector search with Amazon OpenSearch Service	.
.	1:30 PM – 2:30 PM PST \| Wynn ANT306-R2 \| Orchestrate data and ML workflows with managed Apache Airflow	.
.	1:30 PM – 2:30 PM PST \| Caesars Forum ANT307-R1 \| Seamless data sharing with Amazon Redshift	.
.	3:00 PM – 4:00 PM PST \| Caesars Forum ANT307-R2 \| Seamless data sharing with Amazon Redshift	.
.	4:30 PM – 5:30 PM PST \| Caesars Forum ANT306-R3 \| Orchestrate data and ML workflows with managed Apache Airflow	.

How to reserve a seat in the sessions

DevOps and Developer Productivity breakout sessions

What are breakout sessions?

DevOps and Developer Productivity chalk talks

What are chalk talks?

DevOps and Developer Productivity workshops

What are workshops?

DevOps and Developer Productivity builders’ sessions

What are builders’ sessions?

DevOps and Developer Productivity lightning talks

What are lightning talks?

DevOps and Developer Productivity code talks

What are code talks?

Want to stay connected?

What is cloud resilience all about?

Resilience fundamentals

Breakout sessions

Workshops

Advanced resilience patterns

Breakout sessions

Workshops

Resilience for customers operating in regulated industries

Breakout sessions

Workshops

Meet experts at the AWS Cloud Resilience kiosk

Keynotes

Analytics Innovation Talk

Breakout sessions

Chalk talks

Builders’ sessions

Workshops

Code talks

Conclusion

About the Authors

Keynotes and innovation talks

Sessions

Accelerating least privilege with advanced access analysis

Fortifying your security posture with threat detection and incident response

Securing the edge against evolving risks with confidence

Safeguarding sensitive data in the age of generative AI

Empowering developers with a security-minded culture

Expo

Still time for fun!

Register today

The Basic: Session types

Getting started with Amazon EC2

Learn about AWS compute innovations

Optimize your compute costs

Maximize you workload’s performance

Customer experiences and applications with machine learning

Accelerate your AWS Graviton adoption journey

Check out workload-specific sessions

Ready to unlock new possibilities?

Breakout sessions and lightning talks

Interactive sessions (chalk talks and workshops)

Keynotes and innovation talks

Sessions

Breakout sessions

Chalk talks

Code talks

Builders’ sessions

Workshops

Expo

Still time for fun!

Register today

Conclusion

Key announcements

Experience content on demand

Keynotes, innovation talks, and leadership sessions

Breakout sessions and new launch talks

Keynotes

Innovation Talks

Sessions

What’s new with Amazon Redshift

About the Author

The collective thoughts of the interwebz