Седмицата (10–15 март)

Post Syndicated from Боряна Телбис original https://www.toest.bg/sedmitsata-10-15-mart/

Седмицата (10–15 март)

За да тръгне някак този бюлетин, трябва да се хвана за нещо различно от пролетта навън (която в кошмарите на повечето хора напоследък рязко се превръща в ядрена зима, макар засега краят на света да се отлага, защото Доналд Тръмп съобщи в личната си социална мрежа, че май е успял да се договори с Путин за спиране на войната). Та това, за което се хващам, е философската концепция на Авишай Маргалит. Според нея прилично функциониращото общество е онова, в което гражданите не са унижавани от властта и също така не се унижават един друг. И като се хвана за тази концепция (понеже ми е проста и ясна, а на мен толкова ми трябва), почвам да я разнасям насам-натам като факел с надеждата да я видя някъде приложена. 

Ами не я виждам.

Накрая се оказва, че унижението върви на всякакви нива, и този факел рязко се превръща в къса клечка, която всички сме изтеглили колективно и само си я предаваме един на друг като щафета.

Тази седмица щафетата поеха в Конституционния съд, за да ни съобщят, че да, последните избори са скандално проведени и пълни с грешки – липсват бюлетини, но за сметка на това в чувалите има канцеларски материали, в случай че на някого му потрябва телбод; „отвлечени“ са цели секции; кочани с бюлетини са прегъвани накуп и подпечатвани на конвейер; на места бюлетини са попълвани от един и същи човек, за да е удобно на всички… 

Прилично функциониращото общество катастрофира с 200 км/ч и няма оцелели. 

Текст по темата в този брой на „Тоест“ има Емилия Милчева – „Властта се люлее, „Величие“ зрее“, от който може да разберете кой влиза в Народното събрание, кой излиза, с колко, защо, как и други важни работи, като например ще има ли „хубави неща за хората“, както обича да казва Делян Пеевски.

Хубави неща за хората специално от Делян Пеевски не знам дали да очакваме, но в „Тоест“ традиционно има хубави неща за четене. Започнахме седмицата с интервю на Ина Иванова с актрисата Боряна Йовчева, която истински харесваме и затова е част от рубриката ни „Тези хора“. 

Продължихме с констатацията на Лина Кривошиева, че всичко е политика, включително и на „Берлинале 2025“. В статията си за фестивала Лина споделя впечатленията си от четири отлични филма от Румъния, Сърбия, Китай и Лесото, които освен всичко, което имат да ни разкажат, ни показват, че светът живее със сходни грижи.

Сходни трябва да са грижите ни и по темата за образованието, защото качественото образование (наред с качественото здравеопазване) е единственото, което можем да завещаем на децата си. И вероятно в едно прилично функциониращо общество, в което гражданите не се унижават един друг, щеше да е точно така. Само че нашето не е такова и затова грижите ни по темата за образованието не просто не са сходни, а отиват в тотално различни посоки. И всичко е за сметка на децата. 

Учителските синдикати държат на въвеждането на задължителен час по религия и оценка за дисциплина. А родителите се борят постоянно с манталитета „тука е така“ и накрая вече дори нямат кой знае какви претенции, само ако може в часовете по английски между 5. и 7. клас децата да не учат отново наученото в частните уроци и курсове, на които повечето от тях ходят още от предучилищната. Защо изучаването на английски език в прогимназиалните класове е толкова неефективно и какво може да се направи, обобщава Донка Дойчева-Попова в статията си „Чуждоезиковото обучение в училище – максимално неефективно, за да е удобно на всички“.

Тази седмица имаме текст и от молекулярната биоложка Анастасия Орманджиева за човешкия папиломавирус (HPV). Анастасия разказва какво представлява този вирус, откога ни е известен и как го лекуваме в наши дни. Част от щамовете му са високорискови и са отговорни за рака на маточната шийка, както и за рака на някои други органи. Де да имаше ваксина за този вирус… А, чакайте! Има! Но ваксинационното ниво в България доскоро беше 3%. Само за сравнение – с най-висок дял на ваксиниране са в Туркменистан и Узбекистан (над 90%), по данни на Protect Europe. 

Оставям ви с тази жизнерадостна информация и с епизод 6 от рубриката ни „Т.Е. от Е.Т.“, в който всичко е 6, разбира се. Както навсякъде.

Ако искате да продължим да се радваме на живота заедно, подкрепете „Тоест“, за да ни е все така бодро и ведро. Всъщност вашата подкрепа понякога изглежда като единствения факел, който никога не се превръща в къса клечка, а продължава да свети в тъмното.

Подкрепете ни

Implementing a serverless architecture to detect absence of Guardrails in Amazon Bedrock inference API calls

Post Syndicated from Art Baudo original https://aws.amazon.com/blogs/compute/implementing-a-serverless-architecture-to-detect-absence-of-guardrails-in-amazon-bedrock-inference-api-calls/

This post is written by Sayan Chakraborty, Senior Solutions Architect, AWS

Implementing a serverless architecture to detect absence of Guardrails in Amazon Bedrock inference API calls

In today’s rapidly evolving artificial intelligence (AI) landscape, organizations are increasingly harnessing the power of foundation models through Amazon Bedrock to build sophisticated generative AI applications. Although this technology opens up exciting possibilities, it also brings forth important considerations around responsible AI implementation and content safety.

Amazon Bedrock Guardrails serve as a crucial safeguard, helping organizations filter out harmful content, prevent prompt injection attacks (LLM01:2025 from OWASP Top 10 for generative AI), and maintain ethical AI practices. These configurable safeguards are essential for enterprises committed to responsible AI development, especially when scaling their applications across various use cases.

However, there’s a critical consideration: although Guardrails are powerful, they’re optional by default in Amazon Bedrock inference API calls. For organizations that mandate the use of Guardrails as part of their responsible AI strategy, a solution is needed to make sure of consistent implementation across all API requests.

In this post, we explore how to build a serverless architecture that automatically detects when Guardrails are absent in Amazon Bedrock inference API calls. We demonstrate how enterprises can implement automated monitoring and alerting systems to maintain compliance with their AI safety standards, making sure that Guardrails are properly implemented wherever needed. This solution is particularly valuable for organizations prioritizing secure and responsible AI deployment at scale.

Prerequisites

Before proceeding with the implementation, make sure that you do the following:

1.Create an AWS account if you do not already have one, and log in. The AWS Identity and Access Management (IAM) user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.

2.Have AWS Command Line Interface (AWS CLI) installed and configured.

3.Have Git Installed.

Architecture

The following diagram shows an event-driven architecture of this solution.

Figure 1: Solution architecture diagram

Figure 1: Solution architecture diagram

Amazon Bedrock supports model invocation logging. When enabled, it collects the full request data, response data, and metadata associated with all model invocation calls performed in your AWS account. Logging can be configured to send the logs to supported destinations such as Amazon CloudWatch Logs and Amazon S3. This solution uses an S3 bucket to collect these logs. Note that this solution supports the below Amazon Bedrock inference APIs:

As logs get stored in the S3 bucket, an Amazon S3 event notification is generated to an Amazon EventBridge event bus. A rule that matches “Object Created” events from Amazon S3 routes these events to an AWS Step Functions state machine, which defines the orchestration logic to inspect the model invocation logs for missing Guardrails, and sends out an alert to a monitored email address when applicable.

Walkthrough of the orchestration

As mentioned previously, the Step Functions state machine is the orchestration engine that performs the business logic for this solution, as events are received from new logs created in the S3 bucket. When opened in the Workflow studio in the Step Functions console, you should observe the following diagram.

Figure 1: Step Functions state machine diagram as seen in workflow studio

Figure 2: Step Functions state machine diagram as seen in workflow studio

1.The first step in the state machine is to call an AWS Lambda function to get the logs from the S3 bucket using the bucket name and object key supplied in the event object received from EventBridge.

2.If the log shows that the Amazon Bedrock API invocation was successful, then the state machine collects the output object of the API response from the log that is needed for further evaluation.

3.The next step is to check if Amazon Bedrock Guardrails was used. This is done by looking for specific objects in the Amazon Bedrock API output that was captured from the logs.

4.If a Guardrail was detected, then the flow completes successfully, and no further action is needed.

5.If a Guardrail was not detected, then the next step in the state machine collects a few pieces of information from the log file that is necessary to record the transaction and adds the transaction date. Then, the transaction is logged in to the transactions table in Amazon DynamoDB.

6.A user or a role may be making a lot of API calls to Amazon Bedrock each day. Therefore, the solution implements a mechanism to prevent the monitored email address from being swamped by emails reporting the same user or role more than once each day. This is done in parallel to Step 5, where the flow checks if the principal’s identity (user ID/IAM role) is recorded as notified in the current date, by querying the notifications table in DynamoDB. If no results were found, meaning that a notification hasn’t been sent yet, then an email is sent out to a monitored email address through an Amazon Simple Notification Service (Amazon SNS) topic. Furthermore, an item is inserted into the notifications table in DynamoDB to prevent sending more notifications on the same day for the same principal.

Solution deployment

For deployment instructions, follow along in the GitHub repo or use this post. An AWS CloudFormation template is provided to deploy the solution.

1.Create an S3 bucket to store the model invocation logs from Amazon Bedrock. Under bucket Properties, turn the EventBridge notifications to On. This enables Amazon S3 to send an event notification to the EventBridge default event bus whenever a log file is created in the bucket by Amazon Bedrock.

2.Go to the Amazon Bedrock console and enable Model invocation logging under Bedrock Configuration > Settings, from the left navigation pane. Specify the bucket created in Step 1 under S3 location.

Figure 2: Amazon Bedrock settings for Model invocation logging

Figure 3: Amazon Bedrock settings for Model invocation logging

3. Create two more S3 buckets: one that is used by the Step Functions state machine to store Bedrock model invocation errors detected from the log, and the other that stores the Lambda function code for this solution. Inside the latter bucket, create a Folder called code (or any other preferred name) and upload the ZIP archive under the lambda-code folder of this repository, into that Amazon S3 folder. Note the names for these two S3 buckets and the Amazon S3 object key for the Lambda ZIP file. These must be specified as input parameters to the CloudFormation template.

4. From the CloudFormation console or using CLI, create a stack using the template provided in this repository called bedrock-guardrails-detection-template.yaml. For inputs, specify the BedrockLogsBucket (from Step 1), BedrockLogsErrorBucket (from Step 3), LambdaFunctionCodeBucket (from Step 3), LambdaFunctionCodeBucketKey (S3 object key for the ZIP file uploaded in Step 3, for example code/get-bedrock-logs-from-s3.py.zip), and NotificationEmailAddress (email address to subscribe to the SNS topic). It may take a few minutes to complete deployment of the CloudFormation stack.

5. When deployment is complete, access the email inbox for the email address specified during the CloudFormation stack deployment, and confirm the subscription using the email sent from the Amazon SNS topic. The email should be titled: AWS Notification – Subscription Confirmation. Choose the Confirm subscription link inside the email to complete the subscription process. The email account is now ready to receive notifications from this solution.

Scaling to multiple AWS accounts

The architecture discussed previously shows how Guardrails can be detected from within the same AWS account where Amazon Bedrock APIs are invoked. However, in most production environments, there are multiple AWS accounts where independent teams may be deploying their own generative AI workloads using Amazon Bedrock in their own accounts. To collect model invocation logs from all those accounts, EventBridge can be configured to send events from event buses in separate source workload accounts to a central event bus deployed in a central destination governance account. This central event bus can have a rule to route events to the Step Functions state machine deployed in that central governance account. The deployment model looks like the following diagram.

To learn more about sending and receiving events between AWS accounts in EventBridge, refer to the documentation.

Figure 3: Cross-account guardrail detection solution

Figure 4: Cross-account guardrail detection solution

Further considerations and clean up

Amazon Bedrock model invocation logging captures requests and responses from model invocations and stores the logs in the destination of your choosing. In this sample it is in an S3 bucket that you create. The following are some more security considerations.

1.To protect information, you may choose to use to encrypt the contents using server-side encryption with AWS KMS keys (SSE-KMS) on the S3 bucket, and specify a customer managed encryption key. More details are in this Amazon Bedrock user guide.

2.Perform regular cleanup of the model invocation logs bucket using an Amazon S3 lifecycle configuration rule as mentioned in this post.

To avoid ongoing charges, clean up your environment by following these steps to delete the resources you created by following this post, if they are no longer needed:

1.Delete the stack:
aws cloudformation delete-stack –stack-name STACK_NAME

2.Confirm the stack has been deleted:
aws cloudformation list-stacks –query “StackSummaries[?contains(StackName,’STACK_NAME’)].StackStatus”

3.Empty contents of the S3 buckets created manually as a prerequisite to deploying the CloudFormation stack and delete the buckets.

4.Turn off model invocation logging from under Settings in the Amazon Bedrock console, if it’s not desired any longer.

Conclusion

This post discussed implementing a serverless event-driven architecture to detect the absence of Guardrails in Amazon Bedrock inference API calls. As organizations increasingly use foundation models through Amazon Bedrock for generative AI applications, making sure of responsible AI implementation becomes crucial.

The solution presents an event-driven architecture that automatically detects when Guardrails are missing in API calls. It uses the Amazon Bedrock model invocation logging, storing logs in an Amazon S3 bucket. When new logs are created, an Amazon S3 event notification triggers an Amazon EventBridge event bus, which routes events to an AWS Step Functions state machine. Then, the state machine inspects the logs for missing Guardrails and sends alerts through Amazon SNS to a monitored email address.

The architecture includes features to prevent notification flooding and can scale across multiple AWS accounts. The post provides detailed deployment instructions using AWS CloudFormation and includes security considerations and cleanup procedures. With this solution you can help your organization maintain compliance with AI safety standards while scaling generative AI applications.

Friday Squid Blogging: SQUID Band

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/03/friday-squid-blogging-squid-band.html

A bagpipe and drum band:

SQUID transforms traditional Bagpipe and Drum Band entertainment into a multi-sensory rush of excitement, featuring high energy bagpipes, pop music influences and visually stunning percussion!

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Metasploit Weekly Wrap-Up 03/14/25

Post Syndicated from Jacquie Harris original https://blog.rapid7.com/2025/03/14/metasploit-weekly-wrap-up-03-14-25/

New module content (1)

InvoiceShelf unauthenticated PHP Deserialization Vulnerability

Metasploit Weekly Wrap-Up 03/14/25

Authors: Mickaël Benassouli, Rémi Matasse, and h00die-gr3y
Type: Exploit
Pull request: #19950 contributed by h00die-gr3y
Path: linux/http/invoiceshelf_unauth_rce_cve_2024_55556
AttackerKB reference: CVE-2024-55556

Description: Deserialization module for CVE-2024-55556, exploiting unauthenticated PHP deserialization vulnerability in InvoiceShelf.

Bugs fixed (3)

  • #19937 from fabpiaf – Fixes a crash when a running HTTP server attempted to perform HTML escaping.
  • #19944 from Takahiro-Yoko – Enhancing existing module for CVE-2025-0655 by adding dynamically generated session for bypassing authentication.
  • #19955 from zeroSteiner – Updates the way we tag URLs in gather/ldap_esc_vulnerable_cert_finder to better support vulnerability reporting.

Documentation

You can find the latest Metasploit documentation on our docsite at docs.metasploit.com.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
commercial edition Metasploit Pro

NEVER MISS AN EMERGING THREAT

Be the first to learn about the latest vulnerabilities and cybersecurity news.

How to Register a Sender ID Using APIs with AWS End User Messaging

Post Syndicated from Tyler Holmes original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-register-a-sender-id-using-apis-with-aws-end-user-messaging/

Introduction

Welcome to our comprehensive guide on using the AWS End User Messaging V2 APIs to obtain a Sender ID in countries that require registration. Sender ID registration is a crucial step for businesses looking to establish a trusted communication channel with their customers via SMS. In countries such as Jordan, the Philippines, Qatar, and others registering a Sender ID is mandatory to ensure compliance with local regulations and to enhance message deliverability. You can view a list of countries that support Sender ID and require registration here.

NOTE: This post is specific to Sender IDs, to learn more about choosing the right originator for your use case read this post on How to Send SMS Globally Using AWS End User Messaging.

This guide will walk you through the high-level process of creating and submitting a Sender ID registration using AWS End User Messaging V2 APIs. While we will use Jordan as a case study in this post, the principles and steps outlined here apply to other countries with similar registration requirements for Sender ID.

High-Level Understanding of the Registration APIs

To successfully register a Sender ID, you’ll need to interact with several AWS End User Messaging APIs. Here’s a brief overview of the key APIs involved and their roles in the registration process:

1. DescribeRegistrationTypeDefinitions

  • Purpose: Retrieves the specified registration type definitions.
  • Use Case: View the requirements for creating, filling out, and submitting each registration type.
  • Key Point: Each country has a specific “RegistrationType” that needs to be used in the next step to create a registration.
  • Documentation: DescribeRegistrationTypeDefinitions

2. CreateRegistration

  • Purpose: Creates a new registration “container”. This is an empty registration that we are going to fill with our data in later steps.
  • Key Field: RegistrationType
    • Controls the type of registration (e.g., Toll-Free, 10DLC, JO_SENDER_ID_REGISTRATION, etc.).
  • Documentation: CreateRegistration

3. DescribeRegistrationFieldDefinitions

  • Purpose: Retrieves the specified RegistrationType field definitions.
  • Use Case: View the requirements for creating, filling out, and submitting each registration type. There are different field types (SELECT, TEXT, ATTACHMENT) dependent on the type of data required to register as well as the actual questions that need to be answered.
  • Documentation: DescribeRegistrationFieldDefinitions

4. CreateRegistrationAttachment

  • Purpose: Uploads additional documentation that some countries require. Not all countries require documentation that needs to be uploaded so this may be an optional step depending on the country you are registering for. We will be registering for Jordan in this example which does require us to upload a document to complete the registration.
  • Key Field: RegistrationAttachmentId
    • Used in the next action to include an attachment in the registration when you are required to provide one. Not all registration types require an attachment.
  • File Specifications:
    • Maximum file size: 500kb
    • Valid file extensions: PDF, JPEG, or PNG
  • Documentation: CreateRegistrationAttachment

5. PutRegistrationFieldValue

  • Purpose: Sets the value for a specific registration field.
    • This action needs to be repeated for all required fields.
  • Documentation: PutRegistrationFieldValue

6. SubmitRegistrationVersion

  • Purpose: Submits the specified registration version for review and approval. You are able to create new versions of the registration using CreateRegistrationVersion so make sure if you created another Registration Version that you use the correct ID.
  • Important Notes:
    • Ensure that you are using the correct Version if you created multiple
    • Ensure all data is correct before submission.
    • Initial status will be “CREATED” and should change to “REVIEWING” within 24 hours.
    • You cannot edit or delete the registration until it is approved or rejected.
  • Documentation: SubmitRegistrationVersion

Detailed Steps for Sender ID Registration in Jordan

In the following sections, we’ll dive deeper into each of these steps, providing more detail on the API calls and responses as well as best practices to ensure a smooth registration process. Whether you’re registering in Jordan or another country with similar requirements, this guide will equip you with the knowledge you need to successfully register a Sender ID.

Step 1: DescribeRegistrationTypeDefinitions

The first step is to run DescribeRegistrationTypeDefinitions to retrieve all specified registration type definitions. This API provides a detailed response that includes various attributes related to the registration process. Let’s break down these attributes for Jordan:

  • Country: JO
    • Description: Specifies the country code for which the registration type definitions are being described. JO stands for Jordan.
    • Implication: Understanding the country code is crucial because registration requirements and processes can vary significantly from one country to another.
  • ResourceType: SENDER_ID
    • Description: Indicates the type of resource for which the registration is being defined.
    • Implication: Knowing that the resource type is SENDER_ID helps you understand that the registration process is focused on obtaining approval specifically for a Sender ID as there are some countries that have multiple registration types.
  • Registration Type: JO_SENDER_ID_REGISTRATION
    • Description: Specifies the exact type of registration required for the given country and resource type.
    • Implication: This registration type is critical because it dictates the specific steps, documentation, and criteria you need to meet to successfully register a Sender ID in Jordan.
    • You will use this value to create the initial registration
  • Association Behavior: ASSOCIATE_ON_APPROVAL
    • Description: Describes how the registered Sender ID will be associated with your AWS account once the registration is approved.
    • Implication: ASSOCIATE_ON_APPROVAL means that the Sender ID will be automatically associated with your account as soon as the registration is approved, simplifying the process post-approval. There will be nothing more for you to do once you have successfully submitted your registration and it has been approved. Some registration types require steps to be taken after the registration is approved, this is not one of them.
  • Disassociation Behavior: DISASSOCIATE_ALL_CLOSES_REGISTRATION
    • Description: Explains what happens when you disassociate the Sender ID from your account.
    • Implication: This behavior is important to note because it means that the origination identity must be disassociated from the registration before the registration can be closed.

Step 2: CreateRegistration

Next, use CreateRegistration with the RegistrationType for Jordan, “JO_SENDER_ID_REGISTRATION,” to generate a blank registration “container” for a Jordan Sender ID. Make sure to log the “RegistrationId” for later use as it will be used throughout the rest of the process.

Below is a screenshot of the registration that was created in the console

Step 3: DescribeRegistrationFieldDefinitions

Use DescribeRegistrationFieldDefinitions with “JO_SENDER_ID_REGISTRATION” as the RegistrationType to pull down all the fields that need to be filled out to submit the registration.

Important Field Attributes

  • FieldPath
    • Value: A string that specifies the path to the field within the registration form (e.g., companyInfo.companyName).
    • Description: The FieldPath attribute provides a hierarchical path to the specific field within the registration form. It identifies one of the fields we will use later to input the data into our registration
    • Implication: Ensures that you place the required information in the correct field, essential for accurate processing.
  • FieldRequirement
    • Value: Indicates whether the field is required, optional, or conditional (REQUIRED, OPTIONAL, CONDITIONAL).
    • Description: The FieldRequirement attribute specifies the necessity of the field in the registration process. A REQUIRED field must be filled out for the registration to be submitted, while an OPTIONAL field can be left blank if not applicable. A CONDITIONAL field depends on other fields and may become required based on certain conditions.
    • Implication: Helps prioritize which fields to complete and ensures critical information is not missed.
  • FieldType
    • Value: Describes the type of data expected in the field (SELECT, TEXT, ATTACHMENT).
    • Description: Indicates the format and type of data that should be entered into the field.
    • Implication: Ensures the registration form is valid and reduces the risk of errors or rejections due to incorrect data formats.

See the categories of the Jordan registration in the screenshot below

Let’s break down the required fields that are common across all Sender Id registrations and their descriptions in a more readable format:

Sender ID Information:

  1. Sender ID (Required)
    1. Must be 3-11 alphanumeric characters
    2. Must contain at least one letter
    3. Example: “EXAMPLE”
  2. Sender ID Description (Optional)
    1. Explain the connection between company name and sender ID
    2. Max 500 characters
  3. Proof of Sender ID Connection (Optional)
    1. Required only if the connection between company name and sender ID isn’t obvious
    2. Must provide evidence of intellectual property rights

Below is a screenshot of the Console version

Company Information:

  1. Company Name (Required)
    1. Legal company name
    2. Max 100 characters
    3. Example: “Example Corp”
  2. Company ID (Required)
    1. Legal identification number (EIN/VAT)
    2. Alphanumeric only
    3. Max 30 characters
  3. DBA Name (Optional)
    1. “Doing Business As” name if different from legal name
    2. Max 100 characters
  4. Website (Required)
    1. Company’s full URL
    2. Example: “https://www.example.com”
  5. Area of Business (Required)
    1. Choose one: Agriculture, Communication, Construction, Education, Energy, Entertainment, Financial, Government, Healthcare, Hospitality, Insurance, Manufacturing, Real estate, Retail, Technology, Other

Below is a screenshot of the Console version

Company Address:

  1. Address Line 1 (Required)
  2. Address Line 2 (Optional)
  3. City (Required)
  4. State/Province (Optional)
  5. Postal Code (Optional)
  6. Country Code (Required)
    1. Two-letter ISO code
    2. Example: “US”

Below is a screenshot of the Console version

Contact Information:

  1. Email Address (Required)
    1. Valid email format
    2. Example: “[email protected] “
  2. Phone Number (Required)
    1. Must contain at least 3 digits
    2. Example: “+12065550100“

Below is a screenshot of the Console version

Messaging Use Case:

  1. Use Case Category (Required)
    1. Choose one: One-time passcodes, Account/security alerts, Purchase/delivery notifications, Public service announcements, Polling/surveys, Info on demand, Other
  2. Use Case Description (Required)
    1. Max 500 characters
  3. Monthly Message Volume (Required)
    1. Choose one: 10, 100, 1,000, 10,000, 100,000, 1,000,000, 10,000,000+
  4. Opt-in Description (Required)
    1. How users will opt-in to receiving messages
    2. Max 500 characters

Below is a screenshot of the Console version

Message Samples:

  1. Message Sample 1 (Required)
    1. Max 306 characters
  2. Message Sample 2 (Optional)
    1. Max 306 characters
  3. Message Sample 3 (Optional)
    1. Max 306 characters

Below is a screenshot of the Console version

Jordan-Specific Requirements

Some countries, not all, also have country-specific requirements. Jordan requires the following for this registration:

  1. Company Registration Documentation (Required)
    1. Must provide company registration docs (both local and international companies)
    2. This is an Attachment type so we will need to use “CreateRegistrationAttachment” first in order to put this value into the registration. We will cover this further down in the process
  2. Transactional Acknowledgement (Required)
    1. Must acknowledge that only transactional messages will be sent
    2. Promotional content is not allowed
    3. This is a “Select” field type with the only option being “Yes” meaning you will be unable to register unless you agree to this acknowledgement

Below is a screenshot of the Console version

Step 4: CreateRegistrationAttachment

Now that you know all the fields required for the Jordan Sender ID registration, you need to gather all the necessary data, including the required attachment. The Jordan Sender ID registration requires only one attachment. Let’s review how to add this attachment to your registration.

To add an attachment to your registration, use the CreateRegistrationAttachment API. You have two options for providing this document:

  1. Specify a file in S3:
    1. Ensure you use Standard buckets, S3 Express is not supported.
  2. Upload it as a Base64-encoded binary data object:
    1. This method allows you to directly upload the document without needing to store it in S3 first.

Important: After successfully running the CreateRegistrationAttachment command, you will receive a “RegistrationAttachmentId.” Make sure to log this ID, as you will need it to attach the upload to your registration.

Step 5: PutRegistrationFieldValue

Use PutRegistrationFieldValue to input data into each field. Loop through each input as each call only inputs a single value. Specify the “RegistrationId” logged earlier when making each call. Your options for field values are:

  • RegistrationAttachmentId: The unique identifier for the registration attachment
  • SelectChoices: An array of values for the form field that were specified in the payload earlier
  • TextValue: The text data for a free form field

Step 6: Review your registration

Once you have successfully inputted all the values for your registration, you can review your inputs in one of two ways:

  1. View in the Console:
    1. Your registration details are accessible directly in the AWS Management Console.
  2. Use DescribeRegistrationFieldValues:
    1. You can pull down your registration and review each of its values using the DescribeRegistrationFieldValues API. Make sure to use the right version ID of your registration.

Step 7: Submitting Your Registration

Once all values in your registration are complete and correct, submit your registration for review using the SubmitRegistrationVersion API. Provide your “RegistrationId” when making this call.

Initial Status:

  • The initial status of your registration will be “CREATED.”
  • It should change to “REVIEWING” within 24 hours.
  • If the status does not change from “CREATED” to “REVIEWING” within 24 hours and you’ve confirmed you SUBMITTED your registration version, create a support case for assistance.

Sender ID Registration Review Process

Each country has its own review process and timeline. Each registration is examined on a first-in/first-out basis by the registrar for each country. AWS does not review your registrations so make sure that you are completing this registration with this in mind. There are humans reviewing these registrations that do not know your company or your use case so make sure to be clear and concise in your responses.

If the registrar is confused by your submission, they may reject it, and you will need to start the process over, amending the registration where necessary and resubmitting. This puts you in the back of the line again, which will increase your timeline.

Conclusion

Completing a Sender ID registration using the End User Messaging V2 APIs involves several steps, each critical to ensuring compliance and successful registration. By following this guide, you can navigate the process efficiently, from understanding the required fields to submitting your registration for review. Whether you’re registering in Jordan or another country with different requirements, this guide provides the knowledge and steps needed to successfully obtain a Sender ID and establish a trusted communication channel with your customers via SMS.

Upcoming Speaking Engagements

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/03/upcoming-speaking-engagements-44.html

This is a current list of where and when I am scheduled to speak:

The list is maintained on this page.

Architect fault-tolerant applications with instance fleets on Amazon EMR on EC2

Post Syndicated from Deepmala Agarwal original https://aws.amazon.com/blogs/big-data/architect-fault-tolerant-applications-with-instance-fleets-on-amazon-emr-on-ec2/

Organizations rely on Amazon EMR on EC2 clusters to process large-scale data workloads using frameworks like Apache Spark, Apache Hive, and Trino. Events such as TV advertisements or unplanned promotions might lead to an increase in demand of compute capacity, making effective capacity planning necessary to make sure your workloads don’t hit capacity limits or job failures.

A common scenario is to run daily Spark jobs on Amazon EMR using consistent Amazon Elastic Compute Cloud (Amazon EC2) instance types (for example, a single instance size and family for the cluster). Although this might work well to sustain the baseline, spikes can trigger auto scaling, which narrows the chances of capacity availability when trying to stop and relaunch a larger EMR cluster, because the specific on-demand instance pool might lack capacity to meet the demand.

In this post, we show how to optimize capacity by analyzing EMR workloads and implementing strategies tailored to your workload patterns. We walk through assessing the historical compute usage of a workload and use a combination of strategies to reduce the likelihood of InsufficientCapacityExceptions (ICE) when Amazon EMR launches specific EC2 instance types. We implement flexible instance fleet strategies to reduce dependency on specific instance types and use Amazon EC2 On-Demand Capacity Reservation (ODCRs) for predictable, steady-state workloads. Following this approach can help prevent job failures due to capacity limits while optimizing your cluster for cost and performance.

Solution overview

Instance fleets in Amazon EMR offer a flexible and robust way to manage EC2 instances within your cluster. This feature allows you to specify target capacities for On-Demand and Spot Instances, select up to five EC2 instance types per fleet (or 30 when using the AWS Command Line Interface [AWS CLI] and API with an allocation strategy), and use multiple subnets across different Availability Zones. Importantly, instance fleets support the use of ODCRs, enabling you to align your EMR clusters with pre-purchased EC2 capacity. You can configure your instance fleet to prefer or require capacity reservations, making sure that your EMR clusters use your reserved capacity efficiently.

EMR workload patterns typically fall into two categories: stable and variable (spiky). In the following sections, we explore how to optimize for each pattern using various options available with instance fleets, starting with stable workloads and then addressing variable workloads.

Stable workloads are workloads with a predictable pattern of resource utilization over time; for example, a pharmaceutical provider needs to process 21 TB of research data, patient records, and other information daily. The workload is consistent and needs to run reliably every day on long-running persistent clusters. For critical business operations requiring high reliability and guaranteed capacity, we recommend reserving the baseline capacity as part of your capacity planning. We demonstrate the following steps:

  1. Use AWS Cost and Usage Reports (AWS CUR) to estimate the baseline of existing workloads.
  2. Reserve the baseline capacity using ODCR.
  3. Configure Amazon EMR to use the targeted ODCR.

Spiky workloads are defined by unpredictable and often significant fluctuations in processing demands. These surges can be triggered by various factors (such as batch processing, real-time data streaming, or seasonal business fluctuations) that trigger Amazon EMR to request more capacity to match the demand. We address the resource allocation by using instance and Availability Zone flexibility, with the following steps:

  1. Introduce EC2 instance flexibility with EMR instance fleets.
  2. Achieve resiliency through intelligent subnet selection with EMR instance fleets.
  3. Use managed scaling to automatically manage scaling in and out.

Stable workloads

In this section, we demonstrate how to define your baseline, configure AWS Identity and Access Management (IAM) permissions, create an ODCR, and associate your reservations to a capacity group and configure Amazon EMR to use targeted ODCRs. You can opt for a mixed ODCR strategy—for example, one ODCR with a short period of duration that supports the launch of your EMR cluster, and another ODCR with a longer period of duration that supports your task nodes based on the baseline capacity reservation.

Estimate the baseline

Make sure to activate the AWS generated cost allocation tag aws:elasticmapreduce:job-flow-id. This enables the field resource_tags_aws_elasticmapreduce_job_flow_id in the AWS CUR to be populated with the EMR cluster ID and is used by the SQL queries in the solution. To activate the cost allocation tag from the AWS Billing Console, complete the following steps:

  1. On the AWS Billing and Cost Management console, choose Cost allocation tags in the navigation pane.
  2. Under AWS generated cost allocation tags, choose the aws:elasticmapreduce:job-flow-id tag.
  3. Choose Activate.

It can take up to 24 hours for tags to activate. For more information, see here.

After the tags are activated, you can use AWS CUR and perform the following query on Amazon Athena to find the compute resources used by the EMR cluster ID vs. the timeline of usage. For more details, see Querying Cost and Usage Reports using Amazon Athena. Update the following query with your CUR table name, EMR cluster ID, desired timestamps, and AWS account ID, and run the query on Athena:

SELECT bill_payer_account_id as Payer,
    product_product_family as PFamily,
    product_product_name as PName,
    resource_tags_aws_elasticmapreduce_job_flow_id,
    line_item_usage_account_id as LinkedAccount,
    line_item_usage_start_date as UsageDate,
    bill_billing_period_start_date as BillingDate,
    SPLIT_PART(line_item_usage_type, ':', 2) AS InstanceType,
    line_item_availability_zone AS AvailabilityZone,
    COUNT(line_item_resource_id) as ResourceIDCount
FROM <YOUR_CUR_TABLE_NAME>
WHERE (
        line_item_usage_start_date BETWEEN TIMESTAMP 'YYYY-MM-DD 00:00:00'
        AND TIMESTAMP 'YYYY-MM-DD 23:59:59' 
    )
    AND line_item_operation LIKE '%%RunInstance%%'
    AND line_item_line_item_type LIKE '%%Usage%%'
    AND product_product_family NOT IN ('Data Transfer')
    AND resource_tags_aws_elasticmapreduce_job_flow_id LIKE '%%<emr-cluster-id>%%'
    AND line_item_usage_account_id IN (
        '<aws_account_id>'
)
GROUP BY 1,2,3,4,5,6,7,8,9

As an example, the preceding query filters instances usage per hour for a given account and EMR cluster for the period of 6 months, to generate the following figure. You can export the results in CSV format and analyze the data. Now that you have a visual representation of your workloads’ baseline and bursts, you can define the strategy and configuration of your EMR cluster.

Create an ODCR to reserve the baseline capacity

ODCRs can be either open or targeted:

  • With an open ODCR, new instances and existing instances that have matching attributes (such as operating system or instance type) will run using the capacity reservation attributes first.
  • With a targeted ODCR, instances must match the attributes of the ODCR specification and the ODCR is specifically targeted at launch. This approach is recommended if you have multiple concurrent EMR clusters consuming capacity from the shared On-Demand pool of EC2 instances. EMR clusters larger than the targeted ODCR quantity will fall back to On-Demand Instances that are in the same Availability Zone.

In this example, we use a targeted ODCR with an EMR instance fleet in the us-east-1a Availability Zone. The following diagram illustrates the workflow.

Complete the following steps:

  1. Use the create-capacity-reservation AWS CLI command to create the ODCR and make a note of the CapacityReservationArn value in the output:
aws ec2 create-capacity-reservation \
     --availability-zone <Input Your Availability Zone> \
     --instance-type r8g.2xlarge \
     --instance-match-criteria targeted \
     --instance-platform Linux/UNIX \
     --instance-count <enter the number of instances out of your baseline estimation>

We get the following output:

{
     "CapacityReservation": {
         "CapacityReservationId": "cr-0123456f9907xxxxx",
         "OwnerId": "XXXX",
         "CapacityReservationArn": "arn:aws:ec2:us-east-1:XXXX:capacity-reservation/cr-0123456f9907xxxxx",
         "InstanceType": "r8g.2xlarge",
         "InstancePlatform": "Linux/UNIX",
         "AvailabilityZone": "us-east-1a"

 ....
     }
 }

You can use Amazon CloudWatch to monitor ODCR usage and trigger an alert for unused capacity. For more details, see Monitor Capacity Reservations usage with CloudWatch metrics.

  1. Create a resource group named EMRSparkSteadyStateGroup and make a note of GroupArn values in the output:
aws resource-groups create-group --name EMRSparkSteadyStateGroup \
--configuration '{"Type":"AWS::EC2::CapacityReservationPool"}' '{"Type":"AWS::ResourceGroups::Generic", "Parameters":[{"Name":"allowed-resource-types","Values":["AWS::EC2::CapacityReservation"]}]}'

We get the following output:

"Group": {
         "GroupArn": "arn:aws:resource-groups:us-east-1:XXXX:group/EMRSparkSteadyStateGroup",
         "Name": "EMRSparkSteadyStateGroup"
     }, ...
  1. Use the following code to associate the capacity reservation to the resource group. You can have multiple capacity reservations associated to a resource group.
aws resource-groups group-resources --group EMRSparkSteadyStateGroup \
 --resource-arns arn:aws:ec2:us-east-1:XXXX:capacity-reservation/cr-0123456f9907xxxxx
  1. As a best practice for effective management and cleanup, Create a tag Purpose=EMR-Spark-Steady-State for the newly created ODCR and the resource group.
# Tag your Capacity Reservation
 aws ec2 create-tags \
 --resources cr-0123456f9907xxxxx \
 --tags Key=Purpose,Value=EMR-Spark-Steady-State
# Tag your Resource Group
 aws resource-groups tag \
 --arn "arn:aws:resource-groups:us-east-1:XXXX:group/EMRSparkSteadyStateGroup" --tags Purpose=EMR-Spark-Steady-State

Implement Amazon EMR with ODCR

Complete the following steps to create an EMR cluster tagged with the specific targeted ODCR:

  1. Add required permissions to the EMR service role before using capacity reservations. With these permissions, you can lock down the resource with the specific Amazon Resource Name (ARN) of the group name to be created with the following code:
{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Resource": "*",
             "Action": [
                 "ec2:CreateFleet",
                 "ec2:RunInstances",
                 "ec2:CreateLaunchTemplate",
                 "ec2:CreateLaunchTemplateVersion",
                 "ec2:DeleteLaunchTemplateVersions",
                 "ec2:DescribeCapacityReservations",
                 "ec2:DescribeLaunchTemplateVersions",
                 "resource-groups:ListGroupResources"
             ]
         }
     ]
 }
  1. Configure the EMR cluster to use ODCR with instance fleets. We use the CapacityReservationOptions parameter to configure the EMR cluster, as shown in the following example:
  {
 ...
     "LaunchSpecifications": {
       "OnDemandSpecification": {
         "AllocationStrategy": "LOWEST_PRICE",
         "CapacityReservationOptions": {
           "UsageStrategy": "USE_CAPACITY_RESERVATIONS_FIRST",
           "CapacityReservationResourceGroupArn": "arn:aws:resource-groups:us-east-1:xxxxxx:group/EMRSparkSteadyStateGroup"
         }
       }
     }
   }

The following step-by-step breakdown illustrates the Amazon EMR decision-making process when prioritizing targeted capacity reservations, from core node provisioning through task node allocation:

  • Cluster provisioning initiation:
    • The user chooses to override the lowest-price allocation strategy.
    • The user specifies targeted capacity reservations in the launch request.
  • Core node provisioning:
    • Amazon EMR evaluates all EC2 instance capacity pools with targeted capacity reservations, and selects the pool with the lowest price that has sufficient capacity for all requested core nodes.
    • If no pool with targeted reservations has sufficient capacity, Amazon EMR reevaluates all specified EC2 instance capacity pools and selects the lowest-priced pool with sufficient capacity for core nodes. Available open capacity reservations are applied automatically.
  • Availability Zone selection:
    • After the core capacity is acquired, Amazon EMR locks in the Availability Zone for your cluster.
  • Primary and task node provisioning:
    • Amazon EMR evaluates EC2 instance capacity pools within that Availability Zone for primary and task fleets. First, Amazon EMR evaluates all the pools with targeted ODCRs specified in the request, ordered by lowest price by default.
    • From the ordered list, Amazon EMR launches as much capacity as possible from the unused targeted ODCRs of each instance pool until the request is fulfilled.
    • If the unused targeted ODCRs don’t fulfill the request yet, Amazon EMR continues to launch the remaining capacity into On-Demand pools, in the lowest-price order by default.

For more details about the allocation strategy, refer to Allocation strategy for instance fleets or Amazon EMR Support for Targeted ODCR.

Spiky workloads

Spiky workloads are defined by unpredictable and often significant fluctuations in processing demands, triggered by factors such as infrequent but resource-intensive periodic batch processing jobs. For example, a geographic information system processes location data from millions of users in real time to provide up-to-date traffic information, calculate routes, and suggest points of interest. User location data is constantly being generated, but the volume can spike dramatically during rush hour or special events, as illustrated in the following figure. This graph shows the number of used resources (Amazon EC2) by hour; it varies from 1 when the cluster scales in, waiting for jobs, to spikes of 1,000 nodes.

If you’re running spiky workloads with limited flexibility in instance type, family, and Availability Zone, you might face ICE errors when the available capacity can’t meet the cluster’s scaling requirements. To address this, we explore a set of best practices for EMR cluster creation to maximize availability and balance price-performance. Although spiky workloads present a unique challenge in resource management, configuring EMR instance fleets offers a powerful solution. By using diverse instance types, prioritized allocation strategies, Availability Zone flexibility, and managed scaling, organizations can create a robust, cost-effective infrastructure capable of handling unpredictable workload patterns. This configuration offers the following benefits:

  • Improved availability – By diversifying instance types and using multiple Availability Zones, the cluster mitigates insufficient capacity issues
  • Cost savings – Allocation strategies reduce costs while minimizing interruptions
  • Resilience for spiky workloads – Prioritizing instance generations provides seamless scaling under varying demands
  • Optimized performance – Managed scaling dynamically adjusts resources to meet workload demands efficiently

Introduce EC2 instance flexibility and instance fleets with a prioritized allocation strategy

Amazon EMR supports instance flexibility with instance fleet deployment. Instance fleets give you a wider variety of options and intelligence around instance provisioning. You can now provide a list of up to 30 instance types with corresponding weighted capacities and spot bid prices (including spot blocks) using the AWS CLI or AWS CloudFormation. Amazon EMR will automatically provision On-Demand and Spot capacity across these instance types when creating your cluster. This can make it more straightforward and more cost-effective to quickly obtain and maintain your desired capacity for your clusters. In August 2024, Amazon EMR introduced the prioritized allocation strategy to enhance instance flexibility with instance fleets. This feature allows you to specify priority levels for your instance types, enabling Amazon EMR to allocate capacity to the highest-priority instances first. This strategy helps improve cost savings and reduces the time required to launch clusters, even in scenarios with limited capacity. For more details, see Amazon EMR support prioritized and capacity-optimized-prioritized allocation strategies for EC2 instances. To maximize cost-efficiency and availability for spiky workloads, combine the price-performance advantages of new-generation instances with the broader availability of previous-generation instances. For workloads with strict latency requirements, fix the instance size to maintain consistent performance. This approach takes advantage of the strengths of both instance generations, providing flexibility and reliability decreasing the likelihood of capacity constraints. For On-Demand nodes, choose the prioritized allocation strategy, so the cluster tries to use newer-generation instances first. While configuring the instance fleet, arrange instances in a prioritized order reflecting price-performance and availability trade-offs, for example:

  • Primary node – m8g.12xlarge > m8g.16xlarge > m7g.12xlarge > m7g.16xlarge
  • Core node – r8g.8xlarge > r8g.12xlarge > r7g.8xlarge > r6g.16xlarge > r5.16xlarge
  • Task Node – r8g.8xlarge > r8g.12xlarge > r7g.8xlarge > r6g.16xlarge > r5.16xlarge

For Spot Instances, make sure the capacity-optimized prioritized allocation strategy is selected to reduce interruptions. See the following CloudFormation template snippet as an example:

...
       "Properties": {
         "Instances": {
          "MasterInstanceFleet": {
            "Name": "cfnMaster",
            "InstanceTypeConfigs": [
               {
                 "BidPrice": "10.50",
                 "InstanceType": "m5.xlarge",
                 "Priority": "1",
 ...
             "LaunchSpecifications": {
               "SpotSpecification": {
                 "TimeoutAction": "SWITCH_TO_ON_DEMAND",
                 "TimeoutDurationMinutes": 20,
                 "AllocationStrategy": "CAPACITY_OPTIMIZED_PRIORITIZED"
               },
               "OnDemandSpecification": {
                "AllocationStrategy": "PRIORITIZED"
               }
 ...

Select subnets with EMR instance fleets

When creating a cluster, specify multiple EC2 subnets within a virtual private cloud (VPC), each corresponding to a different Availability Zone. Amazon EMR provides multiple subnet (Availability Zone) options by employing subnet filtering at cluster launch, and selects one of the subnets that has adequate available IP addresses to successfully launch all instance fleets. If Amazon EMR can’t find a subnet with sufficient IP addresses to launch the whole cluster, it will prioritize the subnet that can at least launch the core and primary instance fleets.

Use managed scaling

Managed scaling is another powerful feature of Amazon EMR that automatically adjusts the number of instances in your cluster based on workload demands. This makes sure that your cluster scales up during periods of high demand to meet processing requirements and scales down during idle times to save costs. With managed scaling, you can set minimum and maximum scaling limits, giving you control over costs while benefiting from an optimized and efficient cluster performance.

The following workflow illustrates Amazon EMR configured with instance fleets and managed scaling.

The workflow consists of the following steps:

  1. The user defines the EMR instance configurations and instance types, along with their launch priority.
  2. The user selects subnets for the Amazon EMR configuration to provide Availability Zone flexibility.
  3. Amazon EMR calls the Amazon EC2 Fleet API to provision instances based on the allocation strategy.
  4. The EMR instance fleet is launched.
  5. The cycle is repeated for scaling operations within the launched Availability Zone, providing optimized performance and scalability.

Conclusion

In this post, we demonstrated how to optimize capacity by analyzing EMR workloads and implementing strategies tailored to your workload patterns. As you implement any of the preceding strategies, remember to continuously monitor your cluster’s performance and adjust configurations based on your specific workload patterns and business needs. With the right approach, the challenges of spiky workloads can be transformed into opportunities for optimized performance and cost savings.

To effectively manage workloads with both baseline demands and unexpected spikes, consider implementing a hybrid approach in Amazon EMR. Use ODCRs for consistent baseline capacity and configure instance fleets with a strategic mix of ODCR, On-Demand, and Spot Instances prioritizing ODCR usage.

Try these strategies with your own use case, and leave your questions in the comments.

About the Authors

Deepmala Agarwal works as an AWS Data Specialist Solutions Architect. She is passionate about helping customers build out scalable, distributed, and data-driven solutions on AWS. When not at work, Deepmala likes spending time with family, walking, listening to music, watching movies, and cooking!

Suba Palanisamy is a Senior Technical Account Manager, helping customers achieve operational excellence on AWS. Suba is passionate about all things data and analytics. She enjoys traveling with her family and playing board games.

Flavio Torres is a Principal Technical Account Manager at AWS. Flavio helps Enterprise Support customers design, deploy, and scale resilient cloud applications. Outside of work, he enjoys hiking and barbecuing.

AWS Pi Day 2025: Data foundation for analytics and AI

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-pi-day-data-foundation-for-analytics-and-ai/

Every year on March 14 (3.14), AWS Pi Day highlights AWS innovations that help you manage and work with your data. What started in 2021 as a way to commemorate the fifteenth launch anniversary of Amazon Simple Storage Service (Amazon S3) has now grown into an event that highlights how cloud technologies are transforming data management, analytics, and AI.

This year, AWS Pi Day returns with a focus on accelerating analytics and AI innovation with a unified data foundation on AWS. The data landscape is undergoing a profound transformation as AI emerges in most enterprise strategies, with analytics and AI workloads increasingly converging around a lot of the same data and workflows. You need an easy way to access all your data and use all your preferred analytics and AI tools in a single integrated experience. This AWS Pi Day, we’re introducing a slate of new capabilities that help you build unified and integrated data experiences.

The next generation of Amazon SageMaker: The center of all your data, analytics, and AI
At re:Invent 2024, we introduced the next generation of Amazon SageMaker, the center of all your data, analytics, and AI. SageMaker includes virtually all the components you need for data exploration, preparation and integration, big data processing, fast SQL analytics, machine learning (ML) model development and training, and generative AI application development. With this new generation of Amazon SageMaker, SageMaker Lakehouse provides you with unified access to your data and SageMaker Catalog helps you to meet your governance and security requirements. You can read the launch blog post written by my colleague Antje to learn more details.

Core to the next generation of Amazon SageMaker is SageMaker Unified Studio, a single data and AI development environment where you can use all your data and tools for analytics and AI. SageMaker Unified Studio is now generally available.

SageMaker Unified Studio facilitates collaboration among data scientists, analysts, engineers, and developers as they work on data, analytics, AI workflows, and applications. It provides familiar tools from AWS analytics and artificial intelligence and machine learning (AI/ML) services, including data processing, SQL analytics, ML model development, and generative AI application development, into a single user experience.

SageMaker Unified Studio

SageMaker Unified Studio also brings selected capabilities from Amazon Bedrock into SageMaker. You can now rapidly prototype, customize, and share generative AI applications using foundation models (FMs) and advanced features such as Amazon Bedrock Knowledge Bases, Amazon Bedrock Guardrails, Amazon Bedrock Agents, and Amazon Bedrock Flows to create tailored solutions aligned with your requirements and responsible AI guidelines all within SageMaker.

Last but not least, Amazon Q Developer is now generally available in SageMaker Unified Studio. Amazon Q Developer provides generative AI powered assistance for data and AI development. It helps you with tasks like writing SQL queries, building extract, transform, and load (ETL) jobs, and troubleshooting, and is available in the Free tier and Pro tier for existing subscribers.

You can learn more about SageMaker Unified Studio in this recent blog post written by my colleague Donnie.

During re:Invent 2024, we also launched Amazon SageMaker Lakehouse as part of the next generation of SageMaker. SageMaker Lakehouse unifies all your data across Amazon S3 data lakes, Amazon Redshift data warehouses, and third-party and federated data sources. It helps you build powerful analytics and AI/ML applications on a single copy of your data. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with Apache Iceberg–compatible tools and engines. In addition, zero-ETL integrations automate the process of bringing data into SageMaker Lakehouse from AWS data sources such as Amazon Aurora or Amazon DynamoDB and from applications such as Salesforce, Facebook Ads, Instagram Ads, ServiceNow, SAP, Zendesk, and Zoho CRM. The full list of integrations is available in the SageMaker Lakehouse FAQ.

Building a data foundation with Amazon S3
Building a data foundation is the cornerstone of accelerating analytics and AI workloads, enabling organizations to seamlessly manage, discover, and utilize their data assets at any scale. Amazon S3 is the world’s best place to build a data lake, with virtually unlimited scale, and it provides the essential foundation for this transformation.

I’m always astonished to learn about the scale at which we operate Amazon S3: It currently holds over 400 trillion objects, exabytes of data, and processes a mind-blowing 150 million requests per second. Just a decade ago, not even 100 customers were storing more than a petabyte (PB) of data on S3. Today, thousands of customers have surpassed the 1 PB milestone.

Amazon S3 stores exabytes of tabular data, and it averages over 15 million requests to tabular data per second. To help you reduce the undifferentiated heavy lifting when managing your tabular data in S3 buckets, we announced Amazon S3 Tables at AWS re:Invent 2024. S3 Tables are the first cloud object store with built-in support for Apache Iceberg. S3 tables are specifically optimized for analytics workloads, resulting in up to threefold faster query throughput and up to tenfold higher transactions per second compared to self-managed tables.

Today, we’re announcing the general availability of Amazon S3 Tables integration with Amazon SageMaker Lakehouse  Amazon S3 Tables now integrate with Amazon SageMaker Lakehouse, making it easy for you to access S3 Tables from AWS analytics services such as Amazon Redshift, Amazon Athena, Amazon EMR, AWS Glue, and Apache Iceberg–compatible engines such as Apache Spark or PyIceberg. SageMaker Lakehouse enables centralized management of fine-grained data access permissions for S3 Tables and other sources and consistently applies them across all engines.

For those of you who use a third-party catalog, have a custom catalog implementation, or only need basic read and write access to tabular data in a single table bucket, we’ve added new APIs that are compatible with the Iceberg REST Catalog standard. This enables any Iceberg-compatible application to seamlessly create, update, list, and delete tables in an S3 table bucket. For unified data management across all of your tabular data, data governance, and fine-grained access controls, you can also use S3 Tables with SageMaker Lakehouse.

To help you access S3 Tables, we’ve launched updates in the AWS Management Console. You can now create a table, populate it with data, and query it directly from the S3 console using Amazon Athena, making it easier to get started and analyze data in S3 table buckets.

The following screenshot shows how to access Athena directly from the S3 console.

S3 console : create table with AthenaWhen I select Query tables with Athena or Create table with Athena, it opens the Athena console on the correct data source, catalog, and database.

S3 Tables in Athena

Since re:Invent 2024, we’ve continued to add new capabilities to S3 Tables at a rapid pace. For example, we added schema definition support to the CreateTable API and you can now create up to 10,000 tables in an S3 table bucket. We also launched S3 Tables into eight additional AWS Regions, with the most recent being Asia Pacific (Seoul, Singapore, Sydney) on March 4, with more to come. You can refer to the S3 Tables AWS Regions page of the documentation to get the list of the eleven Regions where S3 Tables are available today.

Amazon S3 Metadata—announced during re:Invent 2024— has been generally available since January 27. It’s the fastest and easiest way to help you discover and understand your S3 data with automated, effortlessly-queried metadata that updates in near real time. S3 Metadata works with S3 object tags. Tags help you logically group data for a variety of reasons, such as to apply IAM policies to provide fine-grained access, specify tag-based filters to manage object lifecycle rules, and selectively replicate data to another Region. In Regions where S3 Metadata is available, you can capture and query custom metadata that is stored as object tags. To reduce the cost associated with object tags when using S3 Metadata, Amazon S3 reduced pricing for S3 object tagging by 35 percent in all Regions, making it cheaper to use custom metadata.

AWS Pi Day 2025
Over the years, AWS Pi Day has showcased major milestones in cloud storage and data analytics. This year, the AWS Pi Day virtual event will feature a range of topics designed for developers and technical decision-makers, data engineers, AI/ML practitioners, and IT leaders. Key highlights include deep dives, live demos, and expert sessions on all the services and capabilities I discussed in this post.

By attending this event, you’ll learn how you can accelerate your analytics and AI innovation. You’ll learn how you can use S3 Tables with native Apache Iceberg support and S3 Metadata to build scalable data lakes that serve both traditional analytics and emerging AI/ML workloads. You’ll also discover the next generation of Amazon SageMaker, the center for all your data, analytics, and AI, to help your teams collaborate and build faster from a unified studio, using familiar AWS tools with access to all your data whether it’s stored in data lakes, data warehouses, or third-party or federated data sources.

For those looking to stay ahead of the latest cloud trends, AWS Pi Day 2025 is an event you can’t miss. Whether you’re building data lakehouses, training AI models, building generative AI applications, or optimizing analytics workloads, the insights shared will help you maximize the value of your data.

Tune in today and explore the latest in cloud data innovation. Don’t miss the opportunity to engage with AWS experts, partners, and customers shaping the future of data, analytics, and AI.

If you missed the virtual event on March 14, you can visit the event page at any time—we will keep all the content available on-demand there!

— seb

How is the News Blog doing? Take this 1 minute survey!

(This survey is hosted by an external company. AWS handles your information as described in the AWS Privacy Notice. AWS will own the data gathered via this survey and will not share the information collected with survey respondents.)

[$] The burden of knowledge: dealing with open-source risks

Post Syndicated from jzb original https://lwn.net/Articles/1013614/

Organizations relying on open-source software have a wide range of
tools, scorecards, and methodologies to try to assess security, legal,
and other risks inherent in
their so-called supply chain. However, Max Mehl argued
recently in a short talk at FOSS Backstage in Berlin (and
online) that all of
this objective information and data is insufficient to truly
understand and address risk. Worse, this information doesn’t provide
options to improve the situation and encourages a passive mindset. Mehl, who works as part of
the CTO group at DB Systel, encouraged better risk assessment using
qualitative data and direct participation in open source.

Unlocking MSSP Success: Why CTEM is Critical

Post Syndicated from Rapid7 original https://blog.rapid7.com/2025/03/14/unlocking-mssp-success-why-ctem-is-critical/

Co-authored by Thomas Green and Sid Nanda

What is Continuous Threat Exposure Management (CTEM)?

Unlocking MSSP Success: Why CTEM is Critical

Continuous Threat Exposure Management (CTEM) is a five-stage, continuous security program introduced by Gartner in 2022. It proactively assesses an organization’s exposure across networks, systems, cloud infrastructure, IoT devices, applications, and identities. Unlike traditional vulnerability assessments, CTEM prioritizes risk mitigation strategies and iteratively refines security postures through continuous validation and remediation.

By emphasizing offensive security techniques such as continuous red teaming and simulation-based testing, CTEM goes beyond basic vulnerability prioritization to identify and address weaknesses before adversaries can exploit them. The result is an adaptable, intelligence-driven security framework that enables organizations to transition from reactive defenses to proactive resilience.

Why Should Service Providers Care?

The MSSP market is increasingly competitive, and differentiation is critical. Gartner has identified CTEM as a key opportunity for MSSPs in 2024, emphasizing that “product leaders who differentiate their portfolios by offering services that result in prioritized remediation outcomes and measured reduction in exposure stand out in the crowded MSSP market.”

Moreover, the expansion of the attack surface – combined with the inability to patch all vulnerabilities in a timely manner -has made traditional vulnerability management insufficient. MSSPs must demonstrate how their services deliver tangible security benefits to customers. A well-structured CTEM program enables MSSPs to provide a continuous, data-driven security validation framework that reduces risk while aligning stakeholders across IT, security, and leadership teams.

CTEM: A Programmatic Approach

Unlike traditional security programs that rely on static tools, CTEM is an adaptive program that integrates offensive security methodologies, including red teaming, penetration testing, vulnerability management, cloud security posture management, and web application security. This holistic approach ensures alignment with Governance, Risk, and Compliance (GRC) initiatives while delivering continuous security improvements.

The Five Stages of a CTEM Program

1. Scoping: Defining the Attack Surface

Scoping, as defined by Gartner, involves identifying an organization’s complete attack surface. This is where Surface Command, a key component of the Exposure Command suite, provides critical visibility by combining Cyber Asset Attack Surface Management (CAASM) with External Attack Surface Management (EASM). Unlike point-in-time assessments, Surface Command offers continuous monitoring of assets across on-prem, cloud, and SaaS environments, ensuring that no shadow IT or misconfigured exposure goes undetected.

2. Discovery: Mapping Assets, Vulnerabilities, and Risks

Discovery involves identifying both known and hidden assets, vulnerabilities, and misconfigurations. A common pitfall is confusing scoping with discovery—simply identifying a large number of vulnerabilities does not equate to security success.

Rapid7 Exposure Command enhances the discovery phase by integrating InsightVM for vulnerability management, InsightAppSec for dynamic application security testing (DAST), and InsightCloudSec for cloud security posture management (CSPM). These tools work together to provide comprehensive visibility intoexposure across hybrid environments.

3. Prioritization: Focusing on What Matters

Not all security issues require immediate remediation. Effective prioritization should factor in:

  • Business risk and potential impact
  • Urgency and exploitability
  • Availability of compensating controls
  • Tolerance for residual attack surface

Rapid7 Exposure Command’s risk-based prioritization framework goes beyond CVSS scoring by incorporating real-world exploitability data, asset criticality, and threat intelligence. The Command Platform provides a unified view of risks and remediation priorities, enabling MSSPs to help customers focus on the most impactful security improvements.

4. Validation: Proving Security Effectiveness

Validation is the cornerstone of CTEM. Organizations must confirm that vulnerabilities are exploitable, understand potential attack paths, and assess the effectiveness of security controls.

MSSPs can use continuous red teaming, penetration testing, and adversary simulations to validate security postures. Additionally, Security Information and Event Management (SIEM) solutions provide real-time threat correlation, ensuring organizations can detect and respond to threats before they escalate.

5. Mobilization: Operationalizing Security Improvements

The final stage of CTEM is mobilizing findings into actionable security improvements. This involves:

  • Streamlining approval workflows for remediation
  • Automating patch management and configuration changes
  • Ensuring alignment between IT, security, and executive teams

Rapid7 Exposure Command facilitates mobilization by providing automation and orchestration capabilities, reducing friction in vulnerability remediation processes. By integrating with existing IT workflows, MSSPs can ensure that security enhancements are implemented efficiently and effectively.

Achieving a Secure Environment with CTEM

The burden of threat management continues to grow as attack surfaces expand and adversaries evolve. MSSPs must help organizations move beyond traditional vulnerability management to a continuous, risk-driven security approach.

By leveraging ExposureCommand, MSSPs can:

  • Provide continuous visibility into evolving threats and vulnerabilities
  • Enable proactive risk mitigation through prioritized remediation
  • Validate security effectiveness through ongoing testing and adversary simulations
  • Streamline remediation efforts with automation and orchestration

CTEM is not just a security strategy—it’s a key differentiator for MSSPs. By embedding CTEM into their service offerings, MSSPs can deliver measurable risk reduction, enhance customer trust, and solidify their role as strategic security partners.

Learn More about Rapid7’s Exposure Command ▶︎

Confidently identify and prioritize exposures from endpoint to cloud with full attack surface visibility and threat-aware risk context.

The collective thoughts of the interwebz