How BMW Group built a serverless terabyte-scale data transformation architecture with dbt and Amazon Athena

Post Syndicated from Philipp Karg original https://aws.amazon.com/blogs/big-data/how-bmw-group-built-a-serverless-terabyte-scale-data-transformation-architecture-with-dbt-and-amazon-athena/

Businesses increasingly require scalable, cost-efficient architectures to process and transform massive datasets. At the BMW Group, our Cloud Efficiency Analytics (CLEA) team has developed a FinOps solution to optimize costs across over 10,000 cloud accounts. While enabling organization-wide efficiency, the team also applied these principles to the data architecture, making sure that CLEA itself operates frugally. After evaluating various tools, we built a serverless data transformation pipeline using Amazon Athena and dbt.

This post explores our journey, from the initial challenges to our current architecture, and details the steps we took to achieve a highly efficient, serverless data transformation setup.

Challenges: Starting from a rigid and costly setup

In our early stages, we encountered several inefficiencies that made scaling difficult. We were managing complex schemas with wide tables that required significant effort in maintainability. Initially, we used Terraform to create tables and views in Athena, allowing us to manage our data infrastructure as code (IaC) and automate deployments through continuous integration and delivery (CI/CD) pipelines. However, this method slowed us down when changing data models or dealing with schema changes, therefore requiring high development efforts.

As our solution grew, we faced challenges with query performance and costs. Each query scanned large amounts of raw data, resulting in increased processing time and higher Athena costs. We used views to provide a clean abstraction layer, but this masked underlying complexity because seemingly simple queries against these views scanned large volumes of raw data, and our partitioning strategy wasn’t optimized for these access patterns. As our datasets grew, the lack of modularity in our data design increased complexity, making scalability and maintenance increasingly difficult. We needed a solution for pre-aggregating, computing, and storing query results of computationally intensive transformations. The absence of robust testing and lineage solutions made it challenging to identify the root causes of data inconsistencies when they occurred.

As part of our business intelligence (BI) solution, we used Amazon QuickSight to build our dashboards, providing visual insights into our cloud cost data. However, our initial data architecture led to challenges. We were building dashboards on top of large, wide datasets, with some hitting the QuickSight per-dataset SPICE limit of 1 TB. Additionally, during SPICE ingest, our largest datasets required 4–5 hours of processing time due to performing full scans each time, often scanning over a terabyte of data. This architecture wasn’t helping us be more agile and quick while scaling up. The long processing times and storage limitations hindered our ability to provide timely insights and expand our analytics capabilities.

To address these issues, we enhanced the data architecture with AWS Lambda, AWS Step Functions, AWS Glue, and dbt. This tool stack significantly enhanced our development agility, empowering us to quickly modify and introduce new data models. At the same time, we improved our overall data processing efficiency with incremental loads and better schema management.

Solution overview

Our current architecture consists of a serverless and modular pipeline coordinated by GitHub Actions workflows. We chose Athena as our primary query engine for several strategic reasons: it aligns perfectly with our team’s SQL expertise, excels at querying Parquet data directly in our data lake, and alleviates the need for dedicated compute resources. This makes Athena an ideal fit for CLEA’s architecture, where we process around 300 GB daily from a data lake of 15 TB, with our largest dataset containing 50 billion rows across up to 400 columns. The capability of Athena to efficiently query large-scale Parquet data, combined with its serverless nature, enables us to focus on writing efficient transformations rather than managing infrastructure.

The following diagram illustrates the solution architecture.

Using this architecture, we’ve streamlined our data transformation process using dbt. In dbt, a data model represents a single SQL transformation that creates either a table or a view—essentially a building block of our data transformation pipeline. Our implementation includes around 400 such models, 50 data sources, and around 100 data tests. This setup enables seamless updates—whether creating new models, updating schemas, or modifying views—triggered simply by creating a pull request in our source code repository, with the rest handled automatically.

Our workflow automation includes the following features:

  • Pull request – When we create a pull request, it’s deployed to our testing environment first. After passing validation and being approved or merged, it’s deployed to production using GitHub workflows. This setup enables seamless model creation, schema updates, or view changes—triggered just by creating a pull request, with the rest handled automatically.
  • Cron scheduler – For nightly runs or multiple daily runs to reduce data latency, we use scheduled GitHub workflows. This setup allows us to configure specific models with different update strategies based on data needs. We can set models to update incrementally (processing only new or changed data), as views (querying without materializing data), or as full loads (completely refreshing the data). This flexibility optimizes processing time and resource usage. We can target only specific folders—like source, prepared, or semantic layers—and run the dbt test afterward to validate model quality.
  • On demand – When adding new columns or changing business logic, we need to update historical data to maintain consistency. For this, we use a backfill process, which is a custom GitHub workflow created by our team. The workflow allows us to select specific models, include their upstream dependencies, and set parameters like start and end dates. This makes sure that changes are applied accurately across the entire historical dataset, maintaining data consistency and integrity.

Our pipeline is organized into three primary stages—Source, Prepared, and Semantic—each serving a specific purpose in our data transformation journey. The Source stage maintains raw data in its original form. The Prepared stage cleanses and standardizes this data, handling tasks like deduplication and data type conversions. The Semantic stage transforms this prepared data into business-ready models aligned with our analytical needs. An additional QuickSight step handles visualization requirements. To achieve low cost and high performance, we use dbt models and SQL code to manage all transformations and schema changes. By implementing incremental processing strategies, our models process only new or changed data rather than reprocessing the entire dataset with each run.

The Semantic stage (not to be confused with dbt’s semantic layer feature) introduces business logic, transforming data into aggregated datasets that are directly consumable by BMW’s Cloud Data Hub, internal CLEA dashboards, data APIs, or In-Console Cloud Assistant (ICCA) chatbot. The QuickSight step further optimizes data by selecting only necessary columns by using a column-level lineage solution and setting a dynamic date filter with a sliding window to ingest only relevant hot data into SPICE, avoiding unused data in dashboards or reports.

This approach aligns with BMW Group’s broader data strategy, which includes streamlining data access using AWS Lake Formation for fine-grained access control.

Overall, as a high-level structure, we’ve fully automated schema changes, data updates, and testing through GitHub pull requests and dbt commands. This approach enables controlled deployment with robust version control and change management. Continuous testing and monitoring workflows uphold data accuracy, reliability, and quality across transformations, supporting efficient, collaborative model iteration.

Key benefits of the dbt-Athena architecture

To design and manage dbt models effectively, we use a multi-layered approach combined with cost and performance optimizations. In this section, we discuss how our approach has yielded significant benefits in five key areas.

SQL-based, developer-friendly environment

Our team already had strong SQL skills, so dbt’s SQL-centric approach was a natural fit. Instead of learning a new language or framework, developers could immediately start writing transformations using familiar SQL syntax with dbt. This familiarity aligns well with the SQL interface of Athena and, combined with dbt’s added functionality, has increased our team’s productivity.

Behind the scenes, dbt automatically handles synchronization between Amazon Simple Storage Service (Amazon S3), the AWS Glue Data Catalog, and our models. When we need to change a model’s materialization type—for example, from a view to a table—it’s as simple as updating a configuration parameter rather than rewriting code. This flexibility has reduced our development time dramatically, allowed us to focus on building better data models rather than managing infrastructure.

Agility in modeling and deployment

Documentation is crucial for any data platform’s success. We use dbt’s built-in documentation capabilities by publishing them to GitHub Pages, which creates an accessible, searchable repository of our data models. This documentation includes table schemas, relationships between models, and usage examples, enabling team members to understand how models interconnect and how to use them effectively.

We use dbt’s built-in testing capabilities to implement comprehensive data quality checks. These include schema tests that verify column uniqueness, referential integrity, and null constraints, as well as custom SQL tests that validate business logic and data consistency. The testing framework runs automatically on every pull request, validating data transformations at each step of our pipeline. Additionally, dbt’s dependency graph provides a visual representation of how our models interconnect, helping us understand the upstream and downstream impacts of any changes before we implement them. When stakeholders need to modify models, they can submit changes through pull requests, which, after they’re approved and merged, automatically trigger the necessary data transformations through our CI/CD pipeline. This streamlined process enabled us to create new data products within days compared to weeks and reduced ongoing maintenance work by catching issues early in the development cycle.

Athena workgroup separation

We use Athena workgroups to isolate different query patterns based on their execution triggers and purposes. Each workgroup has its own configuration and metric reporting, allowing us to monitor and optimize separately. The dbt workgroup handles our scheduled nightly transformations and on-demand updates triggered by pull requests through our Source, Prepared, and Semantic stages. The dbt-test workgroup executes automated data quality checks during pull request validation and nightly builds. The QuickSight workgroup manages SPICE data ingestion queries, and the Ad-hoc workgroup supports interactive data exploration by our team.

Each workgroup can be configured with specific data usage quotas, enabling teams to implement granular governance policies. This separation provides several benefits: it enables clear cost allocation, provides isolated monitoring of query patterns across different use cases, and helps enforce data governance through custom workgroup settings. Amazon CloudWatch monitoring per workgroup helps us track usage patterns, identify query performance issues, and adjust configurations based on actual needs.

Using QuickSight SPICE

QuickSight SPICE (Super-fast, Parallel, In-memory Calculation Engine) provides powerful in-memory processing capabilities that we’ve optimized for our specific use cases. Rather than loading entire tables into SPICE, we create specialized views on top of our materialized semantic models. These views are carefully crafted to include only the necessary columns, relevant metadata joins, and appropriate time filtering to have only recent data available in dashboards.

We’ve implemented a hybrid refresh strategy for these SPICE datasets: daily incremental updates keep the data fresh, and weekly full refreshes maintain data consistency. This approach strikes a balance between data freshness and processing efficiency. The result is responsive dashboards that maintain high performance while keeping processing costs under control.

Scalability and cost-efficiency

The serverless architecture of Athena eliminates manual infrastructure management, automatically scaling based on query demand. Because costs are based solely on the amount of data scanned by queries, optimizing queries to scan as little data as possible directly reduces our costs. We use the distributed query execution capabilities of Athena through our dbt model structure, enabling parallel processing across data partitions. By implementing effective partitioning strategies and using Parquet file format, we minimize the amount of data scanned while maximizing query performance.

Our architecture offers flexibility in how we materialize data through views, full tables, and incremental tables. With dbt’s incremental models and partitioning strategy, we process only new or modified data instead of entire datasets. This approach has proven highly effective—we’ve observed significant reductions in data processing volume as well as data scanning, particularly in our QuickSight workgroup.

The effectiveness of these optimizations implemented at the end of 2023 is visible in the following diagram, showing costs by Athena workgroups.

The workgroups are illustrated as follows:

  • Green (QuickSight): Shows reduced data scanning post-optimization.
  • Light blue (Ad-hoc): Varies based on analysis needs.
  • Dark blue (dbt): Maintains consistent processing patterns
  • Orange (dbt-test): Shows regular, efficient test execution.

The increased dbt workload costs directly correlate with decreased QuickSight costs, reflecting our architectural shift from using complex views in QuickSight workgroups (which previously masked query complexity but led to repeated computations) to using dbt for materializing these transformations. Although this increased the dbt workload, the overall cost-efficiency improved significantly because materialized tables reduced redundant computations in QuickSight. This demonstrates how our optimization strategies successfully manage growing data volumes while achieving net cost reduction through efficient data materialization patterns.

Conclusion

Our data architecture uses dbt and Athena to provide a scalable, cost-efficient, and flexible framework for building and managing data transformation pipelines. Athena’s ability to query data directly in Amazon S3 alleviates the need to move or copy data into a separate data warehouse, and its serverless model and dbt’s incremental processing minimize both operational overhead and processing costs. Given our team’s strong SQL expertise, expressing these transformations in SQL through dbt and Athena was a natural choice, enabling rapid model development and deployment. With dbt’s automatic documentation and lineage, troubleshooting and identifying data issues is simplified, and the system’s modularity allows for quick adjustments to meet evolving business needs.

Starting with this architecture is quick and straightforward: all that is needed is the dbt-core and dbt-athena libraries, and Athena itself requires no setup, because it’s a fully serverless service with seamless integration with Amazon S3. This architecture is ideal for teams looking to rapidly prototype, test, and deploy data models, optimizing resource usage, accelerating deployment, and providing high-quality, accurate data processing.

For those interested in a managed solution from dbt, see From data lakes to insights: dbt adapter for Amazon Athena now supported in dbt Cloud.

About the Authors

Philipp Karg is a Lead FinOps Engineer at BMW Group and has a strong background in data engineering, AI, and FinOps. He focuses on driving cloud efficiency initiatives and fostering a cost-aware culture within the company to leverage the cloud sustainably.

Selman Ay is a Data Architect specializing in end-to-end data solutions, architecture, and AI on AWS. Outside of work, he enjoys playing tennis and engaging outdoor activities.

Cizer Pereira is a Senior DevOps Architect at AWS Professional Services. He works closely with AWS customers to accelerate their journey to the cloud. He has a deep passion for cloud-based and DevOps solutions, and in his free time, he also enjoys contributing to open source projects.

Best practices for least privilege configuration in Amazon MWAA

Post Syndicated from Elizabeth Davis original https://aws.amazon.com/blogs/big-data/best-practices-for-least-privilege-configuration-in-amazon-mwaa/

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) provides a secure and managed environment to run Apache Airflow on AWS. Airflow is often used in highly regulated industries, such as finance and healthcare. These customers might want to further restrict access and traffic to enhance security posture than what the Amazon MWAA default configurations provide. This post covers some recommended practices.

The principle of least privilege is a fundamental tenet that should be followed diligently. When it comes to configuring AWS services, it’s essential to grant only the minimum required permissions to resources, avoiding overly broad or permissive policies.

In this post, we explore how to apply the principle of least privilege to your Amazon MWAA environment by tightening network security using security groups, network access control lists (ACLs), and virtual private cloud (VPC) endpoints. We also discuss the Amazon MWAA execution and deployment roles and their respective permissions.

Understanding the Amazon MWAA environment

When an Amazon MWAA environment is created, resources are created in an AWS managed service VPC and your customer managed VPC. In the customer VPC provided at environment creation, the necessary resources to run the Airflow environment are deployed, including schedulers and workers running on Amazon Elastic Container Service (Amazon ECS) clusters. These clusters are deployed in your VPC and they assume Elastic Network Interfaces (ENIs) with private IP addresses in the customer account. These ENIs span private subnets across two Availability Zones to connect to the Airflow database and web server, which reside in the service-owned account (if in private access mode). The following diagram illustrates this architecture.

MWAA Architecture

VPC security groups act as virtual firewalls that can control network traffic at the ENI level, or instance level. Security groups are stateful, meaning that inbound traffic is automatically permitted outbound and vice versa. The default security group configuration in a VPC starts with is no inbound rules and an outbound rule allowing all traffic. By definition, a security group with no inbound rules denies all ingress traffic that wasn’t allowed out through the 0.0.0.0/0 outbound rule.

Amazon MWAA offers two web server access modes inside the customer VPC: public and private. Public web server mode must have a way for traffic to access the web servers in the customer-owned VPC through the public internet. This requires routing to the public internet using public subnets and a NAT gateway. A NAT gateway can be used to provide internet access for resources in private subnets. With private access mode, the security group for the Amazon MWAA environment doesn’t need to allow traffic to and from the NAT gateway, only granting access to the Airflow UI to users with appropriate permissions from within the VPC. An Application Load Balancer is only provisioned in public mode to route traffic to the public web servers. The customer must provision the rest of the networking components.

If your Amazon MWAA environment needs to communicate with resources outside your VPC (such as external data sources or APIs), you might need to configure appropriate security group rules and routing to allow the necessary traffic. In such cases, you would typically use a NAT gateway or VPN connection to facilitate the communication between your Amazon MWAA environment and the external resources and VPC endpoints for AWS resources.

For tighter security restrictions, an environment with private routing without internet access is possible, and finer-grained security group rules can be applied and VPC endpoint policies can be used. Because this post is focusing on least privilege, we will focus on the minimum security requirements needed for an Amazon MWAA environment.

Security groups: Minimizing permissions

Your Amazon MWAA environment will have a security group associated with your VPC’s environment resources. This security group is also used by the ENIs created by the interface VPC endpoint that is used to communicate with the database and web server. By default, security groups deny all inbound traffic and security group rules need to be explicitly stated, denoting the ports and source that the instance will allow network traffic from. At a minimum, the Amazon MWAA environment must allow for traffic to and from the Amazon Aurora PostgreSQL-Compatible Edition metadata database that is owned and managed by Amazon MWAA. The metadata database is a crucial component of Airflow that acts as a centralized source of truth for task execution, configuration, and monitoring. Both the scheduler and workers require access to this database to perform their respective roles in orchestrating and running tasks. This database listens on TCP port 5432. Additionally, the web server traffic can be restricted to HTTPS through TCP port 443. At a minimum, the Amazon MWAA security group must have the two inbound rules, detailed in the following table.

Type Protocol Port Range Source Type Source
Custom TCP TCP 5432 Custom sg-xxxxx / my-mwaa-vpc-security-group
HTTPS TCP 443 Custom sg-xxxxx / my-mwaa-vpc-security-group

Many customers have other AWS resources residing in VPCs, to which the Amazon MWAA workers need access. These resources can be granted network access in a private routing configuration using security groups as well. If the resource sits in the same security group, add an additional inbound rule with the port needed. For example, if an Amazon Redshift cluster sits in the same security group, add the following rule.

Type Protocol Port Range Source Type Source
Custom TCP TCP 5439 Custom sg-xxxxx / my-mwaa-vpc-security-group

If the Redshift cluster is in a different security group, change the source to the Redshift security group.

Type Protocol Port Range Source Type Source
Custom TCP TCP 5439 Custom sg-xxxxx / redshift-security-group

If the resources are in another VPC, then VPC peering must be enabled before referencing that other VPC’s security group. For resources that don’t reside in a subnet, a VPC endpoint will also provide private routing to and from the Amazon MWAA environment and those resources. For example, a VPC endpoint for Amazon Simple Storage Service (Amazon S3) can provide enhanced security, improved performance, and lower costs.

Network ACLs: Minimizing permissions

Network ACLs can manage (by allow or deny rules) inbound and outbound traffic at the subnet level. An ACL is stateless, which means that inbound and outbound rules must be specified separately and explicitly. It is used to specify the types of network traffic that are allowed in or out from the instances in a VPC network.

Every Amazon VPC has a default ACL that allows all inbound and outbound traffic, with a rule as follows.

Rule number Type Protocol Port Range Source Allow/Deny
100 All IPv4 traffic All All 0.0.0.0/0 Allow
* All IPv4 traffic All All 0.0.0.0/0 Deny

You can edit the default ACL rules or create a custom ACL and attach it to your subnets. A subnet can only have one ACL attached to it at any time, but one ACL can be attached to multiple subnets. To implement least privilege in your Amazon MWAA environment, restrict the inbound ACL to allow traffic from the metadata database and web server and restrict the outbound to allow traffic to only the clients in the private subnet. Note the following examples use example private IPs for the subnets used.

Inbound NACL

Rule number Type Protocol Port Range Source Allow/Deny Comments
100 Custom TCP TCP 5432 10.192.21.0/16 Allow Allow inbound database traffic from private subnet
110 HTTPS TCP 443 10.192.21.0/16 Allow Allow inbound HTTPS traffic from private subnet
* All traffic All All 0.0.0.0/0 Deny Denies all inbound IPv4 traffic not already handled by a preceding rule (not modifiable)

Outbound NACL

Rule number Type Protocol Port Range Source Allow/Deny Comments
100 Custom TCP TCP 1024-65535 10.192.21.0/24 Allow Allows outbound return IPv4 traffic to clients in private subnet
* All traffic All All 0.0.0.0/0 Deny Denies all outbound IPv4 traffic not already handled by a preceding rule (not modifiable)

VPC endpoints: Minimizing permissions

When you create an Amazon MWAA environment, it is deployed within a VPC. This allows you to control the network access and security of your Airflow deployment. However, some customer workloads executing in the Amazon MWAA environment might need to orchestrate tasks using other AWS services, such as Amazon S3 to access files, AWS Glue to start ETL (extract, transform, and load) jobs, or Amazon Redshift for running data warehouse queries, which reside outside of your VPC. To establish a secure and private connection between your Amazon MWAA environment and these external AWS services, you can use VPC endpoints. The purpose of VPC endpoints in Amazon MWAA is to provide a secure and private connection between your Amazon MWAA environment and other AWS services within your VPC. VPC endpoints are virtual devices that are provisioned within your VPC and act as an entry point for the specified AWS service, allowing your Amazon MWAA environment to communicate with the service using a private IP address, without needing to go through the public internet. The following diagram illustrates this architecture.

VPCEndpointsMWAA

VPC endpoints allow you to keep your Amazon MWAA environment’s network traffic within the AWS network, reducing the exposure to the public internet and enhancing the overall security of your Airflow deployment. Although private VPC endpoints are automatically created for the database and web server, to create a least privileged environment without internet access, additional VPC endpoints will be needed for the additional Amazon MWAA required resources. Amazon S3, Amazon Simple Queue Service (Amazon SQS), Amazon CloudWatch, and optionally AWS Key Management Service (AWS KMS) will need VPC endpoints created. For more details, see Creating the required VPC service endpoints in an Amazon VPC with private routing. Outside of the necessary services, many customers run Amazon MWAA workflows that orchestrate additional AWS services, such as Amazon Redshift, Amazon EMR, and AWS Glue. Let’s look at an example VPC endpoint that we want to use to connect to Amazon Redshift, which is commonly called in the Airflow DAGS using the Redshift Operator for workflows that interact with Amazon Redshift as a data warehouse. For more information on creating Amazon VPC interface endpoints, see Access an AWS service using an interface VPC endpoint.

Create a VPC endpoint

Complete the following steps to create a VPC endpoint using Amazon Virtual Private Cloud (Amazon VPC):

  1. On the Amazon VPC console, create a new VPC endpoint for the amazonaws.region.redshift service, where region is the AWS Region where your Amazon MWAA environment and Redshift cluster are located. Make sure that private DNS is enabled.
  2. Create a VPC endpoint policy. This can be used to limit access to the Redshift cluster only to the Amazon MWAA environment, preventing unauthorized access from other resources. The following is an example policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:role/YourMWAAExecutionRoleName"
        ]
      },
      "Action": [
        "redshift:DescribeClusters",
        "redshift:DescribeClusterParameters",
        "redshift:DescribeClusterSecurityGroups",
        "redshift:DescribeClusterSubnetGroups",
        "redshift:DescribeEventSubscriptions",
        "redshift:DescribeLoggingStatus",
        "redshift:DescribeReservedNodeOfferings",
        "redshift:DescribeReservedNodes",
        "redshift:DescribeTableRestoreStatus",
        "redshift:DescribeTags",
        "redshift:GetClusterCredentials",
        "redshift:ListTagsForResource",
        "redshift:PurchaseReservedNodeOffering",
        "redshift:ResetClusterParameterGroup",
        "redshift:RestoreFromClusterSnapshot",
        "redshift:RevokeClusterSecurityGroupIngress",
        "redshift:RevokeSnapshotAccess",
        "redshift:ViewQueriesInConsole"
      ],
      "Resource": "arn:aws:redshift:us-east-1:123456789012:cluster/my-redshift-cluster"
    }
  ]
}

The policy contains the following parameters:
  • The Version field specifies the policy language version.
  • The Statement section contains a single statement that allows the specified actions on the Redshift cluster.
  • The Effect field is set to Allow, which means the policy grants the specified permissions.
  • The Principal field specifies the AWS Identity and Access Management (IAM) role associated with your Amazon MWAA execution role, which is authorized to access the Redshift cluster.
  • The Action field lists the specific Redshift actions that the Amazon MWAA execution role is allowed to perform, such as describing the cluster, getting cluster credentials, and restoring from a snapshot.
  • The Resource field specifies the Amazon Resource Name (ARN) of the Redshift cluster that the policy applies to.
  1. Associate the VPC endpoint with the correct route table. This route table should be used by the subnets where your Amazon MWAA environment is deployed. If using a VPC interface endpoint, associate the endpoint with the two private subnets and security group used by Amazon MWAA.
  2. Make sure that the security groups associated with the Amazon MWAA environment and the Redshift cluster allow the necessary inbound and outbound traffic between them. This typically includes allowing access on the Redshift port (typically 5439) from the Amazon MWAA environment’s security group.
  3. On the Amazon MWAA console, under Admin, Connections, update the Redshift connection details to use the VPC endpoint address instead of the public Redshift endpoint. This makes sure that the connection between Amazon MWAA and Amazon Redshift is secure and stays within the VPC.

By configuring VPC endpoints for the AWS services your Amazon MWAA environment needs to access, you can provide secure, private, and efficient communication between your Airflow deployment and AWS resources.

Restricting traffic within AWS with a customer managed endpoints for Amazon MWAA resources

As mentioned earlier, Amazon MWAA integrates with various AWS services, such as CloudWatch for logging, Amazon S3 for DAGs and requirements, Amazon SQS as a messaging middleware, and optionally AWS KMS for encryption. You can create VPC endpoints for these services to make sure traffic stays within the AWS network. Access to these endpoints can be restricted by allowing only the Amazon MWAA security group as the ingress source. For details on how to create these endpoints and policies, see Introducing shared VPC support on Amazon MWAA. If the Amazon MWAA environment was updated after April 2, 2024, it will be on AWS Fargate v1.4 and will not use Amazon Elastic Container Registry (Amazon ECR) and therefore you will not need to create a VPC endpoint for it.

Managing permissions to deploy an Amazon MWAA environment

To create and deploy an Amazon MWAA environment, you need to have the appropriate permissions granted to your IAM user or role. The required permissions can be granted through an IAM policy attached to your user or role. When you create an Amazon MWAA environment, you can specify an execution role that will be assumed by the Airflow workers to perform tasks. The execution role should have the necessary permissions to access the required AWS services and resources based on your workflow requirements. It’s important to follow the principle of least privilege when granting permissions to IAM roles and users. You should only grant the minimum permissions required for your Amazon MWAA environment and Airflow workflows to function correctly.

Amazon MWAA trust policy

Amazon MWAA needs to be able to assume the execution role in order to perform actions on your behalf.  To do this, create a trust policy, allowing the Amazon MWAA service the ability to AssumeRole. To avoid the confused deputy problem, we add a condition to the trust policy, and replace the AWS account number and Region as needed. The following is an example policy:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
            "Service": ["airflow.amazonaws.com","airflow-env.amazonaws.com"]
        },
        "Action": "sts:AssumeRole",
        "Condition":{
            "ArnLike":{
               "aws:SourceArn":"arn:aws:airflow:your-region:123456789012:environment/your-environment-name"
            },
            "StringEquals":{
               "aws:SourceAccount":"123456789012"
            }
         }
      }
   ]
}

VPC endpoint permissions for the deployer role

Although the service-linked role creates the VPC endpoints, the deployer role requires permissions to create VPC endpoints and perform a dry run. You can limit these permissions by allowing the ec2:CreateVpcEndpoint action and specifying resource ARNs for VPC endpoints, VPCs, subnets, and security groups. Additionally, you can use the aws:CalledVia condition key to restrict access to the airflow.amazonaws.com service.

Amazon MWAA execution role: Required permissions

When creating an Amazon MWAA environment, you need to specify an execution role that grants the necessary permissions for Airflow to interact with other AWS services. Instead of using a wildcard policy, you can create a custom policy with the minimum required permissions.

The following is an example of an execution role policy that allows Amazon MWAA to interact with various services using an AWS managed key:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "airflow:PublishMetrics",
            "Resource": "arn:aws:airflow:{your-region}:{your-account-id}:environment/{your-environment-name}"
        },
        { 
            "Effect": "Deny",
            "Action": "s3:ListAllMyBuckets",
            "Resource": [
                "arn:aws:s3:::{your-s3-bucket-name}",
                "arn:aws:s3:::{your-s3-bucket-name}/*"
            ]
        },
        { 
            "Effect": "Allow",
            "Action": [ 
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::{your-s3-bucket-name}",
                "arn:aws:s3:::{your-s3-bucket-name}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents",
                "logs:GetLogEvents",
                "logs:GetLogRecord",
                "logs:GetLogGroupFields",
                "logs:GetQueryResults"
            ],
            "Resource": [
                "arn:aws:logs:{your-region}:{your-account-id}:log-group:airflow-{your-environment-name}-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetAccountPublicAccessBlock"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:PutMetricData",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage"
            ],
            "Resource": "arn:aws:sqs:{your-region}:*:airflow-celery-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:GenerateDataKey*",
                "kms:Encrypt"
            ],
            "Resource": "arn:aws:kms:your-region:your-account-id:key/your-kms-cmk-id",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "sqs.{your-region}.amazonaws.com",
                        "s3.{your-region}.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

This policy grants Amazon MWAA the necessary permissions to interact with CloudWatch Logs, Amazon S3, Amazon SQS, and AWS KMS when using the AWS managed key offering, while explicitly specifying the resources it can access. You can further refine this policy based on your specific requirements.

The following is an example of an execution policy that allows Amazon MWAA to interact with various services using a KMS customer managed key:

{
    "Version": "2012-10-17",
    "Statement": [
        { 
            "Effect": "Deny",
            "Action": "s3:ListAllMyBuckets",
            "Resource": [
                "arn:aws:s3:::{your-s3-bucket-name}",
                "arn:aws:s3:::{your-s3-bucket-name}/*"
            ]
        }, 
        { 
            "Effect": "Allow",
            "Action": [ 
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::{your-s3-bucket-name}",
                "arn:aws:s3:::{your-s3-bucket-name}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents",
                "logs:GetLogEvents",
                "logs:GetLogRecord",
                "logs:GetLogGroupFields",
                "logs:GetQueryResults"
            ],
            "Resource": [
                "arn:aws:logs:{your-region}:{your-account-id}:log-group:airflow-{your-environment-name}-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetAccountPublicAccessBlock"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:PutMetricData",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage"
            ],
            "Resource": "arn:aws:sqs:{your-region}:*:airflow-celery-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:GenerateDataKey*",
                "kms:Encrypt"
            ],
            "Resource": "arn:aws:kms:{your-region}:{your-account-id}:key/{your-kms-cmk-id}",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "sqs.{your-region}.amazonaws.com",
                        "s3.{your-region}.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

For the use case of using the customer managed key, attach the following JSON policy to the key to provide access to the Airflow logs in CloudWatch Logs:

{
    "Sid": "Allow logs access",
    "Effect": "Allow",
    "Principal": {
        "Service": "logs.{your-region}.amazonaws.com"
    },
    "Action": [
        "kms:Encrypt*",
        "kms:Decrypt*",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:Describe*"
    ],
    "Resource": "*",
    "Condition": {
        "ArnLike": {
            "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:{your-region}:{your-account-id}:*"
        }
    }
}

You can attach multiple policies to the execution role as needed to allow your workers to access additional AWS resources. For example, let’s explore how to enable Amazon EMR access. You can create a JSON policy that contains the narrowest permissions you can configure, as in the following example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:RunJobFlow"
            ],
            "Resource": "arn:aws:elasticmapreduce:*:xxxxxxxxxxxx:cluster/*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": [
                "arn:aws:iam::xxxxxxxxxxxx:role/EMR_EC2_DefaultRole",
                "arn:aws:iam::xxxxxxxxxxxx:role/EMR_DefaultRole"
            ]
        }
    ]
}

Conclusion

In this post, we discussed best practices for least privilege configuration in Amazon MWAA. By following these approaches, you can adhere to the principle of least privilege and maintain a secure posture within your Amazon MWAA environment, without compromising functionality or relying on overly permissive policies. Security is always top priority; to learn more about security in Amazon MWAA, see Security in Amazon Managed Workflows for Apache Airflow and Security best practices on Amazon MWAA.

About the Authors

elizaws-headshotElizabeth Davis is a Sr Solutions Architect at Amazon Web Services (AWS). She currently works with educational technology companies and has a passion for serverless and data orchestration technologies. She has been an Amazon MWAA as a subject matter expert (SME) for the last 3+ years.

mark headshotMark Richman is a Principal Solutions Architect at Amazon Web Services with 30 years of experience building complex web and enterprise software. He contributes to Apache Airflow, bringing his expertise in cloud computing and serverless technologies to the open-source platform. Mark is also an accomplished writer and speaker who has authored commercial publications and AWS courses while regularly presenting at industry events.

Intel Foundry Direct Connect 2025 Make-or-Break Time For Intel

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/intel-foundry-direct-connect-2025-make-or-break-time-for-intel/

We are at the Intel Foundry 2025 event which is a make-or-break moment for Intel as it looks to bolster its semiconductor foundry business

The post Intel Foundry Direct Connect 2025 Make-or-Break Time For Intel appeared first on ServeTheHome.

Barnes: Parallel ./configure

Post Syndicated from corbet original https://lwn.net/Articles/1019303/

Tavian Barnes takes on
the tedious process of waiting for configure scripts to run.

I paid good money for my 24 CPU cores, but ./configure can only
manage to use 69% of one of them. As a result, this random project
takes about 13.5× longer to configure the build than it does to
actually do the build.

The purpose of a ./configure script is basically to run the
compiler a bunch of times and check which runs succeeded. In this
way it can test whether particular headers, functions, struct
fields, etc. exist, which lets people write portable software. This
is an embarrassingly parallel problem, but Autoconf can’t
parallelize it, and neither can CMake, neither can Meson, etc.,
etc.

(Thanks to Paul Wise).

Meet B2 Overdrive: Terabit-Speed Throughput for AI/ML and HPC Workloads

Post Syndicated from David Ngo original https://www.backblaze.com/blog/b2-overdrive-announcement/

A decorative image showing a drive, the Backblaze logo, and a speedometer.

If you’re wrangling massive datasets for AI, machine learning (ML), high-performance compute (HPC), content delivery networks (CDNs), or analytics, you’re familiar with the trade-off: Pay a premium for the highest speeds, or compromise on performance to keep costs manageable. 

Backblaze B2 Overdrive changes that. You can now move exabyte-scale datasets at up to terabit speeds without the eye-watering price tag. Starting at $15 per terabyte per month, Backblaze B2 Overdrive gives you the power to run data-intensive workloads at peak performance, with unlimited free egress and private networking options that keep things fast, secure, and predictable.

See it in action

Join our upcoming webinar with Pat Patterson, Chief Technical Evangelist and Dave Ngo, Chief Product Officer, to learn more about how B2 Overdrive supercharges your data.

Sign Up ➔ 

What makes B2 Overdrive different?

B2 Overdrive offers a specialized cloud object storage solution at a fraction of competitors’ costs. Here’s what you get:

  • Up to 1Tbps throughput: In other words, the kind of speed that lets you move petabytes of data fast without complex architecture. 
  • Unlimited free egress: Move as much data as you want, whenever you want, to wherever you want. Egress is totally free. 
  • Private networking support: Transfer data at maximum speed through secure private networking connections to your infrastructure.

It’s built on the foundation of our always-hot cloud storage infrastructure, with no minimum file size requirements, no deletion fees, and powerful features like Event Notifications so you can build responsive and automated workflows. We’ll be sharing some of the innovations under the hood in the coming months—so, stay tuned to our series on the engineering behind performance. 

Who’s it for?

The simple answer: The status quo isn’t cutting it. Today’s workloads demand both the ability to move massive datasets and predictable economics that don’t penalize success. B2 Overdrive challenges the assumption that mind-bending performance has to come with mind-boggling prices.

We need to store an insane amount of data and, at the same time, download it to different GPU clusters around the world, and for all that to not cost an insane amount of money. That’s why we chose Backblaze.

—Dean Leitersdorf, CEO and Co-Founder, Decart

Ready to go?

Backblaze B2 Overdrive is generally available today for organizations with multi-petabyte storage instances and workloads. 

Want to learn more or see if it’s the right fit for your team? Get in touch with our Sales team—we’d love to talk about how we can help you ditch the trade-offs and go full speed ahead.

The post Meet B2 Overdrive: Terabit-Speed Throughput for AI/ML and HPC Workloads appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Reinforcing resilience with financial assurance: Breach protection matters now more than ever

Post Syndicated from Cindy Stanton original https://blog.rapid7.com/2025/04/29/reinforcing-resilience-with-financial-assurance-breach-protection-matters-now-more-than-ever/

Introducing Rapid7’s value-added Breach Protection Warranty that delivers confidence, clarity, and coverage when it matters most.

Reinforcing resilience with financial assurance: Breach protection matters now more than ever

Life’s old adage often applies in security: Hope for the best, prepare for the worst. In today’s threat landscape, even the best-prepared organizations can’t guarantee immunity from cyberattacks. The cost of a breach is no longer just a line item—it’s a business risk with board-level visibility. In 2024, the average total cost of a breach soared to $4.88 million, a record high and a 10% increase from the previous year​.1

As threats grow in complexity and breach response becomes more expensive and unpredictable, security leaders aren’t just investing in detection and response—they’re looking for assurance that they are, in fact, prepared for the worst. That’s why Rapid7 is introducing its Breach Protection Warranty: real-world financial coverage, built directly into our flagship Managed Detection and Response (MDR) offering, Managed Threat Complete Ultimate. This coverage is designed to give customers confidence that if that dreaded day comes, they’re ready.

Rapid7 continues to invest heavily in this industry-leading service—trusted by thousands of customers worldwide—which processes trillions of events and investigates millions of alerts annually. Leveraging this immense scale to ensure robust threat detection and rapid response, the results speak for themselves: 99.6% of our MDR customers remain unaffected by ransomware. But beyond numbers, our commitment is clear—we’re dedicated to continuously enhancing MDR capabilities and standing shoulder-to-shoulder with our customers to safeguard their operations.

Built-in financial protection, when it counts

The Rapid7 Breach Protection Warranty provides up to $1 million in breach-related coverage, based on the size of a customer’s environment. It’s designed to offset the real-world costs organizations face in the wake of a cyberattack, including:

  • Forensic investigation expenses
  • Legal consultation fees
  • Public relations costs
  • Post-security incident expenses

Unlike standalone breach coverage or third-party insurance policies, this warranty comes at no additional cost to eligible customers. There are no upsells or hidden fees—just value-added protection that’s already embedded in the service​.

Integrated into a world-class detection and response program

Rapid7’s holistic and outcome-driven approach to detection and response includes unlimited digital forensics and incident response (DFIR), remote containment, and active remediation of commodity malware. All of these capabilities are amplified by the Rapid7 Command Platform, which seamlessly integrates vulnerability findings and threat intelligence to deliver complete coverage across the entire security incident lifecycle.

Together with the Breach Protection Warranty, this cohesive model transforms your cybersecurity program into one not just backed by technology and expertise, but also a financial safety net built into the service.

Simplifying breach response, not complicating it

A warranty is, at its core, a legal agreement—and it’s understandable that many come with conditions or “strings attached.” Some require customers to exhaust other forms of insurance before coverage kicks in. Others recapture the financial benefit through billable incident response services, which can quietly reduce the actual value received.

Rapid7 takes a different approach. We’ve built the Breach Protection Warranty to maximize customer value with transparency. There are no hidden clauses designed to funnel reimbursement back to us. And because unlimited incident response (IR) is already included in the service, customers don’t need to worry about separate IR contracts or an unexpected changing of the guard in the frenetic aftermath of an incident​.

Strengthening resilience through readiness

While financial protection is the headline, eligibility for the warranty reinforces the fundamentals of a strong security posture. Customers must meet a set of best-practice requirements that align with Rapid7’s proven approach to resilience, including but not limited to:

  • Hardened endpoint configurations
  • Deployment of core protection modules such as Ransomware Prevention
  • Updated, compliant operating systems and software across covered assets​

These aren’t just checkboxes—they’re meaningful controls that improve visibility, reduce risk, and better prepare customers to prevent and respond to threats; with the added benefit of unlocking a meaningful financial backstop.

Ready to learn more?

This is about more than cost coverage. It’s about trust—trust that your security investments are driving true resilience and that your provider is prepared to stand beside you when it matters most.

Rapid7’s Breach Protection Warranty is now available to all Managed Threat Complete Ultimate customers. To learn more about your eligibility or sign up, please reach out to your Rapid7 account team.

1 IBM

InsightIDR AI Alert Triage Automatically Classifies Alerts with 99.93% Accuracy

Post Syndicated from Chris Wraight original https://blog.rapid7.com/2025/04/29/insightidr-ai-alert-triage-automatically-classifies-alerts-with-99-93-accuracy/

Rapid7 AI Alert Triage helps SOC analysts quickly and accurately triage thousands of daily alerts, improving efficiency and enabling focus.

InsightIDR AI Alert Triage Automatically Classifies Alerts with 99.93% Accuracy

One universal truth in Security Operations Centers (SOCs) is that analysts are overwhelmed by the high volume of alerts they receive. In a recent survey, SOC teams reported they are inundated with an average of 4,484 alerts daily, with a staggering 67% being ignored due to alert fatigue and the high volume of false positives. Also reported in the same survey, the number of security alerts they received had “significantly increased” in the last three years. All this can lead to alert fatigue, resulting in missed or ignored alerts and potentially exposing an organization to legitimate threats, impacting SOC performance.

Introducing AI Alert Triage for InsightIDR

Rapid7’s AI Alert Triage –  trained and tested by the Rapid7 global MDR service across trillions of alerts worldwide — will soon be available to users of our next-gen SIEM, InsightIDR, at no additional cost. The AI Alert Triage engine quickly suggests an initial disposition (benign or malicious) for alerts, providing clarity into why that disposition was chosen and supporting information from the investigation.

InsightIDR AI Alert Triage Automatically Classifies Alerts with 99.93% Accuracy
The new AI Suggested Disposition field shows the alert classification, along with detailed information to assist the SOC analyst.

Without access to the Rapid7 AI Alert Triage capability, SOC teams can waste significant time manually evaluating and correctly classifying malicious alerts, increasing their threat exposure and contributing to SOC inefficiency. With AI Alert Triage, SOC analysts can automatically and accurately focus limited security resources on legitimate threats and improve SOC performance.

Built on Decades of AI Expertise in Security

Rapid7 is not new to infusing AI with its security applications. Rapid7 is a pioneer in AI development for security use cases, starting in our earliest days with our VM Expert System in the early 2000s. Since then, Rapid7 has integrated Generative AI into the Command Platform to supercharge SecOps and augment MDR services.

Our AI-powered platform processes 4.8 trillion alerts weekly for our MDR customer base with a 99.93% benign alert closure rate. This has resulted in hundreds of hours of manual effort saved for the SOC analysts.

AI Alert Triage Improves Your SOC’s Effectiveness

InsightIDR AI Alert Triage Automatically Classifies Alerts with 99.93% Accuracy
Having thousands of daily alerts automatically and correctly classified is a huge productivity boost for overwhelmed SOC analysts.

SOC analysts being overwhelmed by high volumes of daily alerts is hardly a new phenomenon. However, up until now, SOC analysts have been unable to effectively deal with this massive number of alerts, and organizations have become victims of alert fatigue. The same survey referenced above reports that analysts spend nearly 3 hours (2.7) each day manually triaging alerts, a figure rising to more than 4 hours a day for 27% of respondents. And, on average, security analysts are unable to deal with over two-thirds (67%) of the daily alerts they receive. What’s more, they say 83% of these alerts are false positives and not worth their time.

By using AI Alert Triage, Rapid7 customers can leverage decades of proven Rapid7 AI technology to quickly and accurately classify the deluge of alerts and not be forced into a situation where they intentionally ignore alerts. Key capabilities include:

  • Rapid identification and prioritization of genuine threats: AI Alert Triage helps customers quickly distinguish true positives from noise, enabling security teams to prioritize investigations based on validated, high-confidence alerts.
  • Enhanced Threat Detection Speed and Accuracy: Leveraging MDR-validated AI ensures alerts reflect real threats, helping SOC teams respond swiftly and confidently to advanced threats and subtle attack indicators.
  • Human oversight of automatic classification: AI Alert Triage has attained a 99.93% benign alert closure rate with nearly 5 trillion weekly alerts; every alert is documented, with full transparency and opportunity for human intervention.
  • Reduced Alert Fatigue and False Positives: With AI Alert Triage validated by MDR analysts, customers experience dramatically reduced false positives, significantly cutting down time wasted on non-critical alerts.
  • Streamlined Workflows and Focus: AI Alert Triage automates repetitive tasks to streamline initial analysis, enabling security teams to jumpstart investigations and dedicate more time and resources to critical initiatives.

Transform Your SOC Performance Through Proven AI Assistance

Unmatched 99.93% accuracy and speed in Rapid7’s AI models drive trust and confidence in automatic decision-making and reduce alert fatigue for SOC analysts, improving SOC performance and speed. At Rapid7, we are pioneering the infusion of artificial intelligence into the Command platform, empowering  SOCs around the globe and dramatically transforming their effectiveness through GenAI.

To learn more about AI Alert Triage, contact your account team or your Customer Success Advisor.

Deepening the MDR partnership: Rapid7 now delivers Active Remediation with Velociraptor

Post Syndicated from Conner Goldstein original https://blog.rapid7.com/2025/04/29/deepening-the-mdr-partnership-rapid7-now-delivers-active-remediation-with-velociraptor/

Rapid7 is expanding its response capabilities to meet the demands and relentless pace of today’s threat landscape – and the operational needs of our customers.

Deepening the MDR partnership: Rapid7 now delivers Active Remediation with Velociraptor

Partnership means many things to us here at Rapid7. It means showing up with trusted expertise, providing clear guidance in moments of uncertainty, and helping security teams stay ahead of ever-evolving threats. Most of all, we see partnership as foundational to building security resilience – and that requires not only a proactive,  risk-aware mindset but also the capability to respond when the inevitable happens.

As attacks grow faster, more complex, and more persistent, the need for decisive, transparent remediation has become more urgent – some estimates place the average time-to-ransom at just 16.88 hours1 –  with that kind of speed, every moment matters. We pride our Managed Detection and Response (MDR) service on delivering best-in-class detection, investigation, and actionable response guidance. Now, we are evolving that partnership – and the strength of your security program – even further.

Introducing Active Remediation with Velociraptor

Powered by our best-in-class, open-source digital forensics and incident response (DFIR) tool, Rapid7 MDR analysts can take direct, approved remediation actions on your behalf – removing malware, terminating rogue processes, and restoring system integrity while minimizing the need to reimage affected endpoints unless it’s truly required. Every action is executed with precision, transparency, and within clearly defined boundaries.

This is more than a new capability. It’s a reflection of our commitment to move in lockstep with you – not just at the point of detection, but all the way through to resolution. From unlimited incident response support to deeply collaborative investigations and tailored recommendations, Rapid7 has always prioritized being hands-on when you need us most. Active Remediation with Velociraptor extends that same principle to the final – and often most difficult – step: taking action on your behalf to eradicate threats.

Delivered with Precision, Transparency, and Trust

Active Remediation with Velociraptor is designed not just to take action, but to take the right action, the right way. Every remediation workflow is executed by Rapid7’s expert analysts using Velociraptor’s purpose-built query language (VQL) – a DFIR language engineered for precision, traceability, and scale. This allows the analyst to target specific artifacts, processes, and configurations – avoiding the blunt-force actions that often lead to full endpoint reimaging.

  • You stay in control – Remediation is performed based on clearly defined and approved scopes and parameters aligned to your security policies.
  • You see what we do – Every action is logged, auditable, and built using readable logic within Velociraptor, with full visibility provided through detailed post-incident reports.
  • You gain precision without disruption – Remove only what’s malicious and reverse unauthorized configurations without pulling systems offline or fully reimaging machines.

Rapid7’s New Response Workflow

Deepening the MDR partnership: Rapid7 now delivers Active Remediation with Velociraptor
  1. Alert detection: Identify malicious activity across customer endpoints and network.
  2. Active Response: Quarantine affected endpoints to stem the spread of the attack.
  3. Rapid7 investigation: SOC validates threat, determines scope, and develops response plan.
  4. Active Remediation with Velociraptor: Rapid7 analysts remove malicious artifacts with precision.
  5. Mitigation guidance: Recommendations to help your team prevent threat reemergence.

Remediating in the Real World

Our approach brings analyst-led, logic-driven remediation into live environments – solving the post-containment challenges security teams face every day. Unlike session-based access that relies on endpoints being on and connected to the internet, Rapid7 are delivering remediation that meets the auditability, practicality, and scalability needs of the real world:

  • Targeted threat removal without reimaging: Identify and remove only malicious artifacts – files, processes, persistence mechanisms, or unauthorized configurations – linked to a confirmed threat.
  • Outcome: Your endpoints stay online and productive, while the threat is neutralized with minimal disruption. Avoiding unnecessary reimaging means faster recovery, reduced IT workload, and less downtime for end users.
  • Controlled execution with transparent logic: Every remediation workflow is written in VQL – visible and reviewable by customers before deployment. There’s no scripting or ‘trust us’ execution.
  • Outcome: Builds trust and accountability into the remediation process. You get full visibility into every action, supporting compliance requirements and reducing uncertainty in regulated environments.
  • Distributed remediation across endpoints: When multiple endpoints are compromised by a single campaign – such as credential theft malware – we will queue high-fidelity remediation workflows across many machines simultaneously – even if some are offline.
  • Outcome: Lays the foundation for consistent threat removal across your environment without manual intervention or system-by-system cleanup. This enables a timely, coordinated response that keeps pace with fast-moving attacks.
  • Reducing friction between security and IT teams
    Rather than working through lengthy remediation steps with your IT team, we execute the most critical actions directly – within approved scope – and document every step.
  • Outcome: Fewer delays and less back-and-forth between teams. With Rapid7 handling the complete, end-to-end lifecycle of an alert, internal teams stay focused on business priorities, knowing remediation is being executed safely and effectively.

Setting the stage for remediation with Active Response

Remediation begins with strategic containment and detailed investigation. Rapid7’s Active Response enables rules-based quarantining of affected endpoints in the immediate aftermath of a credible threat detection. This stops lateral movement before it begins and preserves the system state for investigation.

Active Remediation builds directly on this foundation. By first containing the threat, we can then investigate confidently and move quickly to identify and remove malicious artifacts – mitigating the risk of reinfection or spread. The integrated workflow – from containment to investigation to remediation – helps ensure our response is not only fast, but precise.

Together, Active Response and Active Remediation form the cornerstone of a continuous response pipeline that reduces attacker dwell time, limits impact, and restores normal operations faster.

Unlimited incident response – now deeper than ever

Rapid7 MDR customers have long relied on Rapid7’s unlimited DFIR support to guide them through the most critical moments of a threat. That hands-on expertise – delivered without surprise fees, hourly caps, or the need to navigate third-party providers and tools – is a defining part of how we ensure customers receive the fastest, most comprehensive response possible.

Active Remediation builds on that foundation by closing the final gap in the response lifecycle. Where detection, containment, and investigation have long been Rapid7’s strengths, we can now fully execute on the next step: resolving the threat. This combination of expert-led triage with decisive, hands-on remediation, delivers a more unified, end-to-end response – minimizing delays, reducing reliance on your internal resources, and accelerating your path to recovery. It’s not just about reacting faster – it’s about responding smarter, start to finish.

More than a capability – it’s a commitment

The Rapid7 MDR service has always been built around standing shoulder to shoulder with our customers, especially when it matters most. As we expand this partnership through the finale of the detection and response lifecycle – taking action to remove threats, reduce disruption, and accelerate recovery – we do it with the same transparency, accountability, and respect for your control that defines every part of the Rapid7 experience. In the name of building true security resilience, partnership doesn’t end with guidance – it means staying with you all the way through to resolution. Active Remediation with Velociraptor is in closed early access and will roll out to MDR customers in mid-May. To learn more, please contact your account team or Cybersecurity Advisor.

1 gbhackers

[$] Cache awareness for the CPU scheduler

Post Syndicated from corbet original https://lwn.net/Articles/1018334/

The kernel’s CPU scheduler has to balance a wide range of objectives. The
tasks in the system must be scheduled fairly, with latency for any given
task kept within bounds. All of the CPUs in the system should be kept busy
if there is enough work to do, but unneeded CPUs should be shut down to
reduce power consumption. A task should also run on the CPU that is most
likely to have cached the memory that task is using. This patch
series from Chen Yu aims to improve how the scheduler handles cache
locality for multi-threaded processes.

Driving down MTTR with Remediation Hub, Available in Rapid7 Exposure Command

Post Syndicated from Rapid7 original https://blog.rapid7.com/2025/04/29/driving-down-mttr-with-remediation-hub-available-in-rapid7-exposure-command/

Driving down MTTR with Remediation Hub, Available in Rapid7 Exposure Command

Co-authored by Peter Whibley, Ed Montgomery, and Joel Alcon

Technology innovation combined with the highly fragmented nature of today’s IT landscape means that vulnerabilities are being exploited faster and at greater scale than ever. Security teams contend with a daily surge of new threat actors and attack vectors. Without a unified view of assets, business context, and compensating controls, they waste weeks identifying which risks are truly critical.

Many organizations try to tackle this challenge by implementing exposure management  and risk-based vulnerability management (RBVM) approaches, where vulnerability data from various tools is consolidated into one dashboard. But many of these tools present risk scores without demonstrating a holistic view of the business impact of vulnerabilities, mitigating controls for endpoints, patch management status, and remediation steps.

Without that end-to-end context, security teams are struggling to keep up with the volume of new vulnerabilities. In fact, once the National Vulnerability Database (NVD) announced in February 2024 that it would no longer provide vulnerability scores for all CVEs, the shortcomings of traditional vulnerability management, including RBVM, became more evident.

From chasing vulnerabilities, to proactively mitigating risk

Rapid7’s Remediation Hub enables security teams to go beyond simply identifying vulnerabilities and focus more on remediating risk. By augmenting vulnerability findings with business context, threat intelligence, and compensating controls, organizations gain a continuous, all-in-one view of how to detect and respond to risks across their enterprise. These new capabilities empower security teams to:

  1. Assess the impact of remediation steps. Reimagine your attack surface by viewing the number of vulnerabilities addressed by each remediation action.
  2. Prioritize remediation with confidence. Leverage dynamic, threat-aware risk scores to assess the criticality of issues and quickly go from vulnerability to action.
  3. Optimize risk mitigation. Accelerate risk response through streamlined remediation workflows.

Third-party vulnerability findings elevate risk remediation

Security teams leverage multiple vulnerability scanning tools for different parts of their infrastructure, including cloud environments, containers, web applications, and endpoints. Each tool reports findings in its own format and utilizes different scoring methods, making it difficult to get a clear, unified picture of an organization’s risk exposure.

By unifying this data into a centralized platform, security teams reduce unnecessary noise caused by redundant vulnerability findings, streamlining triage efforts, reducing silos, and driving faster, more informed remediation efforts.

Driving down MTTR with Remediation Hub, Available in Rapid7 Exposure Command

Rapid7 Remediation Hub delivers this normalized view of third-party vulnerabilities, enabling teams to stop wasting time chasing low-impact issues or overlook high-severity threats. The solution takes this unified lens further via risk scores that combine these vulnerability findings with business context to help security teams quickly identify the most critical vulnerabilities, allocate resources efficiently, and communicate risk more effectively to stakeholders. These capabilities not only boost operational efficiency, but also strengthen an organization’s security posture.

Context-based visibility into endpoint protection and patch management

Context is an essential component of managing risk in today’s increasingly complex technology landscape. By solely relying on vulnerability scores without also understanding business impact or breach likelihood, security teams are left with a hazy, incomplete view of their attack surface.

Rapid7 Exposure Command empowers security teams to prioritize vulnerabilities based on attacker behavior, exploitability, and potential impact – all without the need to export data into separate security tools. Rapid7 delivers deep, multi-layered risk scores calculated from Rapid7 Labs’s threat intelligence, first-party scans, third-party vulnerability findings, and an organization’s unique mitigating controls. Furthermore, Remediation Hub is seamlessly integrated with Rapid7 Surface Command, arming security teams with a continuous view of key mitigating controls of assets across the enterprise, including endpoint protection and patch management in place.

Driving down MTTR with Remediation Hub, Available in Rapid7 Exposure Command
  • Endpoint protection – Remediation Hub displays which assets have active endpoint protection, as well as the protection type on the asset. Users can use intuitive filters to hone in on critical findings, such as the assets that lack endpoint protection and prioritize remediation efforts via a risk-based approach that gives higher priority to assets that lack endpoint protection.
  • Patch management – Remediation Hub shows the patch management availability status of each asset, arming security teams with a view of assets that are available for patching by a patch management system. Users can filter on assets with vulnerabilities where no patching is active.

Faster risk response, fewer security silos

Security teams often operate in silos, with a team handling risk identification and another focused on remediation. CISA recommends that critical vulnerabilities be remediated within 15 calendar days of initial detection, but to achieve this, organizations require tight collaboration between these disparate teams.

Unfortunately, because these groups operate with poorly integrated security tools, going from vulnerability finding to risk remediation can take months, with some vulnerabilities going unpatched for years. For instance, the 2024 Verizon Data Breach Investigations Report finds that it takes an estimated 55 days to remediate 50% of critical vulnerabilities once their patches are available.

Driving down MTTR with Remediation Hub, Available in Rapid7 Exposure Command

Remediation Hub tackles this challenge with purpose-built SOAR integrations that help improve collaboration and drive down MTTR (mean time to remediate). The new capabilities automatically trigger remediation workflows, with notifications auto-generated and sent to adjacent teams responsible for implementing the recommended remediations.

For example, users can leverage Remediation Hub to automatically trigger a workflow in Jira or create an incident report in ServiceNow based on the severity or business impact of a vulnerability. Each workflow is fully customizable based on unique security thresholds.

Embracing faster, continuous exposure management

Organizations are rapidly transitioning from traditional vulnerability management to more continuous, exposure management approaches. Rapid7’s Remediation Hub – an integral component of the Exposure Command platform – empowers security teams to embrace the shift.

With a remediation-based approach to vulnerability management and risk reduction, organizations are taking command of their attack surface and discovering a simpler, more effective approach to managing and truly mitigating risk.

If you are interested in learning more about Remediation Hub and our Exposure Command platform, check out our Exposure Command product tour.

From Exposure to Assurance: Unified Remediation Across the Security Lifecycle

Post Syndicated from Rapid7 original https://blog.rapid7.com/2025/04/29/from-exposure-to-assurance-unified-remediation-across-the-security-lifecycle/

From Exposure to Assurance: Unified Remediation Across the Security Lifecycle

When it comes to defending your organization, every second counts. The time to detect, respond, and remediate is critical, but speed alone isn’t enough. Fragmentation across security tools, siloed teams, and manual workflows leaves organizations constantly reactive, overwhelmed by alerts, and at risk of breaches. Rapid7 is here to change that.

Organizations need solutions that unify their approach, streamline processes, and accelerate response times. Rapid7 delivers the industry’s broadest, most unified view of the attack and detection surface. Today, we’re thrilled to announce a series of strategic launches that further this integrated approach and deliver unified remediation across the full breach timeline, integrating proactive exposure management with intelligent detection and automated response. This comprehensive approach provides security teams with the precise tools and deep insights needed to effectively secure their organization and shift from proactively reducing vulnerabilities to swiftly resolving active threats.

From Exposure to Assurance: Unified Remediation Across the Security Lifecycle

Left of Boom: Proactive Exposure Remediation

The most effective security strategy begins before a breach ever happens. Rapid7’s Exposure Command directly addresses this gap, combining advanced risk-based vulnerability management (RBVM) with environmental context, threat intelligence, and native workflow automation.

Launching this week at RSA, we’re excited to announce a trio of updates to Remediation Hub aimed at helping organizations unify and modernize their vulnerability management programs:

  • Enhanced Automated Remediation Workflows: We’ve significantly expanded our workflow automation capabilities to streamline exposure remediation. Users can now easily launch both pre-built and fully customizable remediation workflows—including notifications, ticketing, and patch deployment—directly from the intuitive Remediation Hub interface. This seamless integration simplifies the remediation process, allowing teams to swiftly address vulnerabilities and maintain robust security hygiene.
  • Advanced Compensating Controls Assessment: Remediation Hub now provides comprehensive insights into existing compensating controls, empowering teams to strategically deprioritize vulnerabilities that present minimal or no practical risk due to limited accessibility or exploitability—such as a compromised asset running antivirus or behavior prevention. This enhanced visibility is particularly vital for managing unpatchable workloads or addressing vulnerabilities where patches or permanent fixes are currently unavailable.
  • Expanded Third-Party Vulnerability Integration: Exposure Command has always integrated valuable telemetry from third-party vulnerability scanners such as Tenable, Qualys, and Wiz. Now, we’ve enhanced this capability by incorporating vulnerability findings and detailed risk scoring directly into the Remediation Hub. This allows vulnerabilities identified from any 3rd-party integration to be effectively prioritized using Active Risk assessments and effortlessly embedded into your team’s existing remediation and patch management workflows, streamlining vulnerability management across diverse scanning solutions.

With these new enhancements to Remediation Hub, security teams are empowered with a real-time, validated understanding of exposures enriched with business context, adversary intelligence, and insight into existing compensating controls, not just a list of CVEs. And because the Exposure Command platform brings together native scanning from Rapid7 and vulnerability findings from third-party tools, teams can prioritize vulnerabilities based on attacker behavior, exploitability, and potential impact without spending valuable time porting data into separate tools.

Instead of just alerting your team to a vulnerability, Exposure Command helps you own the risk conversation with the business by aligning on what matters most to the business, the risks already addressed, and outlining a path to closing any remaining gaps. Security teams no longer have to guess which vulnerabilities pose the most risk; instead, they can proactively remediate with certainty, preventing vulnerabilities from escalating into incidents.

Right of Boom: Intelligent Detection, Confident Response, and Financial Assurance

Despite best efforts, security incidents and breaches are ever-present. To reduce their impact and the cost of remediation, security teams need rapid, intelligent responses to evolving incidents, helping them to prioritize and triage, and leverage automation to reduce the volume of potential investigations, and improve their ability to scale to meet remediation tasks. This is why Rapid7 is focusing efforts around building in support post event, marking a significant shift in our capabilities to remediate malicious attacker behavior:

  • AI Triage and Transparency within InsightIDR: Rapid7 was a pioneer in AI development for security use cases, starting in our earliest days with our VM Expert System in the early 2000s. Since then, Rapid7 has integrated Generative AI into the Command Platform to supercharge SecOps and augment MDR services. This has culminated in Rapid7’s AI-Assisted Triage delivering industry-leading precision, accurately distinguishing critical threats from benign alerts with a 99.89% accuracy rate.  Without access to the Rapid7 AI Alert Triage capability, SOC teams waste significant time manually evaluating and correctly classifying malicious alerts, increasing their threat exposure and contributing to SOC inefficiency. With AI Alert Triage, SOC analysts can automatically and accurately focus limited security resources on legitimate threats and improve SOC performance.
  • Active Remediation with Velociraptor: The response capabilities of the Rapid7 SOC have expanded to include the swift and precise removal of malware and breach artifacts from impacted endpoints. This progression beyond remote containment and guided remediation represents a significant deepening of the MDR partnership between Rapid7 and customers. It relieves security teams not only from the burden of coordinating remediation actions with IT teams, but also helps preserve endpoint integrity, reduce downtime, and avoid unnecessary endpoint rebuilds. With real-time remediation capabilities, the Command Platform links actions directly back to known vulnerabilities, providing valuable context for future prevention and significantly shortening incident response cycles.
  • Breach Protection Warranty: Investing in security solutions is about more than technology and expert service delivery. It’s about guaranteed results and peace of mind. The Rapid7 SOC analyzes trillions of events each year, and 99.6% of MDR customers remain unaffected by ransomware. Recognizing this, and reinforcing our commitment to ensuring cybersecurity resilience, customers in our premium tier, Managed Threat Complete Ultimate, will now receive up to $1 million in breach-related financial coverage through our Breach Protection Warranty. This represents a tangible demonstration of our confidence in our solutions and our commitment to protecting your organization’s critical assets while also assuring you that, in the unlikely event of a compromise, we are right there by your side.

As our detection and response capabilities continue to expand, we’re pushing to deliver smarter, faster, and more complete security outcomes for our customers. With alert fatigue diminished through precise AI-Assisted Alert Triage, security analysts can spend more time on validated threats and strategic initiatives to enhance organizational posture. The expansion of Rapid7’s response workflow to include remediation redefines effective response while ensuring customer visibility and control. And now, our Breach Protection Warranty offers up to $1 million in breach-related financial coverage: we’re not just preventing and helping you recover from threats, we’re standing behind our ability to do so. Together, these capabilities mark a meaningful shift in how Rapid7 supports customers post-incident: with intelligence, speed, and confidence that extends all the way through recovery.

One Connected Journey, End-to-End

Cybersecurity incidents are complex, evolving threats requiring seamless integration of proactive and reactive security measures. Rapid7’s Command Platform bridges the traditional divides between proactive vulnerability management, intelligent threat detection, and automated incident remediation. With a unified, continuous security lifecycle, your organization can remain agile, informed, and resilient against emerging threats.

Take your cybersecurity posture to the next level. Discover how Rapid7’s unified remediation strategy delivers measurable results and helps secure your organization effectively against breaches. Interested in learning more about how Rapid7’s unified remediation strategy can transform your organization’s security posture? Learn more here.

Getting Started with Zabbix – Hosts, Items, and Triggers

Post Syndicated from Arturs Lontons original https://blog.zabbix.com/getting-started-with-zabbix-hosts-items-and-triggers/30190/

Hosts, items, and triggers are some of the most basic concepts in Zabbix. To successfully configure their monitoring workflows, Zabbix users need to have a clear understanding of how these entities are used. This article is aimed at Zabbix beginners and should help anyone better understand the basics of Zabbix while providing guidance on how to start monitoring your initial set of hosts.

Table of Contents

Hosts

Hosts are top-level entities in Zabbix and represent your monitored endpoints. Whenever we need to monitor a device, web application, service, or anything else – we start by creating a host.

The host acts as a container for our items (representing the metrics we wish to collect) and triggers (problem threshold definitions). These entities can be created directly on the host or inherited from predefined templates.

Every host has 2 mandatory parameters – its unique name and at least a single host group. Host groups are used for grouping, filtering, and assigning read/write permissions to hosts. Hosts are not limited when it comes to the number of host groups they are assigned to.

A simple Linux server host with an agent interface and a Linux template

An interface might also be required, depending on the type of items we will create on the host. Interfaces define host addresses and, in case of SNMP interfaces, some additional authentication and security parameters.

There are 4 types of interfaces in total, representing 4 different data collection methods:

  • Agent
  • SNMP
  • JMX
  • IPMI
An SNMP device host with SNMP interface

Zabbix supports other types of data collection methods, but for these 4 methods in particular an interface is required on the host. Other data collection methods define endpoint addresses directly in the item configuration or use push data collection (trapping) where Zabbix is not required to know the endpoint address.

Templates

Templates contain a set of predefined items and triggers and can be linked to hosts. This enables the standardization of monitoring workflows in your environment. Changes made on the template will be immediately applied on the hosts to which the template is linked. Zabbix comes prepackaged with over 300 templates for a variety of vendors and endpoint types.

Zabbix comes pre-packaged with over 300 official templates

Zabbix users aren’t limited to just the official templates – anyone can create their own templates with items and triggers tailored to the requirements of a particular environment. We also recommend adjusting the official templates – disable the unnecessary items and adjust the triggers so they don’t generate any unnecessary noise.

Items

Items are used to define the metrics that we wish to collect, and are configured on hosts or templates. Items can be of various types. The type of the item usually defines the protocol and the methods used to collect metrics via this item. Some examples of item types:

  • Zabbix agent
  • SNMP agent
  • SNMP trap
  • Simple check
  • HTTP agent
  • IPMI agent
  • JMX agent
  • SSH agent
  • …and many others.

The key of the item is used to specify what particular metric should be collected. There are some exceptions to this – for example, for SNMP agent items it’s the OID field, while the key can be written arbitrarily. The key should be unique per host.

Available memory Zabbix agent item

The key uses a <key>[<parameters>] format. For example, if we wish to collect available memory by utilizing Zabbix agent, we will use the vm.memory.size[available] item key. If we wish to collect available memory in percent, we would use the vm.memory.size[pavailable] item key. A quick item key reference is available by pressing select next to the Key field. You can find more about the available item keys and other configuration details in our documentation.

The update interval specifies how often metrics should be collected for this key, and the history/trend storage periods define for how long the collected data should be retained.

Triggers

Once we have configured our items, we should create triggers to react to item values reaching problem thresholds. First, let’s define a simple trigger name. The name should be simple enough for our Zabbix administrators to understand the goal of the trigger simply by glancing at it.

Trigger reaction to low available memory over the last 10 minutes

The event name field is used to define the name with which our problems will be displayed. Since the problem event name is often used not just in Zabbix but also in the alerts that your administrators will receive in their mailboxes or via messaging and ITSM systems, the event name should be more descriptive, giving general details about the problematic situation.

Operational data fields are used to display information about the current state of items analyzed by the trigger. By default, the field will display the current value of our item (available memory, for example). This allows users to compare the current item values with item values at the time of problem creation and decide if any additional interference is necessary to resolve the problem.

The expression field defines the logic behind detecting a problem. Here, we can either type in the expression manually or press the add button and build the expression by selecting the item that we wish to analyze – plus one of the various functions used for analysis. For example, the last function is used to analyze only the last received value and can generate a lot of noise when used for resource monitoring. Meanwhile, average, minimum, and maximum functions can be used to analyze values less sensitively over time. There are many more functions available for a variety of more advanced use cases – from string analysis functions to predictive functions and many others.

A large selection of functions can be used in trigger expressions to detect problems

Once the trigger is created, it will be recalculated every time any of the related items receive a new value.

This article covered only the basics of Host, item and trigger configuration. There are many more options for more advanced use cases. If you’re interested or need help with more advanced Zabbix features, please check out a variety of tutorials, how-tos and case studies in our blog and YouTube channel.

The post Getting Started with Zabbix – Hosts, Items, and Triggers appeared first on Zabbix Blog.

Signing key change for Kali Linux

Post Syndicated from corbet original https://lwn.net/Articles/1019255/

The Kali Linux distribution has announced
that software updates will soon start failing for all users:

This is not only you, this is for everyone, and this is entirely
our fault. We lost access to the signing key of the repository, so
we had to create a new one. At the same time, we froze the
repository (you might have noticed that there was no update since
Friday 18th), so nobody was impacted yet. But we’re going to
unfreeze the repository this week, and it’s now signed with the new
key.

The announcement includes instructions for how to recover from the problem.

Security updates for Tuesday

Post Syndicated from corbet original https://lwn.net/Articles/1019272/

Security updates have been issued by AlmaLinux (glibc, php:8.1, and thunderbird), Debian (libreoffice), Fedora (caddy), Mageia (chromium-browser-stable), Red Hat (php:8.1), SUSE (glow), and Ubuntu (kicad, linux-aws-5.15, linux-azure-nvidia, linux-gcp-5.15, mistral, python-mistral-lib, tomcat8, and trafficserver).

Applying Security Engineering to Prompt Injection Security

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/04/applying-security-engineering-to-prompt-injection-security.html

This seems like an important advance in LLM security against prompt injection:

Google DeepMind has unveiled CaMeL (CApabilities for MachinE Learning), a new approach to stopping prompt-injection attacks that abandons the failed strategy of having AI models police themselves. Instead, CaMeL treats language models as fundamentally untrusted components within a secure software framework, creating clear boundaries between user commands and potentially malicious content.

[…]

To understand CaMeL, you need to understand that prompt injections happen when AI systems can’t distinguish between legitimate user commands and malicious instructions hidden in content they’re processing.

[…]

While CaMeL does use multiple AI models (a privileged LLM and a quarantined LLM), what makes it innovative isn’t reducing the number of models but fundamentally changing the security architecture. Rather than expecting AI to detect attacks, CaMeL implements established security engineering principles like capability-based access control and data flow tracking to create boundaries that remain effective even if an AI component is compromised.

Research paper. Good analysis by Simon Willison.

I wrote about the problem of LLMs intermingling the data and control paths here.

Should we ditch the term ‘digital native’? The latest conversation on the podcast

Post Syndicated from Meg Wang original https://www.raspberrypi.org/blog/should-we-ditch-the-term-digital-native-the-latest-conversation-on-the-podcast/

The term ‘digital native’ was coined in 2001 to describe young people who grew up surrounded by technology, implying they possess an almost instinctive understanding of digital tools. But how accurate is that label today? Are they truly tech-savvy, or are they simply fluent in using user-friendly devices without a deeper understanding of how they work? 

Thumbnail titled 'Are young people really tech-savvy?'

In our latest podcast episode, we explore this crucial question: does growing up with technology automatically make someone digitally literate, or just digitally dependent?

Host James Robinson (Raspberry Pi Foundation) is joined by colleague Sway Grantham, Anna Lahtinen (Haaga-Helia University), and Nicole O’Connor (Digi Know This) to unpack the complexities of digital skills, access, and education, and to explore how the label of digital native can sometimes hide broader challenges.

This episode wraps up our three-part mini-series on digital literacy. Each episode builds on the conversations from the latest issue of Hello World magazine, offering expert insight and practical ideas from educators driving real change.

Who are the guests on the podcast, and what will I learn?

We’ve assembled a panel of expert guests to discuss and debate the concept of the digital native.

Square headshot of Anna Lahtinen who featured on the Hello World podcast.
Anna Lahtinen, Haaga-Helia University of Applied Sciences

Anna is an internationally recognised scholar who specialises in the transformative effects of artificial intelligence (AI) on working life, businesses, and careers. In the latest issue of Hello World, Anna and her colleague Antonia O’Connell write about assumptions regarding high digital literacy among youth in the setting of a work environment.

In the podcast, Anna explains that the term ‘digital native’ can be useful for starting conversations about the skills, access, and confidence people need to navigate technology, particularly in the workplace. However, she warns that the term can be misleading, as it often assumes that young people have strong digital skills, when many struggle with the practical demands of using technology at work.

Anna also highlights the importance of supporting young people to use technology responsibly and develop a healthy relationship with it. As she puts it:

“If we instil the right values in students, that will eventually lead to a healthy relationship with technology and ethical use of new tools in both their work and their lives.”

Square headshot of Nicole O'Connor who featured on the Hello World podcast.
Nicole O’Connor, Digi Know This

Nicole O’Connor is a two-time female founder who brings her expertise to improving teaching and learning through innovative EdTech, digital literacy, and accessibility.

In the podcast, Nicole challenges the idea of the digital native, calling it both problematic and overly simplistic. She explains that the term overlooks important differences, like access, privilege, and levels of exposure to technology, which all shape a person’s digital skills. For Nicole, it’s a “grey area” that proves we can’t assume young people are automatically tech-savvy just because they grew up with devices.

Nicole sees digital literacy as a journey, not something people are born with. She compares it to learning a language: while young people may be familiar with technology, they still need to develop deeper skills to become critical thinkers, careful users, and aware of issues like data privacy and online safety. As she puts it:

“They might have had some exposure, but over time they need to learn the skills to be digitally literate… They’re not even close to that.”

Listen or watch now

To hear more, listen to or watch the full episode.

Listen or watch here

You can watch or listen to each episode of our podcast on YouTube, or listen via your preferred audio streaming service, whether that’s Apple Podcasts, Spotify, or Amazon Music. 

You’ve listened to the podcast, now subscribe to the magazine

We hope this episode, along with the others in the series, inspires you and helps you to engage your students more in computing. In the comments section below, we’d love to hear your thoughts and experience around digital literacy.

The latest issue of Hello World magazine features more insightful articles on digital literacy. Don’t miss out on this discussion — subscribe to Hello World today to ensure you never miss a podcast episode or issue of the magazine.

Subscribe

The post Should we ditch the term ‘digital native’? The latest conversation on the podcast appeared first on Raspberry Pi Foundation.

The collective thoughts of the interwebz