Amazon Linux default SSM parameter will now track the latest kernel

Post Syndicated from Gokul Govindaraju original https://aws.amazon.com/blogs/compute/amazon-linux-default-ssm-parameter-will-now-track-the-latest-kernel/

Today we are announcing that the Amazon Linux kernel-default AWS Systems Manager (SSM) parameter will now update to point to the latest Amazon Linux kernel version as new kernel versions get released. On August 17, 2026, for Amazon Linux 2023 (AL2023), the SSM parameter was updated from kernel 6.1 to kernel 6.18. As new kernel versions get released (expected annually), the parameter will continue to update to the latest kernel version after a validation period.

This post explains the default kernel behavior, what it means for your workloads, and how to manage the transition.

What’s changing?

Amazon Linux ships multiple kernel versions and has tracked a default kernel for each OS version. For example,
the AL2023 parameter:

ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-{minimal}-kernel-default-{x86_64 arm64}

has remained on kernel 6.1 since launch. Going forward, the kernel-default SSM parameter will update to the latest kernel as new versions are released. Each new kernel will go through a 3- to 6-month validation period after GA before we update the default. This window gives you time to test the new kernel before the change. We will announce the kernel-default upgrade date before it takes effect.

SSM Parameter Resolved to (Before) Resolves to (Now)
al2023-ami-{minimal}-kernel-default-{x86_64, arm64} Kernel 6.1 AMI Kernel 6.18 AMI (what’s changed)
al2023-ami-{minimal}-kernel-6.18-{x86_64, arm64} Kernel 6.18 AMI Kernel 6.18 AMI (unchanged)
al2023-ami-{minimal}-kernel-6.1-{x86_64, arm64} Kernel 6.1 AMI Kernel 6.1 AMI (unchanged)

Note: Already-running instances will keep the kernel they booted with and are not affected by this change. Only new instances launched from the kernel-default parameter will boot kernel 6.18. If you already use a version-specific SSM parameter, nothing changes for you.

Why are we making this change?

The Linux kernel is the foundation of workloads you run on Amazon Elastic Compute Cloud (Amazon EC2) and other services. Each new kernel brings meaningful improvements. For example, kernel 6.18 includes the Earliest Eligible Virtual Deadline First (EEVDF) CPU scheduler for fairer CPU time distribution and improved latency in mixed workloads. The kernel also increases Transmission Control Protocol (TCP) receive buffer for better network throughput on high-bandwidth instances.

Previously, customers who wanted to run the latest Amazon Linux kernel had to manually update their SSM parameter references and redeploy each time a new kernel became available. With this change, you can receive these improvements without needing to manually upgrade.

Evaluating the default kernel upgrade

Staying on the default kernel is the recommended approach as it allows your new instances to always run the latest validated kernel with no manual intervention. However, because the default will now advance annually, you should build processes to validate that the new kernel works for your workload before each upgrade takes effect. If your workload has specific requirements that mandate a fixed kernel version, evaluate whether the new default is compatible or revert to a kernel version that suits your use case.

If you haven’t validated kernel 6.18 yet, we recommend launching test instances on kernel 6.18 using the version-specific SSM parameter al2023-ami-{minimal}-kernel-6.18-{x86_64, arm64}. For instructions on referencing SSM parameters in your launch configuration, see the AL2023 User Guide.

Staying on or reverting to a specific kernel version

If you experience issues with the new default, or if your workload requires a specific kernel version for additional validation time or any other reason, revert to the version-specific SSM parameter. Change your references from al2023-ami-{minimal}-kernel-default-x86_64 to al2023-ami-{minimal}-kernel-{kernel_version}-x86_64 (for example, al2023-ami-kernel-6.1-x86_64). This applies anywhere you resolve an AL2023 AMI, including AWS CloudFormation templates, launch templates, Amazon EC2 Auto Scaling groups, CI/CD pipelines, or CLI scripts. For examples, refer to the AL2023 User Guide.

Each of the supported kernels (6.1, 6.12, and 6.18) continue to receive updates as defined in AL2023 kernel lifecycle. When staying on a specific version, we recommend tracking the kernel lifecycle and planning upgrades before the kernel reaches end of support.

Note: For Federal Information Processing Standards (FIPS) workloads, the default kernel may not always be the FIPS-validated kernel. If you require FIPS mode, see AL2023 FIPS FAQ.

Conclusion

In this post, we announced that the Amazon Linux default SSM parameter will now upgrade to the latest kernel as new kernel versions are released. The AL2023 kernel-default parameter was updated from kernel 6.1 to kernel 6.18 on August 17, 2026. We explained how the new cadence works, how already-running instances are unaffected, and how to stay on a specific kernel version if your workload requires it.

To learn more, see the AL2023 Kernel documentation and the AL2023 release notes. For questions or issues, contact AWS Support.

[$] A look at the Quickshell desktop-component toolkit

Post Syndicated from jzb original https://lwn.net/Articles/1083090/

Quickshell is a toolkit for
building desktop components, such as toolbars or menus. It uses QML, which is a declarative language
for designing GUI applications. Quickshell helps developers create graphical tools
for common desktop use cases with a focus on ease of development. It offers a
convenient method for writing user interfaces and has been adopted by a number
of projects, such as caelestia-shell and DankMaterialShell, that
provide desktop environments for minimal window managers like Sway and niri.

Long-term system tables retention in Amazon Redshift with Amazon S3 Tables

Post Syndicated from Nidhi Nayak original https://aws.amazon.com/blogs/big-data/long-term-system-tables-retention-in-amazon-redshift-with-amazon-s3-tables/

Amazon Redshift system tables capture a continuous stream of operational signals: every query that runs, every connection that is made. This data powers observability, performance analysis, and compliance auditing across your data warehouses. Until now, the system tables retained this critical data for only 7 days, making long-term compliance and auditing difficult without custom workarounds.

Amazon Redshift system table integration with Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), automatically delivers your system table logs data to Amazon S3 Tables and stores them in Apache Iceberg format. You can configure retention periods for Amazon Redshift system table beyond the current 7-day limit, giving you extended compliance, auditing, and cross-warehouse observability without custom ETL pipelines or cluster resource consumption. Your data is open, durable, and queryable from Amazon Redshift, Amazon Athena, AWS Glue, Amazon EMR, or other Apache Iceberg-compatible engines.

In this post, we walk through how the Amazon Redshift system table integration delivers log data to Amazon S3 Tables. This feature is supported on RA3 and RG provisioned clusters and Amazon Redshift Serverless workgroups.

The challenge

If you run Amazon Redshift, you often face operational challenges driven by the 7-day system table retention limit:

  1. Limited query trend visibility: You want to compare how the same query performed 30 days ago compared to today. When performance shifts gradually, extended baselines enable data-driven root cause analysis rather than reactive troubleshooting.
  2. Enable before-and-after comparisons: When you add a new workload, change instance type, or adjust Workload Management (WLM) queues, you want to measure the impact precisely. Extended retention preserves the baseline data you need.
  3. Unlock seasonal capacity planning: Month-end spikes, quarter-close surges, and annual peaks require months of historical data to identify and plan. Extended retention reveals seasonal patterns across months and years.
  4. Custom ETL pipeline overhead: To work around the retention limit, teams build custom pipelines that copy system table data hourly/daily into persistent tables within Amazon Redshift Managed Storage. These pipelines consume cluster resources, compete with production workloads, and require ongoing engineering maintenance. When Amazon Redshift updates system table schemas and data sharing configurations, these pipelines require manual intervention and create gaps in records.
  5. Compliance requirements: Regulated industries are required to maintain audit trails spanning months or years. The 7-day limit requires custom infrastructure to meet these requirements. Amazon S3 Tables integration for Amazon Redshift system tables now addresses this.

How it works

Amazon Redshift system tables integration with Amazon S3 Tables is a fully managed capability that automatically writes Amazon Redshift system table data to Amazon S3 tables in Apache Iceberg format. AWS handles partitioning, compression, and retention management automatically. The log writing process runs in an isolated background process that alleviates resource contention with production workloads. AWS manages the pipelines for you.

The feature supports over 25 system views at launch – see the supported system views documentation.

Setting up

Follow these steps to enable system table integration with Amazon S3 Tables from the Amazon Redshift console:

  1. Open the Amazon Redshift console and navigate to the System table integrations page. You can also access this from the detail page of your provisioned cluster or Serverless workgroup.
  2. Choose Create System table integration. This launches the configuration wizard.
  3. Select the Amazon Redshift Provisioned cluster or Amazon Redshift Serverless workgroup that you want to enable the feature on.

    Amazon Redshift console data warehouse selection step in the create System table integration wizard

    Figure 1: Selecting the Amazon Redshift data warehouse in the System table integration wizard

  4. Choose the system views to publish from the Available system tables list. Select individual SYS_* views, or choose Select all supported system tables to publish all current and future supported views. If you select all, new views added in the future are automatically included without requiring a configuration change.

    Available system tables list in the System table integration wizard with SYS views selected for publishing

    Figure 2: Choosing the system views to publish from the Available system tables list

  5. Select the deployment model. Choose how data is organized in Amazon S3 Tables:
  • Individual S3 table per system table per data warehouse to keep this warehouse’s data in its own set of tables.
  • Shared S3 table per system table across data warehouses to consolidate data from multiple warehouses in the account into a shared set of tables.
  1. Optionally configure encryption with an AWS Key Management Service (AWS KMS) customer managed key. By default, data is encrypted with Amazon S3-managed key (SSE-S3) encryption.
  2. Save your changes. Amazon Redshift begins publishing the selected views to Amazon S3 Tables and continues adding new records on a fixed frequency.

To verify the integration is active:

  • Navigate to your cluster or workgroup detail page.
  • Check the integration status and the last ingestion time for each view.
  • You can also view the published data from the Amazon S3 Tables console.

After it’s enabled, Amazon Redshift writes log data to Amazon S3 tables periodically through an isolated background process, separate from production workloads. To start querying the retained logs, you will need to perform a one-time setup that connects your Amazon Redshift environment to Amazon S3 Tables data through AWS Glue Catalog. Complete the following steps:

  1. Set up an AWS Identity and Access Management (IAM) role with the necessary permissions for AWS Glue Data Catalog and Amazon S3 Tables access, then associate it with your Amazon Redshift cluster or Amazon Redshift serverless namespace.
  2. In AWS Glue Data Catalog, create a resource link that points to the Amazon S3 Tables database where your logs reside.
  3. In Amazon Redshift, create an external schema that references the resource link:
    CREATE EXTERNAL SCHEMA <schema_name>
    FROM DATA CATALOG
    DATABASE '<resource_link_database>'
    IAM_ROLE '<iam_role_arn>';

  4. With this in place, you can query your historical system table data using familiar 2-part notation:
    SELECT * FROM <schema_name>.<table_name>;

Because access to Amazon S3 Tables is read-only, the integrity of your audit trails is inherently preserved.

For detailed setup instructions including IAM policy examples, see Registering the S3 Tables bucket with AWS Glue Data Catalog.

Your data is now in Apache Iceberg

Your system table data is stored in Apache Iceberg, an open table format, so you have the freedom to choose a compatible query engine. Your observability and auditing data works with the tool you already use.

You can analyze your operational data using:

  1. Amazon Redshift: After the S3 table bucket is integrated with AWS Glue Data Catalog, create an external schema in Amazon Redshift pointing at the resource link to query the retained tables.
  2. Amazon Athena: Run serverless SQL queries against historical logs with zero infrastructure provisioning.
  3. AWS Glue: Build automated data processing and transformation jobs on top of your operational data.
  4. Amazon EMR: Run Spark-based analytics at scale for complex cross-warehouse analysis.

Because the data is stored in open Apache Iceberg format in Amazon S3 Tables, you can query it with Amazon Redshift, Amazon Athena, AI agent skills for natural-language queries, Amazon SageMaker Unified Studio, an Iceberg-compatible engine, business intelligence (BI) tools, and observability systems.

Cost efficiency

Log delivery from Amazon Redshift to Amazon S3 Tables incurs no additional cost. You only pay for Amazon S3 Tables storage, maintenance, and querying the data with the engine of your choice.

Solution overview

The following scenarios illustrate how Amazon Redshift system tables integration with Amazon S3 Tables addresses common operational, compliance, and observability challenges across your Amazon Redshift environment. We also built a dedicated skill, querying-aws-redshift, for this feature and embedded it into the AWS MCP Server so you can query Amazon Redshift system tables from Amazon S3 Tables.

With months or years of SYS_QUERY_HISTORY data retained, you can trace how individual queries perform over extended periods. You can compare execution time, queue time, and resource consumption for a query across days, weeks, or months.

You can pinpoint exactly when performance started degrading and correlate it with what changed: a new schema, a spike in data volume, or an additional concurrent workload. Extended retention turns troubleshooting into proactive, data-driven root cause analysis.

Scenario 2: Assess workload impact before and after changes

Every workload change affects your system: a new ETL pipeline, an instance type change, a Workload Management (WLM) queue adjustment, or a new team of analysts running ad hoc queries. The question is always: how did this change affect performance?

With Amazon S3 Tables integration for Amazon Redshift system table, you can make data-driven decisions with confidence. Query SYS_QUERY_HISTORY to compare execution times, queue wait durations, and concurrency scaling events from the weeks before a change versus the weeks after. If you onboarded a new reporting workload two weeks ago and want to understand its effect on existing queries, the data to confirm that is already there, with zero custom pipeline required.

Scenario 3: Build observability dashboards

Your system table data is stored in Apache Iceberg and cataloged in AWS Glue, which means an observability or business intelligence (BI) tool that reads Apache Iceberg can connect directly to it. Visualize workload distribution trends in Amazon Quick Sight for executive reporting. Use Amazon SageMaker Unified Studio for deeper analytical exploration or to power AI-driven insights from your operational data. Beyond AWS services, connect your preferred third-party observability systems and BI tools to track query volumes, monitor connection patterns, set up alerts for anomalies, or correlate Amazon Redshift operational data alongside application-level logs.

Your observability and auditing data works with tools that you already use. Direct access to durable, structured operational data, with a tool you prefer.

Scenario 4: Plan capacity with seasonal context

Workload demand varies throughout the year. Month-end close, quarter-end reporting, annual planning cycles, and promotional events all create predictable usage spikes, but only if you have enough historical data to see the pattern.

With extended retention, you can analyze utilization trends across multiple business cycles. Identify when you consistently approach capacity limits, measure how demand shifts quarter over quarter, and validate whether your provisioned resources align with actual usage.

Scenario 5: Maintain compliance audit trails

For regulated industries, extended retention delivers a fully managed audit trail with built-in integrity.

SYS_CONNECTION_LOG records every authentication attempt. SYS_USERLOG captures user account changes. SYS_QUERY_HISTORY documents every query executed against your warehouse.

Configure retention to match your organization’s data retention policies: whether that is 90 days, one year, or multiple years. The read-only access policy helps prevent records from being altered after they are written, including by administrators.

Scenario 6: Centralize fleet observability across your warehouse

If you run multiple Amazon Redshift warehouses, you benefit from a unified view of operational data. The feature supports two deployment patterns to match your organizational structure:

  1. Individual tables per warehouse: Each warehouse writes to its own dedicated Amazon S3 tables, providing complete data isolation for compliance-sensitive environments. To query multiple warehouses, a UNION operation is required.
  2. Shared tables: Warehouses across the same account and same AWS Region write to a single shared set of Amazon S3 tables, with data distinguished by the warehouse_name column. Filter by warehouse for instant cross-cluster analysis.

Best practices

  1. Identify warehouses with logs requiring isolation for privacy reasons and select the individual table per warehouse option for those. For the remaining warehouses, use the Shared tables (consolidated) option for ease of management.
  2. Align retention duration with your compliance requirements. Configure the minimum retention period that satisfies your compliance requirements to reduce storage costs.
  3. When querying retained system tables, filter on metadata columns such as warehouse_account_id, warehouse_region_name, warehouse_namespace_arn, warehouse_name, and s3_tables_ingestion_time to reduce scan scope and improve performance. This is particularly important when querying large volumes of historical data across multiple warehouses.
  4. Rely on the built-in read-only access for audit trail integrity. Use the Amazon S3 Tables configuration APIs to manage retention and encryption settings.
  5. Plan your encryption strategy early. Choose your encryption key carefully at setup, as changes require recreating the integration. If you anticipate consolidating warehouses in the future, choose a shared AWS KMS key from the start.

Conclusion

Amazon Redshift system table integration with Amazon S3 Tables replaces custom ETL pipelines with a fully managed solution to preserve your Amazon Redshift operational data. With automatic Apache Iceberg-based storage, open format queryability, and built-in audit integrity, you get months or years of observability data, fully managed. You can enable it through the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDKs.

To learn more, visit the Amazon Redshift system tables documentation.


About the authors

Nidhi Nayak

Nidhi Nayak

Nidhi is a Senior Technical Account Manager with AWS, she helps enterprise customers build scalable, high-performance cloud applications and optimize cloud operations. With over a decade of experience in Data Analytics, Nidhi currently focuses on Redshift & Generative AI integration with Redshift.

Raza Hafeez

Raza Hafeez

Raza is a Senior Product Manager, Technical at Amazon Redshift. He has 15+ years of experience building and optimizing enterprise data warehouses and is passionate about making cloud analytics accessible and cost-effective for customers of all sizes.

Shubham Purwar

Shubham is an AWS Analytics Specialist Solution Architect. He helps organizations unlock the full potential of their data by designing and implementing scalable, secure, and high-performance analytics solutions on the AWS platform. With deep expertise in AWS analytics services, he collaborates with customers to uncover their distinct business requirements and create customized solutions that deliver actionable insights and drive business growth. In his free time, Shubham loves to spend time with his family and travel around the world.

Amrita Singh

Amrita Singh

Amrita is a Senior Technical Account Manager at AWS, based in Salt Lake City, USA. She specializes in Amazon Redshift, helping enterprise customers optimize their data warehouse environments for performance, scalability, and cost efficiency. Amrita works directly with AWS customers to provide guidance and technical assistance on their cloud journeys, helping them achieve higher flexibility, scale, and resiliency with AWS services.

AWS Network Firewall now supports rule hit count

Post Syndicated from Preetkumar Shah original https://aws.amazon.com/blogs/security/aws-network-firewall-now-supports-rule-hit-count/

As firewall rule sets grow in complexity, security teams face a common challenge: manual log analysis is used to determine which rules are actively matching traffic and which are consuming capacity without being triggered. This lack of visibility creates operational and compliance gaps. Organizations with governance policies that require removal of dormant rules after a defined period have no mechanism to identify them. Teams responsible for compliance frameworks such as Payment Card Industry (PCI) 4.0 and Digital Operational Resilience Act (DORA) can’t provide evidence that specific controls are actively functioning. Central teams managing firewalls on behalf of multiple business units have no way to determine which rules are unused or need updating.

In this post, you learn how a new AWS Network Firewall capability—rule hit count—addresses these challenges by providing traffic match data for stateful rules across both custom and managed rule groups. With this data, you can identify and remove unused rules, accelerate incident response, and validate security control effectiveness for compliance.

How it works

Rule hit counts track how often each stateful rule matches network traffic. The hit counter increments only when a rule match results in an alert log being created. This means any rule with an alert, drop, or reject action will increment the hit counter, because these actions generate alert logs. However, rules configured with a pass action don’t generate alert logs by default, meaning they won’t appear in the rule hit count metric.

To gain visibility into traffic matching pass rules, you can include the alert keyword within the pass rule. This generates an alert log while still permitting the traffic to its intended destination. The following Suricata rule demonstrates this approach:

pass tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"Pass and Log HTTPS traffic"; alert; sid:1000001; rev:1;)

This rule passes HTTPS traffic to its destination while also generating an alert log, making sure the rule appears in the hit count metric.

The rule hit count feature adds the following metadata to each alert log. Metadata is included by default and doesn’t require additional configuration:

“aws_metadata": { “resource_arn": “arn:aws:network-firewall:us-east-1:123456789012:stateful-rulegroup/StatefulRuleGroup” }

The following example shows a complete alert log with this metadata included:

{ 

    "firewall_name": "egress-and-east-west-firewall", 

    "availability_zone": "us-east-1a", 

    "event_timestamp": "1786112515", 

    "event": { 

        "tx_guessed": true, 

        "aws_category": "", 

        "tx_id": 0, 

        "app_proto": "http", 

        "ip_v": 4, 

        "src_ip": "10.2.1.205", 

        "src_port": 46240, 

        "event_type": "alert", 

        "alert": { 

            "severity": 3, 

            "signature_id": 10000003, 

            "rev": 0, 

            "signature": "Egress HTTP but not port TCP/80", 

            "action": "blocked", 

            "category": "" 

        }, 

        "ts_progress": "request_complete", 

        "flow_id": 927132830538451, 

        "dest_ip": "3.226.253.175", 

        "proto": "TCP", 

        "verdict": { 

            "action": "drop" 

        }, 

        "http": { 

            "hostname": "3.226.253.175", 

            "http_port": 4444, 

            "url": "/", 

            "http_user_agent": "curl/8.17.0", 

            "http_method": "GET", 

            "protocol": "HTTP/1.1", 

            "length": 0 

        }, 

        "tc_progress": "response_started", 

        "dest_port": 4444, 

        "pkt_src": "geneve encapsulation", 

        "aws_metadata": { 

            "resource_arn": "arn:aws:network-firewall:us-east-1:123456789012:stateful-rulegroup/StatefulRuleGroup" 

        }, 

        "timestamp": "2026-08-07T14:21:55.611810+0000", 

        "direction": "to_server" 

    } 

} 

The alert log data in the preceding example is the source for rule hit count metrics. Network Firewall pushes these alert logs to your Amazon CloudWatch Logs or Amazon Simple Storage Service (Amazon S3). To identify the specific rule that generated an alert, you can search using the combination of the sid (signature ID) and resource_arn fields. The firewall monitoring dashboard uses these fields to generate hit counts for each rule, so you can review rule activity directly on the dashboard without querying logs.

You can also access and analyze this data by querying those logs directly using CloudWatch Logs Insights for logs stored in CloudWatch, or Amazon Athena for logs stored in Amazon S3.

Getting started

Network Firewall rule hit count is enabled by default, so you don’t need to perform any additional configuration to start tracking rule hits on your firewall policies. This means that as soon as you deploy your firewall rules, you can begin to monitor which rules are being triggered, helping you gain visibility into your traffic patterns and identify potential security concerns.

Prerequisites

This walkthrough requires an existing network firewall configured to inspect traffic from your Amazon Virtual Private Cloud (Amazon VPC). If you don’t have one set up yet, follow the Getting started with AWS Network Firewall guide.

Additionally, ensure the following:

  1. Alert log delivery must be configured: The firewall must have alert logging enabled. Note that rule hit count metadata is captured regardless of log destination; however, the native dashboard feature requires logs to be sent to CloudWatch Logs or Amazon S3.
  2. Firewall monitoring must be enabled: To see the dashboard widget shown in figure 1, you must enable detailed monitoring through the firewall’s logging configuration or the Monitoring tab in the AWS Management Console for AWS Network Firewall. However, if you have a custom dashboard solution, the metadata required for rule hit count analysis is automatically included in the firewall logs regardless of whether detailed monitoring is enabled—so you can build your own visualizations using the log data directly.
  3. Pass rules must include the alert keyword to appear in hit count metrics: Rules configured with a pass action don’t generate alert logs by default. To track pass rule activity in the hit count metric, include the alert keyword in your pass rules, as demonstrated in the How it works section of this post.

The Top Rule Hits dashboard shows aggregated hit counts per firewall across all Availability Zones within the AWS Region where that firewall is deployed. To view rule hit count metrics, open the Network Firewall console and select your firewall. Navigate to the Monitoring and observability section. Under Top analysis, you will see the Top Rule Hits metric. Select a lookback period to view rule activity within that timeframe.

Figure 1: Rule hit count from the dashboard

Figure 1: Rule hit count from the dashboard

Figure 1 shows the Top Rule Hits panel from the AWS Network Firewall console, displaying the most frequently triggered stateful rules. It includes columns for Hit Count (with bar chart and fraction), percentage of total hits, Resource ARN, Signature ID, Description (the msg field from the Suricata rule), and Last Occurrence (UTC).

Signature IDs 2, 4, 6, and 8 are system-generated signatures corresponding to the firewall policy’s strict order default actions. Because these signatures originate from the policy rather than a rule group, the resource_arn field displays the firewall policy Amazon Resource Name (ARN) instead of a rule group ARN. They appear in the Top Rule Hits when the policy has default actions such as Drop established, Alert established, or their application-layer variants configured. These signatures fire on established connection packets that don’t match any explicit rule, enforcing the policy’s default deny posture.

The following examples demonstrate how rule hit counts help you address common operational challenges.

  • Identifying unused rules: Figure 1 shows all rule signature IDs and their descriptions (the msg field from the Suricata rule) that actively matched traffic during the selected lookback period. Any rule in your firewall policy whose signature ID doesn’t appear in this metric hasn’t matched any traffic during the specified timeframe. These rules are either stale or not ordered correctly within your rule group.
  • Accelerating incident response: Figure 1 shows signature ID 2525124575 (traffic_to_oast [oast[.]fun]) with six hits and a last occurrence of August 7, 2026, at 6:28:44 PM UTC. This rule is detecting traffic to an out-of-band application security testing (OAST) domain, which could indicate an attacker attempting to exfiltrate data or validate a vulnerability in your environment. By filtering the top rule hits metric to the timeframe of a suspected incident, your team can quickly identify this type of suspicious activity and scope the impact without manually parsing thousands of log entries.
  • Validating a newly added rule: Figure 1 shows signature ID 100000010 (Domain Category is AI/ML) with five hits and a last occurrence of August 7, 2026, at 6:28:21 PM UTC. After adding this rule to monitor or restrict traffic to AI/ML related domains, the hit count confirms the rule is actively matching traffic as intended. Similarly, signature ID 100000009 (Drop traffic to countries other than US) shows four hits, validating that the geofencing rule is functioning and blocking outbound connections to destinations outside the United States. These hit counts provide security teams with concrete evidence that newly deployed controls are working.

Pricing

Rule hit counts are included with Network Firewall at no additional cost. However, standard charges apply for storing and querying log data. If you configure log delivery to CloudWatch Logs, CloudWatch pricing applies. If you store logs in Amazon S3 and query them with Athena, standard Amazon S3 storage and Athena query charges apply. For complete pricing details, see AWS Network Firewall pricing.

Considerations

Keep the following in mind when you use rule hit counts:

  • To manage costs, review your log utilization and configure log filtering or retention policies.
  • Rule hit counts apply to stateful rules. Stateless rules don’t support hit count tracking at this time.
  • Rule hit counts are available in all AWS Regions where AWS Network Firewall is supported, except Middle East (UAE) and Middle East (Bahrain).

Conclusion

In this post, you learned how rule hit counts in AWS Network Firewall give you visibility into your firewall rule utilization and effectiveness. By tracking how frequently each rule matches traffic, you can identify unused or redundant rules, optimize rule ordering, validate security controls for compliance, and respond faster during security investigations. For more information, see AWS Network Firewall.

If you have feedback about this post, submit comments in the Comments section below.


Preetkumar Shah

Preetkumar Shah

Preetkumar is a Technical Account Manager at AWS, based in Atlanta, GA. He specializes in helping customers design and operate secure, scalable network architectures in the cloud. At AWS, he works with SMB customers and collaborates closely with service teams to proactively resolve complex challenges and ensure customers get the most from their AWS environment. Outside of work, his interests include spending time with family and going on trails.

Amit Gaur

Amit Gaur

Amit, a Cloud Infrastructure Architect at AWS, brings his passion for technology and knowledge-sharing to the networking community. Specializing in network architecture design, he helps customers build highly scalable and resilient environments on AWS. Through technical guidance and architectural expertise, Amit enables customers to accelerate their cloud adoption journey while making sure their systems are built for scale and reliability.

Santosh Shanbhag

Santosh is a seasoned product leader, specializing in security, data protection, and compliance. At AWS, he focuses on securing workloads through Network and Application Security services, including AWS Network Firewall and active threat defense.

Srivalsan Mannoor Sudhagar

Srivalsan is a Sr. Cloud Infrastructure Architect at Amazon Web Services Professional Services who brings expertise in Cloud Infrastructure and MLOps solutions. He is passionate about networking, container technologies and loves to innovate to help solve customer problems. He enjoys architecting solutions and providing technical guidance to help customers and partners achieve their technical and business objectives.

Cheriyan Mundapuzha

Cheriyan Mundapuzha

Cheriyan, a Cloud Infrastructure Architect at AWS, brings his infrastructure experience to some of the most complex migration challenges in the enterprise space. Through published architectural patterns, hands-on technical leadership, and mentorship of fellow professionals, he enables customers to accelerate their modernization journey while ensuring their systems are built for resilience and operational excellence.

From all-or-nothing to task-based OAuth consent

Post Syndicated from Miller Vargas original https://blog.cloudflare.com/task-based-oauth-consent/

Since June, developers have created thousands of third-party OAuth apps on Cloudflare, with more than a million authorizations since.  

OAuth makes delegated access possible. It lets applications act on a user’s behalf without asking them to handle long-lived credentials or hand over a password. That model works well when an application can describe its access needs with a small set of scopes. 

Developers use OAuth for SaaS integrations, internal tools, CLIs, and agents. Our permission model has become more granular over time to support better scoping of these different workflows. That is great for security, but it makes a purely all-or-nothing consent screen hard to justify.

Cloudflare OAuth already allows clients to request a subset of their configured scopes. But once the client made that request, the user could not narrow it any further on the consent screen. For the user on the consent screen, the experience was still an all-or-nothing one. If an application requested more access than a user was comfortable granting, their only options were to approve the full request, or deny outright. 

MCP servers are a good example of this. An MCP server might request a broad set of permissions, because in theory an agent could use all of them. But most users would not want an agent to have that much access. Before this feature, the only way to handle this was for the app developer to build a custom scope selection screen before sending the user to our consent flow.

Today, we’re introducing OAuth scope customization. Client owners can mark specific scopes as optional when configuring an OAuth client, giving users the ability to grant a narrower subset of an application’s requested access at authorization time.

The OAuth spec already allows authorization servers to grant a narrower set of scopes than what was requested. We built on top of that flexibility to make this work cleanly for every existing app.

More control, without overwhelming users

Our goal with introducing scope selection is to give security conscious users more flexibility to make the right choices for their use case, without turning the consent screen into a long scope checklist. 

With scope customization: 

  • Developers can mark specific scopes on an OAuth client as required or optional
  • At authorization time, users can deselect optional scopes from the requested set
  • Required and optional scopes are evaluated against the scopes requested for that authorization flow
  • If no optional scopes are requested, the consent experience stays the same
  • By default, the consent screen still grants the full requested scope set.  

Scoping to the authorization request

One important detail is that required and optional scopes are evaluated only against the scopes requested in a specific authorization flow, not every scope configured on the client. That matters because OAuth clients do not always request their full configured scope set.

For example, a client might be configured with user-details.read, workers-scripts.write, workers-kv-storage.write, and zone.read, while marking workers-kv-storage.write and zone.read as optional. If that client starts an authorization flow requesting all four scopes, the consent screen will evaluate all four. In that case, user-details.read and workers-scripts.write remain required, while the user can choose whether to grant workers-kv-storage.write and zone.read.

But if the client later requests only workers-scripts.write and zone.read, then only those two scopes are considered for that authorization flow. user-details.read and workers-kv-storage.write would not be shown or enforced, because they were not requested.

This keeps the consent screen focused on the task at hand, rather than every capability the application could request. It also means existing OAuth clients keep their current behavior by default: if a client does not opt into optional scopes, the consent flow remains unchanged.

Configuring an OAuth client to use optional scopes

Developers can opt into scope customization when configuring an OAuth client. Scopes continue to be configured as they are today, and clients can now additionally specify which of those scopes are optional: 

In the example above, the client can request all four scopes, but the user may only opt out of the workers-kv-storage.write and zone.read scopes during consent. user-details:read and workers-scripts.write remain required if they are included in the authorization request. 

If the client later requests only workers-scripts.write and zone.read, then only those two scopes are considered for that authorization flow. user-details.read and workers-kv-storage.write would not be shown or enforced because it was not requested.

Building with partial grants in mind

When a user deselects any optional scopes and completes the authorization flow, the generated access token will only contain the scopes they consented to. For developers, this means you need to check the granted scope set after exchanging the authorization code, rather than assuming the full requested set of scopes was approved.

An app that handles a narrower grant gracefully, for example an agent that operates within whatever subset of permissions it receives, is one that users feel comfortable authorizing. Requesting only the permissions needed and marking the rest as optional is a good sign to users that your app respects their access decisions.

Scopes for every Product

Over the next few weeks, we will be expanding our account & zone-level role surface to cover nearly every Cloudflare product. That means more API token roles, account membership options, and OAuth scopes, giving customers the tools to secure workloads with the right level of access. 

Build with Optional Scopes

Allowing developers and users to better restrict access through optional OAuth scopes is an important step toward a more flexible and trustworthy consent experience on Cloudflare. With optional scopes, developers can build more nuanced authorization flows, and users gain more control over what they approve. 

To get started with Third Party OAuth, take a look at our documentation or jump straight to the OAuth apps page in the dashboard and create your first OAuth app

Thank you to our amazing interns

This feature is one of the many that we built with the help of our 1,111 interns. Congratulations to Miller Vargas and José Enrique Rodriguez on your high impact contributions here. Miller is a senior at the University of Texas – Austin studying computer science and math; and José is a senior at Universidad Panamericana studying engineering, data intelligence, and cybersecurity.

Track SageMaker Unified Studio project costs with custom tags and AWS CUR

Post Syndicated from Nisha Gambhir original https://aws.amazon.com/blogs/big-data/track-sagemaker-unified-studio-project-costs-with-custom-tags-and-aws-cur/

Organizations running machine learning (ML), analytics, and generative AI workloads on Amazon SageMaker Unified Studio domains and projects face a common cost governance challenge. System tags (AmazonDataZoneDomainId and AmazonDataZoneProject) are automatically propagated to all underlying project resources. However, custom tags such as CostCenter, Team, or Environment are not propagated to dynamic resources created through the Studio UI. This creates a gap when you need to report project costs grouped by custom tags.

In this post, we walk through a serverless solution that bridges this gap by enriching AWS Cost and Usage Report (CUR) data with custom project tags. By the end of this post, you can build an Amazon Quick Sight dashboard to filter and analyze Amazon SageMaker Unified Studio project costs by any custom tag dimension that you define. This gives your team the visibility to make informed spending decisions.

Solution overview

The solution consists of three automated subsystems:

  1. Event-driven tag lookup management – An Amazon EventBridge rule captures Amazon DataZone project lifecycle events (Create, Update, Delete) and triggers an AWS Lambda function. The function maintains an Amazon DynamoDB lookup table that maps each project’s DomainId and ProjectId to its custom tags.
  2. CUR enrichment pipeline – An AWS Glue extract, transform, and load (ETL) job reads CUR 2.0 Parquet data from Amazon Simple Storage Service (Amazon S3). The job joins each billing line item with the DynamoDB lookup table using the system tags (DomainId, ProjectId), appends the custom tag values as new columns, and writes the enriched data back to Amazon S3.
  3. Cost visualization – An Amazon Quick Sight dashboard backed by a custom SQL dataset over Amazon Athena provides interactive cost and consumption analytics filtered by custom tags.

Architecture

The following diagram shows the end-to-end architecture:

Figure 1: SageMaker Unified Studio project custom tag cost reporting

The workflow is as follows:

  • An Amazon SageMaker Unified Studio administrator creates or updates a project with custom tags.
  • AWS CloudTrail captures the API call.
  • Amazon EventBridge matches the event.
  • The Lambda orchestrator writes the tag mapping to DynamoDB.
  • Separately, AWS Data Exports delivers CUR data to Amazon S3.
  • The AWS Glue ETL job enriches CUR line items with custom tags from DynamoDB.
  • The AWS Glue Crawler catalogs the enriched data.
  • Amazon Quick Sight visualizes costs by custom tags.

Prerequisites

Before deploying this solution, you need:

  • An Amazon SageMaker Unified Studio domain (you create projects after deployment).
  • AWS Cloud Development Kit (AWS CDK) CLI installed.
  • Python 3.12+.
  • Amazon Quick Sight Enterprise edition enabled in your account.
  • An AWS Identity and Access Management (IAM) user or role with permissions to deploy AWS CloudFormation stacks.

Step 1: Configure custom tags on your project profile

You configure custom tags on project profiles through the Amazon DataZone API. First, enable custom tags on your project profile:

aws datazone update-project-profile \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_PROFILE_ID \
  --region $REGION \
  --allow-custom-project-resource-tags \
  --project-resource-tags '[
  {"key": "CostCenter", "value": "default", "isValueEditable": true},
  {"key": "Team", "value": "default", "isValueEditable": true},
  {"key": "Environment", "value": "default", "isValueEditable": true}
]'

When creating or updating a project, set the tag values:

aws datazone update-project \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_ID \
  --project-profile-version latest \
  --region $REGION \
  --resource-tags '{"CostCenter": "CC-100", "Team": "ML-Platform", "Environment": "Production"}'

Important: The AmazonSageMakerProvisioning-<domainAccountId> role needs an inline policy that permits your custom tag keys. Without this, project environment deployment fails.

The following is the inline policy that’s used for the custom tags shared in this post:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCustomTagKeys",
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:DeleteTags",
        "cloudformation:TagResource",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack"
      ],
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:TagKeys": [
            "AmazonDataZone*",
            "CostCenter",
            "Team",
            "Environment"
          ]
        }
      }
    }
  ]
}

Step 2: Activate cost allocation tags

Activate the SageMaker Unified Studio system tags as cost allocation tags so they appear in CUR data:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Active"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Active"}
]'

These tags take up to 24 hours to start appearing in CUR reports after activation.

Step 3: Deploy the infrastructure

The solution is packaged as a CDK application. Clone the GitHub repository and deploy:

# Install dependencies
pip install -r requirements.txt

# Bootstrap CDK (first time only)
cdk bootstrap aws://$ACCOUNT_ID/$REGION

# Deploy
cdk deploy

This creates the following resources:

  • DynamoDB table (smus-project-tag-lookup) – stores project-to-tag mappings.
  • Lambda function (smus-orchestrator) – processes project lifecycle events.
  • Amazon EventBridge rule – matches Amazon DataZone CreateProject/UpdateProject/DeleteProject events.
  • S3 buckets – for raw CUR and enriched CUR data.
  • AWS Glue ETL job (smus-cur-enrichment) – enriches CUR with custom tags.
  • AWS Glue Crawler – catalogs enriched data.
  • Amazon Simple Notification Service (Amazon SNS) topic – pipeline failure alerts.

Note: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight), so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Step 4: Configure CUR delivery

Create a CUR 2.0 export through AWS Data Exports that delivers Parquet files to the CUR S3 bucket created by the stack. The following screenshots show the complete configuration process in the AWS Billing and Cost Management console.

To create the export, follow these steps:

  1. Go to AWS Billing and Cost Management and then choose Data Exports.
  2. Choose Create in the upper right corner of the Exports and dashboards page. The Data Exports console shows any existing exports, their status, export type, data table, and last refresh date.
  3. On the Create export page, under Export details, select Standard data export and enter an export name. Under Data table content settings, select CUR 2.0.
  4. Under Data table configurations, set Time granularity to Hourly. The configuration page also lets you choose additional export content options such as including resource IDs, split cost allocation data, caller identity allocation data, and capacity reservation columns.
  5. Under Data export delivery options, set Compression type and file format to Parquet. Under Data export storage settings, configure the S3 bucket to: smus-cur-report-{account-id}-{region} and set the S3 path prefix as needed. Choose Create to finish.
Data Exports console listing existing exports with status, type, and last refresh date

Figure 2: Data Exports page listing existing exports

Create export page with Standard data export selected and CUR 2.0 chosen

Figure 3: Create export page with Standard data export and CUR 2.0 selected

Data table configurations with time granularity set to Hourly

Figure 4: Data table configurations with time granularity set to Hourly

Data export delivery options with Parquet format and the S3 storage destination configured

Figure 5: Data export delivery options with Parquet format and S3 storage settings

Step 5: How the event-driven tag capture works

When a project is created or updated in Amazon SageMaker Unified Studio (through the Studio UI or API), the following happens automatically:

  1. CloudTrail logs the Amazon DataZone API call.
  2. Amazon EventBridge matches the event.
  3. Amazon EventBridge invokes the Lambda function.
  4. The Lambda extracts custom tags from the CloudTrail event payload.
  5. The Lambda writes a record to DynamoDB with the DomainId, ProjectId, and all custom tag key-value pairs.

The Lambda function reads tags directly from the responseElements.resourceTags field of the CloudTrail event rather than making a separate GetProject API call. This avoids a race condition where GetProject might return empty tags while the project is in the UPDATING state.

def _extract_tags_from_event(detail):
    tags = {}
    response_elements = detail.get("responseElements") or {}
    for tag_entry in response_elements.get("resourceTags", []):
        if isinstance(tag_entry, dict) and "key" in tag_entry:
            tags[tag_entry["key"]] = tag_entry["value"]
    request_params = detail.get("requestParameters") or {}
    req_tags = request_params.get("resourceTags", {})
    if isinstance(req_tags, dict):
        tags.update(req_tags)
    return tags

Step 6: How the CUR enrichment works

The AWS Glue ETL job runs on a schedule (after each CUR delivery):

  1. Reads CUR Parquet files from the CUR S3 bucket.
  2. Reads all records from the DynamoDB lookup table.
  3. Performs a left outer join on DomainId and ProjectId.
  4. Appends custom tag columns (CostCenter, Team, Environment, and so on) to each CUR line item.
  5. Writes enriched Parquet to the enriched S3 bucket.

Line items without a matching project in the lookup table retain all original columns with NULL custom tag values. No data is dropped.

joined_df = cur_df.join(
    lookup_df,
    on=(
        (cur_df[DOMAIN_COL] == lookup_df["domainId"])
        & (cur_df[PROJECT_COL] == lookup_df["projectId"])
    ),
    how="left_outer",
)

Step 7: Set up the Amazon Quick Sight dashboard

After the first ETL run and crawler execution, set up the Amazon Quick Sight dashboard:

python scripts/setup_quicksight.py \
  --account-id $ACCOUNT_ID \
  --region $REGION \
  --quicksight-user $QUICKSIGHT_USER_ARN

This creates a dashboard with five visuals:

  • Cost by Custom Tag (CostCenter) – horizontal bar chart.
  • Cost by Project – horizontal bar chart.
  • Daily Cost Trend – line chart.
  • Cost by Service per Project – stacked bar chart.
  • Usage by Project & Service – summary table.

And six interactive list filters: Domain, Project, CostCenter, Team, Environment, Service.

The custom SQL includes a CASE statement for service categorization:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_usage_amount,
  line_item_unblended_cost,
  resource_tags_user_amazondatazone_domain_id AS domain_id,
  resource_tags_user_amazondatazone_project AS project_id,
  costcenter, team, environment,
  CASE
    WHEN line_item_product_code = 'AmazonSageMaker' THEN 'SageMaker'
    WHEN line_item_product_code = 'AmazonS3' THEN 'S3'
    WHEN line_item_product_code = 'AWSGlue' THEN 'Glue'
    ELSE line_item_product_code
  END AS service_category
FROM "smus_cost_reporting"."enriched_cur"
WHERE line_item_unblended_cost > 0

Step 8: Verifying the solution

After deploying the infrastructure and setting up the dashboard, verify that each component of the pipeline is functioning correctly.

8.1 Verify Amazon EventBridge is capturing project events

  1. Open the Amazon EventBridge console.
  2. In the navigation pane, choose Rules.
  3. Select the rule created by the CDK stack (for example, SmusCostReporting-ProjectTagRule).
  4. Choose the Monitoring tab.
  5. Confirm that the invocations are being recorded in the metrics.
  6. Create or update an Amazon SageMaker Unified Studio project with custom tags using the following command:
    aws datazone update-project \
      --domain-identifier <domain-id> \
      --identifier <project-id> \
      --custom-tags CostCenter=Engineering Team=DataPlatform Environment=Production

  7. Within a few seconds, the Amazon EventBridge rule should show a new invocation in its metrics.

8.2 Verify DynamoDB schema and tag mappings

The DynamoDB lookup table uses a simple key schema:

Attribute Type Role
domainId String Partition Key
projectId String Sort Key
CostCenter String Custom tag
Team String Custom tag
Environment String Custom tag

Custom tags are stored as dynamic attributes. Any tag key set on a project becomes a column in the table.

8.2.1 Verify DynamoDB table contains tag mappings

  1. Open the DynamoDB console.
  2. Navigate to the table created by the stack (for example, SmusCostReporting-ProjectTagsTable).
  3. Choose Explore table items.
  4. Scan for your project with the following keys:
    Partition key (domainId): <your-domain-id>
    Sort key (projectId): <your-project-id>

  5. Confirm the item contains the expected custom tag attributes (CostCenter, Team, Environment) with the values you assigned.
  6. Alternatively, use the AWS CLI:
    aws dynamodb get-item \
      --table-name SmusCostReporting-ProjectTagsTable \
      --key '{"domainId": {"S": "<domain-id>"}, "projectId": {"S": "<project-id>"}}'

8.3 Verify the AWS Glue ETL job enriches CUR data

  1. Wait for the next CUR delivery (hourly if configured as described in Step 4).
  2. Wait for the subsequent AWS Glue job execution.
  3. Open the AWS Glue console.
  4. In the navigation pane, choose ETL Jobs.
  5. Confirm the job completed successfully (status: Succeeded).
  6. Query the enriched data in Amazon Athena to confirm custom tag columns are populated:
    SELECT
      line_item_usage_start_date,
      line_item_product_code,
      line_item_unblended_cost,
      costcenter,
      team,
      environment
    FROM "smus_cost_reporting"."enriched_cur"
    WHERE costcenter IS NOT NULL
    LIMIT 10;

You should see rows with your custom tag values populated in the costcenter, team, and environment columns.

8.4 Verify the Amazon Quick Sight dashboard displays enriched data

  1. Open the Amazon Quick Sight console and navigate to the dashboard created by the setup script.
  2. Confirm that:
    • The Cost by Custom Tag (CostCenter) bar chart displays cost data grouped by your CostCenter values.
    • The list filters for CostCenter, Team, and Environment contain selectable values.
    • Selecting a filter value correctly narrows the displayed data.
  3. If the dashboard shows no data, verify that:
    • The AWS Glue Crawler has run after the ETL job (check the crawler’s last run status in the AWS Glue console).
    • The SPICE dataset has been refreshed. In the Amazon Quick Sight console, navigate to Datasets, select the dataset, and then choose Refresh now.

Figure 6 shows the Amazon Quick Sight dashboard with two side-by-side horizontal bar charts: Cost by Cost Center and Cost by Project. Domain Name and Project Name list filters appear at the top.

Amazon Quick Sight dashboard with Cost by Cost Center and Cost by Project bar charts and Domain and Project filters

Figure 6: Amazon Quick Sight dashboard showing cost data by custom tags, including Cost by Cost Center and Cost by Project bar charts with Domain Name and Project Name filters

Note: The first end-to-end cycle can take up to 48 hours depending on CUR delivery timing. After the initial cycle completes, subsequent updates will flow automatically on the configured schedule.

Operational considerations

Monitoring: The Amazon SNS topic smus-cost-reporting-alerts receives notifications when the AWS Glue ETL job fails or the Lambda orchestrator encounters repeated errors. Subscribe an email address or Slack webhook to stay informed. For instructions on how to create a subscription, see Subscribing to an Amazon SNS topic.

Cost: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight, SPICE) so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Scaling: The DynamoDB table uses on-demand capacity and can scale to accommodate your projects. You can scale the AWS Glue ETL job by increasing the number of workers for larger CUR datasets. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

New tag keys: When you add new custom tag keys to projects, the ETL automatically picks them up as new columns. The AWS Glue Crawler’s UPDATE_IN_DATABASE policy adds new columns to the catalog table without manual intervention.

Cleanup

Warning: The following cleanup steps will permanently delete all CUR data, project tag mappings, and Amazon Quick Sight dashboards.

To remove all resources:

# Delete Amazon Quick Sight resources
python scripts/setup_quicksight.py --account-id $ACCOUNT_ID --region $REGION --quicksight-user $QS_USER --clean

# Delete CDK stack
cdk destroy

Go to AWS Billing and Cost Management, and then choose Data Exports and delete the CUR 2.0 export created in Step 4.

Deactivate the cost allocation tags that were activated in Step 2:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Inactive"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Inactive"}
]'

Conclusion

In this post, we showed how to build an end-to-end cost reporting solution for Amazon SageMaker Unified Studio projects using custom tags. This solution combines tag capture driven by Amazon EventBridge, CUR enrichment through AWS Glue ETL, and visualization in Amazon Quick Sight. With it, organizations can track and attribute costs by CostCenter, Team, Environment, or any custom dimension. This works even for resources created through the Studio UI that don’t receive custom tag propagation.

This solution serves as an extension to the custom tag propagation feature and reports cost for all project resources. The architecture is fully serverless, automated, and can be deployed to any AWS account using the provided CDK application.

To start building your custom tag cost reporting pipeline, visit the GitHub repository. To learn more about the underlying services, visit the Amazon SageMaker Unified Studio service page. For a related approach to custom tag governance, see Use Amazon SageMaker custom tags for project resource governance and cost tracking

References


About the authors

Nisha Gambhir

Nisha Gambhir

Nisha is a Senior AI/ML & Cloud Architect based out of India. She is passionate about helping customers design, architect and develop secure, scalable and reliable applications using AI/ML and Agentic AI. She loves working on latest technologies, providing simple and scalable solutions that drive positive business outcomes.

Dr Anil Giri

Dr Anil Giri

Anil is a Solutions Architect at AWS, based in London, UK, where he helps ISV customers design and deploy agentic AI systems in production. He specializes in multi-agent orchestration, retrieval-augmented generation, and event-driven serverless architectures on Amazon Bedrock, with a focus on building reliable, secure, and scalable solutions that deliver measurable business outcomes.

Satish Sarapuri

Satish Sarapuri

Satish is a Sr. Data Architect, Data Mesh / Data Lake/Gen AI at AWS. He helps enterprise-level customers build high-performance, highly available, cost-effective, resilient, and secure generative AI, data mesh, data lake, and analytics platform solutions on AWS, through which customers can make data-driven decisions to gain impactful outcomes for their business and help them on their digital and data transformation journey. In his spare time, he enjoys trail running and spending quality time with his family.

Ram Vittal

Ram Vittal

Ram is a Principal GenAI/ML Specialist at AWS. He has over 3 decades of experience building distributed, hybrid, and cloud applications. He is passionate about building secure, scalable, reliable AI/ML and big data solutions to help customers with their cloud adoption and optimization journey. In his spare time, he rides motorcycle and enjoys the nature with his family.

Supply chain attack on arrayref (Rust blog)

Post Syndicated from corbet original https://lwn.net/Articles/1089720/

The Rust blog reports
on a malicious crate, called proc-macro1, that was uploaded to the
crates.io repository.

Furthermore, we discovered that the popular arrayref crate
had recently been republished and made to depend on this crate,
with the most recent versions yanked. We have removed the malicious
version and unyanked the maliciously-yanked versions. Other crates
by that author (internment, append-only-vec) were
also affected so we have done the same for those, and locked the
account as a precaution. We do not believe the author of
arrayref to be acting maliciously, but their computer or
credentials are likely compromised, and we are attempting to
contact them.

[$] The beginning of the 7.3 merge window

Post Syndicated from corbet original https://lwn.net/Articles/1089244/

As of this writing, 2,346 non-merge changesets have been pulled into the
mainline repository for the 7.3 kernel release. That, clearly, is a mere
down payment on the flood that is to come. Even so, those early pulls
brought in some noteworthy changes, including (but not limited to) a
significant reworking of how group scheduling works on multiprocessor
systems.

The collective thoughts of the interwebz