Tag Archives: Amazon Simple Storage Service (S3)

Cut costs and simplify operations with writable warm storage in Amazon OpenSearch Service

Post Syndicated from Bharav Patel original https://aws.amazon.com/blogs/big-data/cut-costs-and-simplify-operations-with-writable-warm-storage-in-amazon-opensearch-service/

Managing petabytes of search data means making tough choices: keep everything fast and expensive, or make it affordable but read-only. UltraWarm is a proven, cost-effective solution for read-heavy historical data. However, some workloads occasionally need to update historical records, such as late-arriving data or compliance corrections. With UltraWarm, you must migrate those indices back to hot, perform the update, and migrate back. What if you could write directly to your cost-effective warm storage instead?

In this post, I show you how writable warm storage removes the costly migration cycle. You can reduce your infrastructure costs by up to 48 percent and update historical data in seconds instead of hours. I walk through a real-world cost comparison and performance benchmarks, and help you decide when to use writable warm versus UltraWarm.

The challenge with tiered storage

Amazon OpenSearch Service handles data-intensive search and analytics workloads, from real-time log analytics and application monitoring to security event detection. As your data volumes grow from terabytes to petabytes, you face a fundamental question: how do you keep recent data fast while making earlier data affordable?

OpenSearch Service addresses this with a tiered storage architecture:

  • Hot – Highest performance for active indexing and search using instance-attached storage.
  • UltraWarm – Cost-effective, read-only tier backed by Amazon Simple Storage Service (Amazon S3) with local caching for less frequently queried data.
  • Cold – Fully detached from the cluster, with the lowest cost for rarely accessed data. Cold indices must be migrated back to UltraWarm or hot before any reads or writes can be performed.

For immutable log data, this model works well. However, a specific class of workloads hits its limitations when they occasionally need to write to earlier data, and read-only becomes a bottleneck.

Prerequisites

To use writable warm storage, you need the following:

  1. An Amazon OpenSearch Service domain running version 3.3 or later.
  2. OpenSearch Optimized (OI2) instance family support in your AWS Region.
  3. Workloads with a minimum 5-second refresh interval.
  4. Data nodes using the OpenSearch Optimized instance family (OR2 for hot, OI2 for warm).

Note: Writable warm doesn’t currently support the cold storage tier.

The UltraWarm bottleneck

With UltraWarm, updating even a single document requires migrating the index back to hot, performing the write, and migrating it back. This round trip involves a force merge (consolidating index segments), snapshot creation, and shard relocation. These operations consume significant CPU, memory, and disk space on your hot nodes, and they take approximately 130 minutes per 100 GB index. This time was measured on a domain with 3 × r6g.2xlarge hot nodes, 3 × ultrawarm1.large warm nodes, and 3 dedicated leader nodes (US East, N. Virginia), using a single-shard index with one replica. Actual times vary based on domain configuration, shard count, segment count, hot node utilization, and migration queue depth. The result is that you over-provision hot nodes, build complex pipelines, or keep data in hot longer than necessary, which increases cost and complexity.

Introducing writable warm storage

OpenSearch Service now offers writable warm nodes that use OpenSearch Optimized (OI2) instances, the same instance family that powers durable, Amazon S3-backed storage on hot nodes. Because data is already persisted on Amazon S3, tier transitions become a lightweight shard relocation rather than a resource-intensive migration. The Lucene engine, which is OpenSearch’s underlying search library, operates identically on both tiers. As a result, writable warm nodes support active writes, background merges, and periodic refreshes, just like hot nodes.

Late-arriving data, compliance backfills, and corrections that previously required a warm-to-hot-to-warm round trip now resolve with a direct write in seconds. There is no force merge, no snapshot, no shard relocation, and no hot node resource consumption.

Diagram comparing UltraWarm and writable warm data flows. In the UltraWarm legacy flow, data is ingested into the hot tier, migrated to read-only UltraWarm, and any update requires a round trip back to hot. In the writable warm flow, indices transition from hot to writable warm, which accepts reads and writes directly without migrating back to hot.

UltraWarm (legacy) data flow: Data is ingested into the hot tier (SSD, read and write). Index State Management (ISM) policies migrate indices to UltraWarm (Amazon S3-backed, read-only). Any update requires migrating the index back to hot (dashed arrow), writing, then migrating back.

Writable warm (new) data flow: Same ingestion path through hot, with ISM transitioning indices to writable warm. The key difference is that writable warm supports both reads and writes. Late-arriving updates go directly to warm, with no migration back to hot. Because both tiers use Amazon S3 as durable storage through OpenSearch Optimized instances, transitions are lightweight shard relocations, not resource-intensive migrations.

The benefits: cost, operations, and flexibility

Writable warm delivers advantages in three areas: cost, operational simplicity, and flexibility.

Cost

Unlike UltraWarm, which only offers on-demand pricing, OI2 instances support Reserved Instance (RI) pricing, a commitment-based discount model. By committing to a 1-year or 3-year Reserved Instance, you can save 31–52 percent compared to UltraWarm nodes. This makes writable warm significantly more cost-effective for predictable, long-running workloads. The newly introduced Database savings plan for OpenSearch Service provides savings of around 22 percent over UltraWarm instances. Both tiers use Amazon S3 for durable storage, so node failure means only temporary unavailability, not data loss. For cost-sensitive workloads that can tolerate brief downtime during node recovery, you can configure zero replicas on warm indices to reduce costs further.

Real-world cost comparison

Consider a workload ingesting 2 TB/day with 210 days total retention, where updates can arrive at any point. With UltraWarm’s read-only constraint, you must keep data in hot for 30 days before migrating to warm. With writable warm, updates happen directly on warm, so hot retention drops to only 7 days.

At small scale, the hot tier reduction benefit is modest. Writable warm is still cost-effective if you need write capability on warm data, can commit to RI pricing, or value the operational simplicity of eliminating migration pipelines. For purely immutable data with short retention, UltraWarm on-demand might still be cheaper. Use the AWS Pricing Calculator to model your specific scenario.

The following table shows estimated monthly costs using on-demand and All Upfront Reserved Instance (AURI) pricing in the US East (N. Virginia) Region as of March 2026. For the latest pricing, see Amazon OpenSearch Service pricing on the AWS website.

Component Hot + UltraWarm (30d hot / 180d warm) Hot + writable warm (7d hot / 203d warm)
Hot data nodes $12,264 (21 × or2.2xlarge) $12,264 (21 × or2.2xlarge)
Hot EBS cost $10,212.84 (21 * 3986 GB) $2,636
Hot remote storage $2,008.28 $518
Warm data nodes $39,128 (20× ultrawarm1.large) $50,409 (15× oi2.8xlarge)
Amazon S3 storage $9,504 $1,070
Leader nodes $1,307 (3 × m8g.2xlarge) $1,307 (3 × m8g.2xlarge)
On-demand total $74,427 $69,297
1-year AURI $69,674 $43,918 (~36% less)
3-year AURI $67,367 $34,939 (~48% less)
Database savings plan $71,708 $55,406 (~22%)

Operations

Reclaim hot node capacity. Writable warm removes two common causes of hot node over-provisioning: reserving 35 percent of disk space for force merge operations, and maintaining extra capacity to temporarily move data back to hot for writes. You can run your hot tier at higher utilization, which reduces the number of hot nodes you need.

Simpler migrations. UltraWarm migrations are multi-step operations (force merge, snapshot, and shard relocation) that need careful scheduling during low-traffic windows, and they are limited to 10 queued at a time. Writable warm simplifies this to a lightweight shard relocation, with more straightforward ISM policies and no scheduling constraints.

Flexibility

UltraWarm offers only two instance sizes: ultrawarm1.medium (1.5 TiB) and ultrawarm1.large (20 TiB). Writable warm with OI2 instances offers a full range from oi2.large to oi2.16xlarge. Each size addresses up to 5× its local cache size, so you can right-size warm capacity precisely to your workload.

Search performance

We benchmarked search latency using the NYC Taxis workload, comparing writable warm (oi2.large) against UltraWarm nodes. All measurements are P90 latencies.

On the NYC_TAXIS benchmark, writable warm matched or beat UltraWarm on 6 of 7 query types at P90, including lightweight filters, ranges, sorts, and time-histogram aggregations. For most real-world search patterns, writable warm delivers comparable or better performance than UltraWarm, plus the ability to write directly to the tier.

Search performance: writable warm compared to UltraWarm

Task Writable warm node latency in ms UltraWarm latency in ms UltraWarm vs. writable warm diff %
NYC_TAXIS workload type ** ** ** ** ** **
default (P90) 21.287 23.857 12.07223
range (P90) 21.23 21.016 -1.00718
distance_amount_agg (P90) 5,069 3929.23 -22.48406
autohisto_agg (P90) 21.076 22.002 4.39348
date_histogram_agg (P90) 21.363 21.792 2.01031
desc_sort_tip_amount (P90) 23.224 23.797 2.46636
asc_sort_tip_amount (P90) 22.483 22.482 -0.00445

When to choose what

Should you switch from UltraWarm to writable warm? It depends on your workload.

Requirement Writable Warm UltraWarm
Write enabled ✓ Read-only
Reserved Instance pricing ✓ ✗
Instance size flexibility Wide range (large–8xlarge) 2 options only
Cold tier support ✗ ✓
Need for OpenSearch Optimized instance families ✗ ✓
Concurrent tier transitions ✓ ✗ (sequential)
Hot node impact during migration Minimal High (CPU/memory)

Clean up resources

If you created a test domain to evaluate writable warm storage, delete it to avoid ongoing charges. In the OpenSearch Service console, select your domain and choose Delete. This removes all nodes and stops Amazon S3 storage charges for that domain.

Summary

In this post, I showed you how writable warm storage eliminates the costly migration cycle that UltraWarm’s read-only limitation creates. You get up to 36 percent cost savings with 1-year Reserved Instances, faster search performance, and a simpler operational model. Writable warm also removes data transitions between tiers, and Reserved Instance pricing becomes available for warm storage for the first time.

Writable warm requires OpenSearch Service version 3.3 or later with OI2 instances. For domains needing cold tier support, earlier OpenSearch Service versions, or non-optimized instance families, UltraWarm remains the right choice.

Next steps: Start by analyzing your current hot and warm split. How many days of data do you keep in hot only to accommodate occasional updates? Use the AWS Pricing Calculator to model your potential savings, and enable writable warm on a test domain in minutes. At the time of this post, writable warm is supported on OpenSearch Service version 3.3. For step-by-step instructions, see Migrating to writable warm storage in the OpenSearch Service documentation.

Have you tried writable warm storage? I’d love to hear about your experience and any questions you have in the comments.


About the author

Bharav Patel

Bharav Patel

Bharav is a Specialist Solution Architect, Analytics at Amazon Web Services. He primarily works on Amazon OpenSearch Service and helps customers with key concepts and design principles of running OpenSearch workloads on the cloud. Bharav likes to explore new places and try out different cuisines.

How BigBasket uses the Iceberg based lakehouse architecture on AWS to power lightning-fast grocery delivery across India

Post Syndicated from Annie Mattoo original https://aws.amazon.com/blogs/big-data/how-bigbasket-uses-the-iceberg-based-lakehouse-architecture-on-aws-to-power-lightning-fast-grocery-delivery-across-india/

Delivering fresh groceries to millions of customers across India in a few minutes demands a radically modern data architecture and resilient processes to help the business make faster decisions. This is what BigBasket was able to achieve by building a lakehouse architecture on AWS.

In this post, we demonstrate how BigBasket implemented the lakehouse architecture on AWS, including their architecture decisions, implementation approach, and the measurable business results you can expect from a similar modernization. Whether you’re facing scalability challenges or planning your own lakehouse implementation, this blueprint provides actionable insights you can adapt for your organization.

About BigBasket

BigBasket (Innovative Retail Concepts Private Limited) is India’s largest online supermarket, serving millions of customers across over 60 cities. Founded in 2011, the company offers groceries, fresh produce, household items, and personal care products through its mobile app and website, operating subscription services (BBDaily) and quick commerce (bbnow). For BigBasket, the ability to deliver groceries on time isn’t only a competitive advantage. It’s the foundation of customer trust, where every minute counts.

However, rapid business growth brought significant operational challenges:

  • Inability to consistently meet on-time delivery adherence because of high order volumes, extended travel times, and more, directly impacting key metrics like on-time rate (OTR)-10 mins and OTR-15 mins.
  • Struggling to meet on-time delivery targets because of picking inefficiency, high order volumes, and extended travel times, directly impacting key metrics like OTR-10 mins and OTR-15 mins.
  • Delays in stock availability impacting vendor fill-rates, inter-distribution center orders, and warehouse operations.
  • Inaccurate stock forecasting for top-selling stock keeping units (SKUs), assortment variety, event SKUs, store capacity, and buying cycles.
  • Lower dark store productivity across picking, stacking, order processing, and goods receipt notes (GRN).

Behind these business challenges lay a fundamental technology problem: the existing data infrastructure couldn’t keep pace. The company experienced rapid store growth, expanding 4x in a short timeframe, which exposed several limitations within their existing data architecture that needed attention.

Understanding the technical bottlenecks

BigBasket’s initial architecture relied heavily on a single data warehouse built on Amazon Redshift to meet all reporting and dashboarding needs. While this traditional approach had served them well initially, several important limitations emerged:

  • Stale data: Extract, transform, load (ETL) pipelines delivered only day-old (D-1) data, making near real-time analysis impossible for dashboard requirements.
  • Extended recovery times: Pipeline failure recovery processes took several hours, causing significant delays in data availability for business users.
  • Schema rigidity: Schema changes in source databases frequently triggered pipeline failures because of a lack of schema evolution support.
  • Scalability constraints: The infrastructure struggled to handle the sudden load increase from 13,000 to over 35,000 transactions for reports and dashboards with more than 1,000 dataset refreshes.
  • Cost implications: Increasing data volumes demanded additional compute resources, driving up costs.

Diagram of the scalability and cost limitations of BigBasket’s legacy Amazon Redshift data warehouse

It became clear that the existing data infrastructure wasn’t able to meet the evolving business requirements and a redesign of their data architecture is needed.

Why lakehouse architecture?

A modern data lakehouse architecture addresses these issues with near real-time data processing, flexible schema evolution, and scalable analytics, capabilities necessary for fast-moving commerce operations. The lakehouse approach combines the flexibility and cost-effectiveness of data lakes with the performance and governance features of data warehouses, combining the strengths of both. The design of a data lakehouse provides interoperability across storage systems for combined analytics activities.

Solution overview

BigBasket partnered with AWS to implement a comprehensive lakehouse architecture using a combination of AWS native services and open-source technologies.

The following diagram shows an elaborated view of Bigbasket’s modernized architecture on AWS.

Detailed lakehouse data flow across bronze, silver, and gold medallion layers on AWS

Data ingestion: Enabling continuous replication

AWS Database Migration Service (AWS DMS) ingests data from online transaction processing (OLTP) databases running on Amazon Relational Database Service (Amazon RDS) into the lakehouse on AWS.

This method continuously replicates data with minimal latency, so your analytics reflect near real-time business operations.

Storage and governance: Building a solid foundation

The lakehouse is built on Amazon Simple Storage Service (Amazon S3) and Amazon Redshift, which serve as the centralized data lake and warehouse following a medallion architecture.

The architecture persists all analytical data using Apache Iceberg as the open table format. Iceberg provides a robust foundation for large-scale analytics with the following capabilities:

  • ACID transactions: Guarantees data consistency and correctness across concurrent read and write operations.
  • Time travel: Supports querying historical table versions for auditing, troubleshooting, and recovery.
  • Schema evolution: Allows schema changes without disrupting existing queries or downstream pipelines.

The medallion architecture structures data across three logical layers within the lakehouse:

  • Bronze layer: Implements change data capture (CDC)-based source replication using AWS DMS. Raw change events flow into Amazon S3 as Apache Parquet files in their original format from source systems, preserving the complete change history. The data pipeline processes and deduplicates these events using Apache Spark on Amazon EMR to create and maintain Apache Iceberg tables that act as replicated source tables.
  • Silver layer: Represents the conformed data model, where data is cleansed, standardized, and validated with enforced quality checks. This layer contains core dimension and fact tables, modeled for analytical consistency and reuse across domains. Data is stored as Apache Iceberg tables on Amazon S3, making it reliable and performant for downstream analytics and transformations.
  • Gold layer: Provides business-ready data marts and wide tables optimized for reporting, dashboarding, and domain-specific use cases. These datasets are curated to align with business metrics and key performance indicators (KPIs) and are served from Amazon Redshift, using Iceberg-backed tables to deliver fast, scalable analytics for business intelligence (BI) tools and end users.

This layered approach maintains a clear separation of concerns across raw ingestion, analytical modeling, and business consumption, while supporting scalability and flexibility across the organization. AWS Lake Formation enforces fine-grained data access controls, and the AWS Glue Data Catalog centrally manages metadata across Amazon S3 and Amazon Redshift, ensuring consistent data discovery and governance across the analytics ecosystem.

Data processing: Flexibility and performance

For data processing and transformations, BigBasket uses Amazon EMR with Apache Spark and dbt, orchestrated by Apache Airflow running on Amazon Elastic Kubernetes Service (Amazon EKS) as the core compute layer of the lakehouse. Apache Spark on Amazon EMR handles large-scale distributed processing, including CDC deduplication, incremental transformations, and complex data reshaping. Apache Iceberg serves as the open table format, which provides several critical capabilities.

dbt is used to define and execute transformation logic using SQL, managing the build of data models such as staging, intermediate, and final tables on top of the raw data. dbt uses the dbt-Trino adapter to run these transformations using the Trino engine, materializing the results as Apache Iceberg tables in Amazon S3. This approach provides a simple, modular, and governed way to manage transformations while taking advantage of Iceberg’s transactional guarantees.

These features are necessary for production lakehouse implementations and help you avoid vendor lock-in while maintaining enterprise reliability.

Online analytical processing (OLAP) and analytics: Hybrid approach for cost optimization

The analytics layer uses a hybrid approach that you can adapt based on your query patterns:

  • Amazon Redshift: For querying of active, frequently accessed data from the Gold layer.
  • Amazon Athena: For ad-hoc queries on historical data.
  • Apache Trino: For federated queries across multiple data sources while powering dbt-driven transformations directly on Apache Iceberg tables.

This hybrid strategy optimizes costs by keeping frequently accessed data in Amazon Redshift while querying historical data directly from Iceberg tables in Amazon S3. Amazon Redshift data sharing supports a multi-warehouse architecture for cross-team collaboration, allowing different teams to access shared datasets without data duplication.

Orchestration: Managing complex workflows

Apache Airflow running on Amazon EKS orchestrates and schedules data pipelines across the entire environment, providing visibility and control over complex workflows. This gives you a unified view for monitoring and managing your data operations.

Machine learning integration

Amazon SageMaker AI powers machine learning workloads for predictive analytics and model training directly on lakehouse data, from demand forecasting to delivery optimization. This tight integration means your data scientists can work with the same governed data that powers your analytics.

Visualization: Making insights accessible

Amazon Quick Sight provides data visualization and business intelligence reporting capabilities, making insights accessible to business users across the organization without requiring technical expertise.

Special focus: Clickstream data processing

BigBasket implemented a sophisticated dual-path architecture for processing clickstream data from mobile apps and web interactions:

  • Real-time path: Data flows through Scala stream collectors on Amazon Elastic Compute Cloud (Amazon EC2) (behind Elastic Load Balancing) to Amazon Kinesis Data Streams and Amazon OpenSearch Service for immediate insights into customer behavior. This path is necessary when you need to react to user actions within seconds, for example detecting fraud or personalizing experiences in real time.
  • Batch path: The batch path validates data, stores it in Amazon S3, processes it through Amazon EMR, and loads it into Amazon Redshift for comprehensive historical analysis. This path handles data quality checks, enrichment, and aggregation for long-term analytics.

The trade-off between these approaches is latency versus completeness. Real-time processing gives you speed but may sacrifice some data quality checks, while batch processing provides accuracy but introduces delay. This dual approach achieves both immediate operational insights and deep analytical capabilities, letting you optimize for different use cases.

The following diagram shows how the clickstream data is handled and effectively processed today.

BigBasket’s dual-path clickstream processing architecture with real-time and batch paths on AWS

The results: measurable business impact

The data platform transformation achieved significant results across multiple dimensions:

Technical improvements

  • Near real-time data: Achieved near real-time data availability for dashboards within 3–5 minutes, replacing previously day-old data.
  • Rapid failure recovery: Pipeline failure re-runs now complete in minutes instead of hours.
  • Comprehensive governance: Full control over data governance with robust observability, lineage, data accuracy, and consistency.
  • Enhanced scalability: Successfully handling over 35,000 reports and dashboards with over 1,000 dataset refreshes.

Business outcomes

  • On-time delivery: Improved monitoring with real-time insights on low-performing stores.
  • Stock availability: Reduced operational issues with visibility into key bottlenecks.
  • Stock forecasting: Improved accuracy and availability of top-selling SKUs.
  • Dark store productivity: Enhanced productivity of warehouse executives across all operations.

Key takeaways: lessons for modern data platforms

BigBasket’s journey offers valuable insights for organizations facing similar challenges:

  1. Quick commerce needs quick observability. In the fast-paced world of quick commerce, faster decision-making directly improves business metrics. Real-time data isn’t a luxury. It’s a necessity.
  2. Embrace ELT for real-time needs. Shifting from traditional ETL to an extract, load, transform (ELT) pattern within a lakehouse architecture is important to unlock near real-time analytics capabilities.
  3. A lakehouse delivers speed and governance. Modern lakehouse architectures don’t force trade-offs. You can achieve both fast data availability and comprehensive control, lineage, and accuracy.
  4. Focus on operational resilience. Designing for rapid failure recovery (re-runs in minutes, not hours) is necessary for maintaining data availability and business trust, especially in customer-facing operations.
  5. Incremental migration. You don’t need to rebuild everything. Evolve your current Amazon S3 data lake or reuse your existing investments in Amazon Redshift to build the data lakehouse capabilities.

The road ahead

BigBasket continues to innovate, now moving to adopt Amazon SageMaker Unified Studio to access all lakehouse components in a simplified manner across the enterprise. This next evolution will further streamline data access and accelerate insights across teams.

The company’s transformation demonstrates that with the right architecture and AWS services, organizations can turn data infrastructure challenges into competitive advantages, delivering not only better analytics but better customer experiences.

As you plan your own lakehouse implementation, use these patterns and lessons learned to accelerate your journey and avoid common pitfalls.


About the authors

Naga Sandeep Grandhi

Naga Sandeep Grandhi

Sandeep is an engineering leader at BigBasket, driving data platform and cloud architecture initiatives, including the next-gen data lake built for scale, reliability, and real-time insights.

Vikram Kumar

Vikram Kumar

Vikram is a Principal Engineer at BigBasket, where he leads the data engineering team. He specializes in designing and scaling modern data platforms on AWS, enabling BigBasket to process large-scale data efficiently and power data-driven decision-making across the organization.

Annie Mattoo

Annie Mattoo

Annie is a Sr. Analytics Specialist at AWS, bringing over 15+ years of expertise in helping customers with their DATA & AI journeys. She has successfully led customer teams to seamlessly adopt AWS Data & AI services and has worked with Fortune 500 customers across the globe in her previous roles.

Vineet Thapliyal

Vineet Thapliyal

Vineet is an Enterprise Account Manager at Amazon Web Services (AWS) in Bengaluru, India, where he manages strategic cloud and generative AI engagements across some of India’s largest conglomerates spanning energy, retail, and technology. He is passionate about helping enterprises unlock business value through AI/ML, cloud modernization, and industry-specific innovation — from renewable energy analytics to retail transformation at scale.

Anirudh Chawla

Anirudh Chawla

Anirudh is an Analytics Solution Architect at AWS. He helps organization empowers businesses to harness their data effectively through AWS’s analytics platform. His interest lies in building highly available distributed systems.

Deploy modern data platforms in minutes with MDAA

Post Syndicated from Sudeshna Dash original https://aws.amazon.com/blogs/big-data/deploy-modern-data-platforms-in-minutes-with-mdaa/

Modern Data Architecture Accelerator (MDAA) is an open source framework that replaces infrastructure code with concise YAML configuration, so your team can deploy a governed, production-ready data architecture, reducing deployment time from months to weeks (depending on complexity and team experience).

Organizations building modern data architecture on AWS face a critical challenge: deploying production-ready, governed infrastructure traditionally requires 6–12 months of custom development, thousands of lines of infrastructure code, and continuous remediation cycles to maintain security and compliance. Governance is often added incrementally, treated as an afterthought that creates compliance gaps and engineering rework.

MDAA addresses this by replacing infrastructure code with concise YAML configuration, achieving up to 97.6 percent code reduction (from approximately 1,800 lines of AWS CloudFormation to 45 lines of MDAA YAML) while embedding governance from the start. The complete Governed Lakehouse Starter Kit deploys 491 AWS resources across 12 stacks from approximately 450 lines of YAML configuration, representing a 66x verbosity ratio where each line automatically expands into production-ready infrastructure.

In this post, we explore how MDAA transforms data architecture development from months of manual coding to production-ready deployment through configuration-driven infrastructure and embedded governance, examine a real customer transformation, and provide a clear implementation pathway for your own data modernization journey.

Customer use case and challenge

A university system office needed to modernize its analytics architecture across 17 campuses while managing sensitive educational data. Their third-party dependency created bottlenecks that slowed feature implementation from weeks to months, and their IT team lacked the cloud skillsets to build modern infrastructure independently.

With MDAA, they achieved:

  • 95 percent reduction in time-to-value for dashboard and feature implementation (from weeks to hours).
  • 17 campuses integrated into a unified, secure architecture.
  • 7.2TB of data and over 8,000 dashboards migrated successfully.
  • Significant cost savings by removing third-party dependencies and reducing license costs.
  • Enhanced security posture for external stakeholders accessing sensitive educational data.

The team used MDAA to implement a modernization strategy with continuous integration and continuous delivery (CI/CD) for automated deployment. The architecture now supports rapid response to stakeholder requests while maintaining strict data governance through AWS Lake Formation.

Their transformation demonstrates what becomes possible when governance is embedded from launch rather than added incrementally, moving from months-long manual development to weeks of production-ready deployment through configuration-driven infrastructure.

Solution: MDAA and its value propositions

MDAA’s capabilities stem from its modular, composable architecture. The accelerator provides over 40 pre-built modules that encapsulate AWS best practices for security, governance, and operational excellence. Organizations describe the outcomes they want in MDAA-specific YAML configuration files (not CloudFormation or Terraform YAML) and the accelerator automatically translates these configurations into AWS Cloud Development Kit (AWS CDK) constructs, which then deploy via CloudFormation with embedded governance.

Configuration over code. The MDAA framework takes a fundamentally different approach: describe the outcomes you want in YAML, and the accelerator deploys production-ready infrastructure with embedded governance. Consider deploying a governed data lake where fraud detection teams need write access to transaction data, while marketing analytics teams require read-only access to customer behavior data. Traditional approaches require over 1,800 lines of CloudFormation across Amazon Simple Storage Service (Amazon S3) buckets, AWS Key Management Service (AWS KMS) keys, AWS Identity and Access Management (IAM) policies, and Lake Formation permissions. With MDAA, the same governed data lake is expressed in 45 lines of configuration, a 97.6 percent reduction, while helping you apply encryption, least-privilege access, and cross-account governance as built-in defaults.

The configuration deploys multi-zone S3 storage with KMS encryption, Lake Formation permissions with tag-based access control (TBAC) enabled, Amazon SageMaker Unified Studio for data product discovery, and encrypted AWS Glue Data Catalog with automated crawlers. All permissions flow through Lake Formation rather than individual IAM policies.

Embedded governance from day one. Governance is declared in YAML and deployed alongside infrastructure from the first run. Fine-grained access controls, encrypted data catalogs, data quality validation, audit trails, and sensitive data classification are all part of the same configuration. MDAA’s Governed Lakehouse starter kit defines an entire governed data architecture in roughly 450 lines of YAML, which produces approximately 29,700 lines of CloudFormation across 12 stacks (a 98.5 percent reduction in infrastructure code).

Modular, composable architecture. Each module is purpose-built to handle a specific capability within the data architecture. Modules communicate through AWS Systems Manager Parameter Store, passing resource identifiers (Amazon Resource Names (ARNs), IDs, and names) between stacks. This approach removes hardcoded dependencies. A KMS key created in one module can be referenced by another through parameter resolution, with all dependencies resolved automatically at deployment time.

The diagram illustrates the deployed architecture and team-level access flow that MDAA generates from the 45-line configuration.

Progressive architecture patterns. MDAA provides four reference architecture patterns that align to progressive stages of data infrastructure maturity:

  • Basic Data Lake deploys a governed data lake with built-in security controls, data quality checks, centralized metadata management using AWS Lake Formation and AWS Glue.
  • Data Science Platform extends the data lake with Amazon SageMaker notebooks, feature stores, and machine learning (ML) pipelines so data science teams can experiment and train models on governed data.
  • SageMaker Unified Studio adds a single interface for analytics and ML collaboration, connecting data engineers, analysts, and data scientists in one workspace.
  • Generative AI Platform layers Amazon Bedrock and Retrieval Augmented Generation (RAG) capabilities on top of your existing data foundation, so teams can build generative AI applications grounded in enterprise data.

Each pattern builds the one before it. You can start with the Basic Data Lake and adopt additional patterns as your team’s needs grow. MDAA’s modular design means you add capabilities without rearchitecting what you already deployed.

The infrastructure is versioned through GitHub, repeatable across environments, and auditable through comprehensive AWS CloudTrail logging. Data engineers focus on data pipelines and business logic while MDAA manages infrastructure complexity and governance integration. This represents the fundamental shift: from writing infrastructure code to describing the outcomes you want through configuration, with governance embedded from the start.

Use case of MDAA: Governed data architecture

DataOps teams spend significant time on governance tasks, including permissions management, compliance validation, and access control, rather than building pipelines and analytics. These aren’t data problems, they’re governance problems that consume engineering capacity meant for higher-value work. MDAA addresses this at the architectural level. Governance is declared in YAML and deployed alongside infrastructure from the first run.

The following sections walk through how each governance module works in practice.

Publish, discover, subscribe, and consume data products between business units: SageMaker Unified Studio

Amazon SageMaker Unified Studio provides a governed data catalog where data producers publish data products, and consumers discover and subscribe to them. Your deployment with MDAA includes a pre-configured domain, blueprints (managed and custom), projects, and environment profiles, all defined in a single configuration file:

# sagemaker.yaml --- 16 lines that deploy 114 CloudFormation resources
domains:
  domain1:
    dataAdminRole:
      id: ssm:/{{org}}/govern1/generated-role/data-admin/id
    description: SMUS Domain 1
    userAssignment: MANUAL

    tooling:
      vpcId: '{{context:vpc_id}}'
      subnetIds:
        - '{{context:private_subnet_id1}}'
        - '{{context:private_subnet_id2}}'

    groups:
      team1:
        ssoId: '{{context:team1-group-sso-id}}'
      team2:
        ssoId: '{{context:team2-group-sso-id}}'

Behind this configuration, MDAA deploys an Amazon SageMaker Unified Studio domain with dedicated KMS keys, execution and provisioning roles, and single sign-on group profiles for team access. Data producers tag and publish assets with metadata, ownership, and classification. Consumers browse a searchable catalog, see only authorized assets, and request access through a governed workflow. Cross-account and cross-business-unit data sharing flows through a subscription model, ensuring every access grant is tracked, auditable, and revocable.

Use case of MDAA: Restricting access to cardholder data using Lake Formation

AWS Lake Formation provides fine-grained access control at database and table levels, removing manual IAM policy management. MDAA deploys AWS Lake Formation with pre-configured settings that disable IAMAllowedPrincipals, the critical governance setting that ensures all permissions flow through centralized governance:

# lakeformation-settings.yaml --- 6 lines that deploy 25 CloudFormation resources
lakeFormationAdminRoles:
  - id: generated-role-id:data-admin
createCdkLFAdmin: true
createDataZoneAdminRole: true
iamAllowedPrincipalsDefault: false

That last flag is the single most important governance setting in the platform. Without it, an IAM principal with glue:GetTable can read tables in the catalog, bypassing the entire access control model. Most manual setups miss this or defer it.

With the data lake configuration, you declare roles and access policies in YAML where admins get full control, engineers get read access to curated data, extract, transform, and load (ETL) roles get scoped write access, and MDAA compiles them into the correct S3 bucket policies and Lake Formation registrations.

Use case of MDAA: Ensuring data integrity with AWS Glue Data Quality

AWS Glue Data Quality runs automated validation rulesets continuously as part of the pipeline, not as periodic batch checks. MDAA’s data quality module supports over 15 built-in rule types, from completeness and uniqueness checks to statistical thresholds and data freshness validation:

# data-quality.yaml
projectName: example-project

rulesets:
  customer-data-quality:
    description: Validate customer data completeness and uniqueness
    targetTable:
      databaseName: project:databaseName/customer-data
      tableName: customers
    ruleset:
      - ruleType: IsComplete
        column: customer_id
      - ruleType: Uniqueness
        column: email
        comparisonOperator: ">"
        threshold: 0.95
      - ruleType: RowCount
        comparisonOperator: ">"
        value: 100

Quality metrics flow into Amazon CloudWatch for real-time alerting. If anomalies are detected, automated workflows quarantine affected records and alert data engineering teams before issues reach downstream consumers.

Protecting metadata at rest: AWS Glue Data Catalog encryption

Table schemas, column names, and partition structures can reveal sensitive information about an organization’s data architecture, even without access to the underlying data. AWS Glue Catalog Encryption secures metadata at rest using AWS KMS-managed keys. MDAA configures catalog encryption by default, so schema definitions and connection passwords are encrypted from initial deployment without requiring manual key management setup. Access to catalog metadata follows the same Lake Formation governance controls applied to the data itself, so teams see only the schemas that they’re authorized to query.

Auditing every data access event: CloudTrail integration

Every data access event must be logged and attributable to a specific identity. Without a complete audit trail, demonstrating compliance during a regulatory review becomes a manual, error-prone process. AWS CloudTrail captures API-level activity across the data infrastructure, recording who accesses what data, when, and from which service. MDAA configures CloudTrail integration by default, so audit logging is active from initial deployment rather than added retroactively. Log data flows into a centralized, tamper-resistant store, giving compliance teams a single location to query access history across all business units and accounts.

Identifying sensitive data automatically: Macie integration

In large environments, sensitive information spreads across dozens of S3 buckets through pipelines, transforms, and ad hoc data drops, and self-reporting data owners consistently produce gaps. Amazon Macie uses machine learning to automatically discover and classify sensitive data in S3, surfacing findings at the object level without manual tagging. MDAA configures Macie across your S3 buckets during deployment, routing findings to Amazon EventBridge where automated workflows can alert owners or trigger remediation.

Together, these controls form a layered defense: Lake Formation governs access to cataloged data, Glue Data Quality validates integrity on arrival, and Macie identifies sensitive data that lands outside governed pipelines to reduce compliance risk.

Multi-account data mesh

MDAA provides extensive support for multi-account data mesh setups, with decentralized data ownership across business units and centralized governance. The data mesh starter kit supports cross-account data product publishing and consumption, allowing organizations to scale data sharing while maintaining consistent security and compliance controls.

Technical implementation

Ready to deploy your modern data architecture? Here are the resources to get started:

MDAA Implementation Guide provides detailed instructions for deploying all starter packages, including architecture patterns, configuration examples, security best practices, and troubleshooting guidance.

MDAA Hands-on Workshop offers step-by-step guided implementation with AWS experts. The workshop covers configuration management best practices, implementation patterns, hands-on labs with real-world scenarios, and cleanup instructions.

GitHub Repository and Documentation provide source code, module reference, and comprehensive documentation.

Organizations approach MDAA from different starting points. Some modernize existing data architectures, migrating from on-premises infrastructure or legacy cloud architectures. Others build new architectures for artificial intelligence and machine learning (AI/ML) initiatives or generative AI applications. Financial services organizations require PCI-DSS compliance from day one. Healthcare organizations need controls that can help support HIPAA. Each journey benefits from MDAA’s configuration-driven approach and embedded governance.

Conclusion

MDAA transforms data architecture development from months of manual coding to production-ready deployment. Configuration-driven infrastructure reduces development time by 40–60 percent while embedding governance from the start. The university system’s 95 percent reduction in time-to-value demonstrates the outcome: organizations deploy secure, compliant, governed data architectures in weeks rather than months.

Financial services organizations can deploy architectures to help them align with PCI-DSS compliance requirements using Lake Formation access controls, Glue Data Quality validation, SageMaker Unified Studio data discovery, comprehensive CloudTrail audit trails, and automated Macie data classification, all inherited from configuration rather than built manually.

Data architecture journeys need not follow six-month timelines with governance added incrementally. MDAA provides an alternative: describe the outcomes you want through YAML configuration, inherit pre-validated security controls, and deploy production-ready infrastructure with comprehensive governance from initial deployment.

Security and compliance is a shared responsibility between AWS and the customer. For more information, see the AWS Shared Responsibility Model.

Need help or have questions? Contact AWS ProServe for personalized guidance on selecting the right package and deployment strategy for your organization.


About the author

Sudeshna Dash

Sudeshna Dash

Sudeshna is a Data Scientist at AWS Professional Services based in Berlin, Germany. She specializes in data architecture, generative AI, and agentic AI systems on AWS. Sudeshna is a contributor to the Modern Data Architecture Accelerator (MDAA) open-source project and helps customers design and deploy governed, production-ready data and AI/ML architectures on AWS.

John Reynolds

John Reynolds is a Principal Engineer with AWS Professional Services based in Seattle, Washington. He leads the architecture and development of Modern Data Architecture Accelerator (MDAA), focusing on turning proven delivery patterns into reusable, production-ready foundations that customers can adopt and extend at scale.

Modernizing Lambda + S3 workloads with Amazon S3 Files

Post Syndicated from Sahithi Ginjupalli original https://aws.amazon.com/blogs/compute/modernizing-lambda-s3-workloads-with-amazon-s3-files/

Learn how Amazon S3 Files simplifies Lambda functions by eliminating transfer code and /tmp constraints. See three modernization patterns with code examples for image processing, ETL pipelines, and multi-agent AI workloads.

AWS Lambda functions that interact with Amazon Simple Storage Service (Amazon S3) typically follow a familiar pattern: download an object to /tmp, process it locally, and upload the result back to S3. This pattern is well-understood and reliable, but it requires you to write code for managing transfers, monitoring /tmp capacity, and cleaning up ephemeral storage alongside your actual processing logic.

Amazon S3 Files changes this by letting your Lambda function mount an S3 bucket as a file system. Your function reads and writes files at a local mount path (such as /mnt/data), and the file system handles synchronization with S3 automatically. The transfer and storage management code goes away, and what remains is your processing logic working directly with files.

In this post, we walk through three common Lambda + S3 workloads and show how to modernize each one by using S3 Files. You will see how the code gets shorter, the /tmp size constraint disappears, and the developer experience improves.

Walkthrough

Prerequisites

Before you begin, make sure you have:

  • An AWS account with permissions to create Lambda functions, S3 file systems, and VPC resources.
  • An existing VPC with private subnets and appropriate security groups.

Getting started

To integrate a Lambda function with S3 Files, you can follow these three steps:

  1. Create an S3 file system for your bucket. You can do this through the S3 console, AWS Command Line Interface (AWS CLI), or AWS CloudFormation. This single operation creates the file system, mount targets in your Amazon Virtual Private Cloud (Amazon VPC), and an access point.
  2. Add the file system configuration to your Lambda function. Specify the access point ARN and local mount path (for example, /mnt/data). Your function must be in a VPC with access to the mount target. For optimal throughput on large files, configure your function with 512 MB or more of memory to enable direct reads from S3.
  3. If you are modernizing your existing Lambda function’s code, replace boto3 transfer code with file paths. Change s3.download_file(bucket, key, '/tmp/file') to open('/mnt/data/' + key) and remove upload and cleanup logic.

Your function’s execution role needs s3files:ClientMount and s3files:ClientWrite permissions (included in the AmazonS3FilesClientReadWriteAccess managed policy). For direct S3 reads on large files, also add s3:GetObject and s3:GetObjectVersion.

Pattern 1: Multi-agent shared workspace

Agentic AI workloads, where multiple autonomous agents collaborate on a task, require shared mutable state. Agents need to read each other’s outputs, write intermediate artifacts, and coordinate without tight coupling. With Lambda today, this typically means serializing state to S3 objects or Amazon DynamoDB between every step, adding latency and code for each handoff.

S3 Files gives multiple Lambda functions a shared file system. Agents communicate through the file system itself, with no S3 API calls and no serialization overhead.

Example: Collaborative research agents

Three Lambda functions mount the same S3 bucket at /mnt/workspace. An orchestrator prepares the task, research agents work in parallel, and a synthesis agent combines their findings:

import os
import json

WORKSPACE = "/mnt/workspace"

# --- Orchestrator Agent ---
def orchestrator_handler(event, context):
    session_id = event["session_id"]
    session_dir = f"{WORKSPACE}/sessions/{session_id}"

    os.makedirs(f"{session_dir}/research", exist_ok=True)
    os.makedirs(f"{session_dir}/output", exist_ok=True)

    # Write task assignments directly to shared workspace
    with open(f"{session_dir}/manifest.json", "w") as f:
        json.dump({
            "query": event["research_query"],
            "agents": ["market_analysis", "technical_review", "competitor_scan"],
            "status": "in_progress"
        }, f)

    return {"session_dir": session_dir}


# --- Research Agent (one of many, running in parallel) ---
def research_agent_handler(event, context):
    session_dir = event["session_dir"]
    agent_name = event["agent_name"]

    # Read task from shared workspace (no S3 GET call)
    manifest = json.load(open(f"{session_dir}/manifest.json"))

    # Perform research (invoke Amazon Bedrock, search, etc.)
    # TODO: Implement perform_research() for your use case
    findings = perform_research(manifest["query"], agent_name)

    # Write results to shared workspace (no S3 PUT call)
    with open(f"{session_dir}/research/{agent_name}.json", "w") as f:
        json.dump(findings, f, indent=2)

    return {"status": "complete", "agent": agent_name}


# --- Synthesis Agent ---
def synthesis_handler(event, context):
    session_dir = event["session_dir"]

    # Read all research outputs from shared directory
    all_findings = {}
    for f in os.listdir(f"{session_dir}/research"):
        with open(f"{session_dir}/research/{f}") as fh:
            all_findings[f.replace(".json", "")] = json.load(fh)

    # Synthesize and write final report
    # TODO: Implement synthesize_findings() for your use case
    report = synthesize_findings(all_findings)
    with open(f"{session_dir}/output/report.md", "w") as f:
        f.write(report)

    return {"report_path": f"{session_dir}/output/report.md"}

In the traditional approach, each agent would need to call s3.get_object() to read the manifest, s3.put_object() to write findings, and the synthesis agent would need to call s3.list_objects() then s3.get_object() for each result. That’s eight or more S3 API calls per workflow run replaced by file I/O.

What the shared workspace pattern gives you:

  • Agents discover each other’s outputs by listing a directory (no coordination logic needed).
  • Sessions, agents, and outputs map to directories, not flat object key conventions.
  • Close-to-open consistency means that when an agent closes a file after writing, the next agent to open it sees the complete content.
  • No need to marshal state into S3 PutObject calls between steps.

Pattern 2: Image thumbnail generation

The S3 thumbnail generator is a common Lambda + S3 pattern. An image is uploaded to S3, a Lambda function is triggered, it downloads the image, resizes it with Pillow, and uploads the thumbnail to a destination bucket.

The traditional approach

import boto3
import os
import uuid
from urllib.parse import unquote_plus
from PIL import Image

s3_client = boto3.client('s3')

def resize_image(image_path, resized_path):
    with Image.open(image_path) as image:
        image.thumbnail(tuple(x / 2 for x in image.size))
        image.save(resized_path)

def handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = unquote_plus(record['s3']['object']['key'])
        tmpkey = key.replace('/', '')
        download_path = '/tmp/{}{}'.format(uuid.uuid4(), tmpkey)
        upload_path = '/tmp/resized-{}'.format(tmpkey)

        s3_client.download_file(bucket, key, download_path)
        resize_image(download_path, upload_path)
        s3_client.upload_file(
            upload_path, '{}-resized'.format(bucket), 'resized-{}'.format(key)
        )

What this approach requires you to manage beyond the core resize logic:

  • Transfer orchestration: Downloading the source, uploading the result, and handling partial transfer failures.
  • Storage capacity: Both the source and resized image must fit in /tmp simultaneously.
  • Ephemeral storage cleanup: If the function fails mid-execution or is reused across invocations, orphaned files can accumulate in /tmp.
  • Redundant downloads: If the same image triggers a retry, it must be downloaded again.

With the file system approach

import os
from urllib.parse import unquote_plus
from PIL import Image

MOUNT = "/mnt/images"

def resize_image(image_path, resized_path):
    with Image.open(image_path) as image:
        image.thumbnail(tuple(x / 2 for x in image.size))
        image.save(resized_path)

def handler(event, context):
    for record in event['Records']:
        key = unquote_plus(record['s3']['object']['key'])
        input_path = f"{MOUNT}/source/{key}"
        output_path = f"{MOUNT}/resized/resized-{os.path.basename(key)}"

        os.makedirs(os.path.dirname(output_path), exist_ok=True)
        resize_image(input_path, output_path)

What changed

The function moves from a download-process-upload pipeline to direct file I/O. No boto3 client, no /tmp management, no upload step. The resize_image function is unchanged because it always worked with file paths. The difference is that those paths now point to a mounted S3 file system instead of ephemeral local storage.

You still handle errors in your processing logic (for example, invalid image formats). What you no longer need to handle are transfer-specific failure modes like partial downloads, failed uploads, or /tmp capacity checks.

Metric Traditional S3 Files
Lines of code (non-blank) 22 15
S3 API calls per invocation 2 (GET + PUT) 0
Max image size Source + output files share /tmp No /tmp constraint
boto3 dependency Required Not needed

Pattern 3: CSV-to-Parquet ETL pipeline

Another commonly used serverless ETL pattern is an S3 event triggers a Lambda function when CSV files land in a bucket. The function downloads the CSV, transforms it to Parquet by using pandas and pyarrow, and uploads the result.

The traditional approach

import boto3
import pandas as pd
import os

s3 = boto3.client("s3")
BUCKET = "data-pipeline-bucket"

def handler(event, context):
    key = event["Records"][0]["s3"]["object"]["key"]
    filename = os.path.basename(key)
    local_input = f"/tmp/{filename}"
    local_output = f"/tmp/{filename.replace('.csv', '.parquet')}"

    try:
        # Download from S3
        s3.download_file(BUCKET, key, local_input)

        # Check /tmp space (10 GB limit)
        tmp_usage = sum(
            os.path.getsize(f"/tmp/{f}")
            for f in os.listdir("/tmp") if os.path.isfile(f"/tmp/{f}")
        )
        if tmp_usage > 9 * 1024**3:  # 9 GB safety margin
            raise RuntimeError("Approaching /tmp storage limit")

        # Transform
        df = pd.read_csv(local_input)
        df["processed_at"] = pd.Timestamp.now()
        df.to_parquet(local_output, engine="pyarrow", compression="snappy")

        # Upload result back to S3
        output_key = key.replace("raw/", "processed/").replace(".csv", ".parquet")
        s3.upload_file(local_output, BUCKET, output_key)

        return {"status": "success", "output_key": output_key}

    finally:
        # Clean up /tmp
        for f in [local_input, local_output]:
            if os.path.exists(f):
                os.remove(f)

What this approach requires you to manage beyond the core transform logic:

  • Storage capacity: Source and output files share /tmp, limiting practical file size.
  • Cold start cost: Initializing the boto3 client adds startup latency.
  • Transfer failure modes: Partial downloads, failed uploads, and orphaned /tmp files need their own handling.
  • Redundant downloads: Retries or reprocessing require downloading the same file again.

With the file system approach

import pandas as pd
import os

MOUNT = "/mnt/data"

def handler(event, context):
    key = event["Records"][0]["s3"]["object"]["key"]
    input_path = f"{MOUNT}/{key}"
    output_path = f"{MOUNT}/{key.replace('raw/', 'processed/').replace('.csv', '.parquet')}"

    # Ensure output directory exists
    os.makedirs(os.path.dirname(output_path), exist_ok=True)

    # Transform: direct file access, no download/upload
    df = pd.read_csv(input_path)
    df["processed_at"] = pd.Timestamp.now()
    df.to_parquet(output_path, engine="pyarrow", compression="snappy")

    return {"status": "success", "output_path": output_path}

What changed

With this change, a developer reading this code sees only the transform logic (read CSV, add column, write Parquet). The storage mechanics are handled by the file system.

Metric Traditional S3 Files
Lines of code (non-blank) 33 14
S3 API calls per invocation 2 (GET + PUT) 0
Max file size Source + output share /tmp No /tmp constraint
Cleanup logic required Yes No
/tmp space monitoring Yes No

Choosing the right approach: file system mounts vs. traditional access

Use case Recommendation
Lambda reads/writes files from S3 S3 Files (eliminates transfer boilerplate)
Multiple functions share data S3 Files (shared mount replaces API coordination)
Files > 10 GB S3 Files (no /tmp size constraint)
Event-driven processing (trigger on upload) S3 Files (S3 event triggers still work, function reads from mount)
Direct S3 API features (presigned URLs, S3 Select, multipart upload) Traditional (these require the S3 API)
Functions outside a VPC Traditional (S3 Files requires VPC connectivity)

Cleaning up

If you created resources while following along with this post, delete them to avoid incurring future costs. Start by removing the file system configuration from your Lambda function settings. Next, remove the S3 file system, which also deletes its associated mount targets and access points. Then delete the S3 buckets used for source and output data, along with the Lambda functions created for the examples. Finally, remove the IAM roles and policies created for Lambda execution or, if you added the S3 Files permissions (s3files:ClientMount, s3files:ClientWrite, s3:GetObject, s3:GetObjectVersion) to an existing role, remove these permissions. Additionally, If you created a new VPC for this tutorial, delete the VPC, which will also remove the associated private subnets, security groups, and route tables. If you used an existing VPC, remove the security groups and subnets created for this testing.

Warning: Deletion of an S3 bucket and its contents permanently deletes all objects in the buckets and cannot be undone. Make sure you have backed up any data you need to retain before proceeding.

Conclusion

In this post, we demonstrated how to modernize three common Lambda + S3 workloads by using Amazon S3 Files. Across image thumbnail generation, ETL pipelines, and multi-agent AI workloads, the migration follows the same principle: replace S3 API transfer logic with native file I/O and let the file system handle synchronization.

The improvements are consistent:

  • Less code: Transfer and cleanup logic goes away, leaving only your processing logic.
  • No /tmp size constraint: Process large files without local storage limits.
  • Zero S3 API calls for data access: Reads and writes go through the file system mount.
  • Fewer failure modes to handle: Transfer-specific issues (partial downloads, failed uploads, orphaned temp files) no longer apply.

For teams running Lambda + S3 workloads today, S3 Files isn’t a new architecture to learn. It’s transfer code you can remove. To learn more, see the S3 Files section in the Lambda documentation. To track upcoming features on the AWS Lambda roadmap, you can refer to the AWS Lambda roadmap.

Why tombola chose Graviton-powered RG instances for Amazon Redshift

Post Syndicated from Prabhu Pandian original https://aws.amazon.com/blogs/big-data/why-tombola-chose-graviton-powered-rg-instances-for-amazon-redshift/

Part of Flutter Entertainment, the world’s largest online sports betting and iGaming operator, tombola is the world’s biggest online bingo community and has been using Amazon Redshift to run its data analytics workloads. Founded in Sunderland, UK, the company traces its roots to the 1950s, when it began printing bingo tickets during the golden age of the game. tombola launched online in 2006 and has since expanded to Italy, Spain, Denmark, and Sweden. The company builds all of its games in-house, holds the most prestigious Safer Gambling award, and recently partnered with Flutter sibling brand Sisal to bring its bingo application to Italian players.

In this post, you learn how tombola followed a strict engineering principle: no changes to production without evidence. That meant a head-to-head comparison of RA3 versus RG on their actual workload. You also see benchmark results on Amazon S3 Tables and the migration from RA3 to RG instances.

Current data architecture

Amazon Redshift sits at the center of tombola’s data architecture. The production cluster runs on RA3 nodes and serves multiple schemas with hundreds of tables, supporting every analytical workload the business runs, from sub-second application lookups to multi-minute extract, transform, load (ETL) transforms. What makes tombola’s Amazon Redshift workload distinctive is the breadth of what flows through it. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) DAGs orchestrate pipelines across over 14 business domains, including segmentation, fraud detection, marketing, finance, and SafePlay responsible-gaming. Configuration-driven ingestion pipelines land data from SQL Server, Amazon DynamoDB, Amazon OpenSearch Service, Postgres, and external APIs into Bronze and Silver layers on Amazon Simple Storage Service (Amazon S3), before loading it into Amazon Redshift. From there, over 250 dbt models running on Amazon Elastic Container Service (Amazon ECS) transform the data into analytical gold layers. Outputs feed multiple downstream consumers: Amazon SageMaker for fraud scoring and churn prediction, Amazon DynamoDB for low-latency APIs, and region-specific pipelines spanning the UK, Italy, Spain, Denmark, and Sweden. As the application grew, with more domains, more DAGs, and more concurrent users, the team began evaluating ways to reduce steady-state query latency and lower compute cost without rearchitecting the system. When AWS made Graviton-powered RG nodes available for Amazon Redshift, the timing was right.

Benchmark performance results

The benchmark infrastructure was fully defined as infrastructure as code (IaC), making sure every test run was reproducible. The team deployed two test benchmark clusters (one RA3 and one RG) in a like-for-like configuration. They mirrored the settings (Amazon Virtual Private Cloud (Amazon VPC), security groups, AWS Key Management Service (AWS KMS), AWS Identity and Access Management (IAM) roles, and parameter groups) from the production environment to remove configuration drift. The benchmark runner was containerized as an Amazon ECS task (python:3.11-slim-bookworm ARM64 base), providing repeatable, isolated execution for each test round. Benchmark workloads were selected by analyzing production cluster logs and metrics, then classified into three tiers:

  • Heavy: ETL queries with multi-table CTE chains, full-table scans, and aggregation windows.
  • Medium: Business intelligence (BI) queries driving reporting and analytics dashboards.
  • Light: Application queries with sub-second response times.

Architecture

Scenarios tested

To validate the performance of Graviton-powered RG instances against the existing RA3 nodes, tombola designed four benchmark scenarios that progressively increase in complexity and realism. Together, these scenarios provide a comprehensive view of performance from isolated query execution through to sustained, real-world analytical workloads.

Scenario 01: Cold-cache, single-stream execution. This scenario isolates raw compute performance by running queries against a cold cache in a single stream, avoiding caching and concurrency as variables.

Per-query speedups ranged from 1.05× (light lookup queries) to 1.68× (heavy ETL transforms). Zero errors on both clusters (28 attempts each).

Weight Class RA3 p50 (ms) RG p50 (ms) Speedup
Heavy (ETL) 210,372 133,855 1.57×
Medium (BI) 2,193 1,642 1.34×
Light (App) 3.20 2.76 1.16×

The following chart shows per-query speedup ratios for the cold-cache scenario. Heavy ETL queries (left) show the largest gains, with speedups of 1.57–1.68×, and lighter queries still benefit at 1.05–1.16×. The pattern is consistent: RG’s advantage scales with query complexity.

Scenario 02: Warm-cache, single-stream execution. This scenario repeats Scenario 01 with the result cache enabled to confirm that RG maintains its latency advantage even when cached results are in play.

Per-query speedups ranged from 1.04× to 1.64×. Zero errors on both clusters (35 attempts each).

Weight Class RA3 p50 (ms) RG p50 (ms) Speedup
Heavy (ETL) 93,636 61,691 1.52×
Medium (BI) 2,189 1,584 1.38×
Light (App) 3.08 2.58 1.19×

With result caching enabled, the speedup pattern holds for non-cached queries. Cache hits on both clusters land in 118–185 ms, confirming the caching subsystem operates identically regardless of node type. The RG advantage appears exclusively on execution paths that bypass the cache.

Scenario 03: Concurrency sweep. This scenario introduces parallel load by sweeping through 1, 5, 10, and 20 concurrent streams, testing how each node type handles contention and queuing under pressure.

Both clusters used the same Concurrency Scaling configuration (max_concurrency_scaling_clusters=1, WLM-only). RG completed 482 more queries in the same wall-clock window.

Metric RA3 RG Improvement
Total queries completed 1,438 1,920 +33% throughput
Light p50 (ms) 3.44 3.04 1.13×
Medium p50 (ms) 20,784 15,055 1.38×
Errors 0 0 —

Under increasing parallel load (1, 5, 10, and 20 concurrent streams), RG maintained lower latencies and completed 33 percent more queries in the same wall-clock window. Both clusters used the same Concurrency Scaling configuration, so the throughput difference is attributable to per-node compute efficiency.

Scenario 04: Mixed realistic workload. This scenario combines the previous elements into a mixed realistic workload, running 10 streams simultaneously for 30 minutes with a weighted distribution of heavy, medium, and light queries to simulate actual production conditions.

This scenario best simulates production. The headline finding: heavy ETL queries saw speedups of up to 2.27× under concurrent load, and RG completed 46 percent more total queries in the same 30-minute window. Zero errors on both clusters.

Metric RA3 RG Improvement
Total queries completed 405 593 +46% throughput
Heavy p50 (ms) 1,186,572 642,294 1.85×
Medium p50 (ms) 2,319 1,631 1.42×
Light p50 (ms) 3.12 2.90 1.08×
Errors 0 0 —

The mixed-realistic scenario best simulates production. Under 10 concurrent streams over 30 minutes, heavy ETL queries showed speedups of up to 2.27×. RG’s per-vCPU throughput advantage compounds under contention, exactly the condition where production clusters spend most of their time.

Extended benchmark: Amazon S3 Tables (Iceberg) performance

tombola’s future data architecture will integrate with agents and revolves around Apache Iceberg, backed by Amazon S3 Tables. Amazon S3 Tables offer Amazon S3 storage that is specifically tuned for analytics, with built-in capabilities that keep making queries faster and helping lower storage costs for table data. They’re purpose-built to hold tabular datasets, such as daily purchase logs, streaming sensor readings, or ad impression events. In this model, data is organized into rows and columns, similar to how information is structured in a traditional database table. With that direction in mind, tombola also benchmarked Graviton’s performance querying Iceberg tables directly. The dataset includes player profiles, game session history, and geolocation data: a mix of wide tables and high-cardinality columns that stress both compute and I/O.

To evaluate performance across different scenarios, tombola generated queries at varying levels of complexity. Medium queries involve standard analytical functions like ranking and aggregation, and Medium-High queries introduce multi-step transformations with joins and cumulative calculations. At the High tier, queries combine distinct counting, conditional pivoting, and time-window aggregations. Very High queries are the most demanding: self-joins across the full dataset, multi-signal scoring logic, and advanced statistical functions. This tiered approach captures how each node type performs as computational demands increase.

As with the previous benchmarks, the team kept the test as comparable as possible: a true like-for-like evaluation between RG (powered by Graviton) and RA3 nodes of equivalent size.

Testing was split into two phases:

Phase 1: Concurrency. All queries were submitted simultaneously to measure how well each node type handles concurrent workloads. The goal was to understand throughput differences: how much more work RG nodes can push through under pressure compared to similarly sized RA3 nodes.

All queries were run simultaneously across multiple rounds:

Grouped bar chart showing total execution time across 3 rounds for RA3 vs Graviton

Phase 2: Sequential execution. Each query was run in isolation with full compute resources available. This removed concurrency as a variable and gave a clean read on raw query performance. The results were clear: RG outperformed RA3 across multiple query types, showing consistent gains when given dedicated compute.

In sequential execution, Graviton (RG) delivered consistent performance gains across all query complexity levels: Medium-complexity queries ran 45–73 percent faster (average 58 percent), Medium-High queries improved by 42 percent, High-complexity queries achieved 57–66 percent faster execution (average 62 percent), and Very High-complexity queries saw gains of 60–67 percent (average 63 percent). The results demonstrate that RG’s advantage scales with workload complexity, delivering the largest improvements on the most demanding analytical queries.

tombola’s modernization approach

tombola is modernizing its Amazon Redshift cluster using the Elastic Resize path to change from RA3 to RG node types. The operation snapshots the existing cluster, provisions a new RG cluster from that snapshot, and transfers data in the background. During this transfer period, the source cluster remains available in read-only mode. When the resize nears completion, Amazon Redshift automatically updates the endpoint to point to the new RG cluster and drops connections to the source. The team chose this approach because it aligns with their engineering principle of evidence-based changes: no production cutover without proof. The benchmark results, with zero errors across all scenarios against production-representative workloads, provided the confidence needed to proceed. After the resize is complete, the external tables, schemas, and query syntax remain unchanged. With RG’s integrated data lake query engine, tombola also removes its dependency on Amazon Redshift Spectrum. Data lake queries now run directly on cluster nodes within the Amazon VPC boundary, using existing IAM roles, with zero per-TB scanning charges.

Conclusion

The benchmark results make a compelling case for migrating tombola’s Amazon Redshift infrastructure from RA3 (Intel Xeon) to RG (Graviton4) instances. Across every scenario tested, RG delivered significant and consistent performance gains:

  • Cold-cache performance: 1.57× faster on heavy ETL queries, with per-query speedups up to 1.68×.
  • Warm-cache performance: 1.52× faster on heavy workloads, maintaining advantage even with result caching enabled.
  • Concurrency: 33 percent higher throughput under parallel load, with RG sustaining lower latencies as streams increased from 1 to 20.
  • Mixed realistic workload: 1.85× faster on heavy ETL queries and 46 percent more total queries completed, the scenario closest to production traffic patterns.
  • Amazon S3 Tables (Iceberg): Up to 51 percent faster under concurrent load and 57 percent faster in sequential execution, critical for tombola’s future lakehouse architecture.

Beyond raw performance, RG delivers architectural benefits that align with tombola’s strategic direction. The integrated data lake query engine removes Amazon Redshift Spectrum overhead and per-TB scan charges. The 4:3 node mapping (4 ra3.4xlarge nodes to 3 rg.4xlarge nodes) reduces infrastructure costs by 25 percent.

Based on these results, tombola are modernizing their production Amazon Redshift cluster to Graviton4-based RG instances. The work has already started and similar results as above are noticed.  The existing RA3 features, including concurrency scaling, data sharing, and system views, are fully supported on RG. This positions tombola to handle growing data volumes and user concurrency with better performance, greater cost efficiency, and a predictable pricing model as the application scales.

The results and benefits described in this post are specific to tombola’s workload and environment. Although Amazon Redshift RG instances powered by AWS Graviton4 processors can deliver significant performance improvements, actual results will vary based on factors including workload characteristics, data volumes, cluster configuration, and query complexity. We encourage you to evaluate RG instances with your own workloads to determine the benefits for your environment. To learn more, visit the Amazon Redshift marketing page and the Amazon Redshift documentation, or get started in the Amazon Redshift console.


About the authors

Prabhu Pandian

Prabhu Pandian

Prabhu has over 15 years of experience spanning data engineering, business intelligence, and data analytics. He has built a career on turning complex data challenges into actionable insights across industries including retail, healthcare, logistics, iGaming, and the public sector. He has led high-performing teams at organisations architecting data warehouses, building ETL pipelines processing tens of millions of records daily, and delivering analytics. Currently, as the Data Engineering Lead at tombola, he is focused on harnessing the power of AWS services to build scalable, optimised data platforms that drive real business value. He is passionate about engineering data infrastructure that is not just robust and efficient, but one that empowers teams to make faster, smarter decisions.

Akshay Srinivasan

Akshay Srinivasan

Akshay is a Data Engineer at tombola, where he runs the Data Platform & Reliability pod, shaping the architecture, scalability, and resilience of the company’s core data infrastructure across batch, streaming, and machine learning workloads. He favors open source tooling and composable AWS services, building platforms designed to be flexible and operationally sustainable. Over the past eight years he has built data platforms from the ground up across fintech, gaming, and enterprise environments, standing up greenfield infrastructure, automating complex operational workflows, and engineering systems in domains where data reliability directly affects regulatory and business outcomes. Having worked with Amazon Redshift since 2017, he has seen its evolution first-hand, from early node types through to the modern lakehouse capabilities the platform offers today.

Sidhanth Muralidhar

Sidhanth Muralidhar

Sidhanth is a Principal Technical Account Manager at AWS, where he partners with enterprise customers to design, scale, and optimize cloud-focused systems. He specializes in guiding organizations through complex architectural decisions across cost efficiency, reliability, performance, and operational excellence. His work increasingly sits at the intersection of data systems and AI as well, helping customers operationalize modern data architectures and build intelligent, production-ready systems.

Vlad Siniavin

Vlad Siniavin

Vlad is a Sr. Technical Account Manager at AWS with over 15 years of experience in building innovative solutions, products and services. He is driven by delivering measurable outcomes for his customers – whether that’s reducing operational risk, optimising costs, or accelerating cloud adoption. He believes the best technical guidance starts with deeply understanding what matters most to the customer and acting in their best interest.

Modernizing financial analytics with Amazon SageMaker Unified Studio

Post Syndicated from Umang Aggarwal original https://aws.amazon.com/blogs/architecture/modernizing-financial-analytics-with-amazon-sagemaker-unified-studio/

Avanse Financial Services is one of India’s leading education loan providers. Their Data Engineering Team had built a data lake on AWS using Amazon Simple Storage Service (Amazon S3), Amazon Athena, and AWS Glue for data ingestion and processing. However, their analytics and reporting layer ran on an external analytics application that wasn’t integrated with AWS. Data had to be copied from Amazon S3 into this external application before analysts could run any report, its license consumed a significant portion of their budget despite low utilization, and every integration with AWS services required custom-built pipelines.

After evaluating their options, Avanse migrated to a cloud-native lakehouse architecture using Amazon SageMaker Unified Studio, which unified their data engineering, analytics, and artificial intelligence (AI) workflows in a single governed environment on AWS. In this post, we walk through their migration journey so you can adapt their approach to your own environment.

Why Avanse chose to modernize

The separation between their AWS data lake and their external analytics application created five problems:

  1. Daily data synchronization bottleneck. Every report required a 4-hour batch copy from Amazon S3 into the external analytics application before analysts could query it. Business decisions were based on data that was at least a day old.
  2. Fixed licensing costs disconnected from usage. The external analytics application charged an annual fee regardless of how many queries analysts ran. Avanse needed usage-based pricing that matched what they actually consumed, not a fixed fee for capacity they weren’t using.
  3. Limited auditability. The external analytics application ran on a shared server where different business units (risk, collections, portfolio management) shared the same resources. It lacked granular audit trails, making it difficult to trace who accessed what data and when, or to allocate costs per team.
  4. No centralized data discovery. Although AWS Glue Data Catalog managed schema metadata for the data lake, the external analytics application couldn’t access it. Analysts working in that application relied on folder structures and manual documentation to find the right datasets, slowing onboarding and increasing the risk of using outdated data.
  5. Disconnected from AWS services. The external analytics application couldn’t query data in Amazon S3 or use AWS Glue catalogs natively. Every data flow required connectors and custom-built pipelines, adding maintenance overhead.

Additionally, some datasets were stored on Network File System (NFS) storage outside of Amazon S3, creating another data silo that needed to be consolidated.

Avanse chose Amazon SageMaker Unified Studio because it addressed all five challenges: direct querying of data in Amazon S3 avoiding synchronization, usage-based compute through Amazon Athena and Amazon EMR Serverless, project-based isolation with per-project billing, lineage tracking with AWS IAM Identity Center, and native integration with their existing AWS services.

Solution overview

The core architectural change was moving from a two-application model to a single integrated stack:

Previous architecture
Avanse’s data ingestion and processing ran on AWS (Amazon S3, AWS Glue, Athena), but analytics and reporting ran on an external analytics application. Data had to be batch-copied from Amazon S3 into this external application daily before analysts could query it. Each system had its own access controls, and there was no shared catalog or lineage tracking between them.
New architecture
Analytics now run directly against data in Amazon S3 through Amazon SageMaker Unified Studio. There’s no data copy step. Analysts query the same data that the ingestion pipelines produce, using Athena for SQL and EMR Serverless for large-scale processing. Governance, access control, and lineage are centralized through IAM Identity Center and SageMaker Catalog.

The following diagram illustrates the target architecture. It follows a lakehouse pattern, storing data in open formats on Amazon S3 while maintaining ACID transaction support for the consistency financial regulators expect.

Three-layer lakehouse architecture for Avanse on AWS, showing the data layer with Amazon S3 and AWS Glue Data Catalog, the compute layer with Amazon SageMaker Unified Studio, AWS Glue ETL, AWS Lambda, Amazon EMR Serverless, Amazon SageMaker AI, and Amazon Bedrock, and the governance layer with AWS IAM Identity Center, SageMaker Catalog, and Amazon DataZone

The architecture has three layers:

  1. Data layer – Amazon S3 stores data in open formats (Parquet, Delta Lake) with S3 Intelligent-Tiering for automatic cost optimization. AWS Glue Data Catalog maintains schema metadata, making data discoverable across tools.
  2. Compute layer – Amazon SageMaker Unified Studio provides project-based workspaces organized by business function. Collections uses the built-in SQL Query Editor powered by Athena, Risk Reporting uses JupyterLab for interactive analysis, and MIS runs large-scale Spark jobs through Amazon EMR Serverless. AWS Glue ETL handles data transformations and AWS Lambda provides event-driven triggers for report generation. For machine learning (ML) workloads, Amazon SageMaker AI supports model training and deployment, with Amazon Bedrock available for generative AI capabilities such as enhancing risk narratives.
  3. Governance layer – IAM Identity Center provides SSO and audit logging across workspaces. SageMaker Catalog serves as the business glossary with data lineage tracking and access controls. Amazon DataZone connects components through a common metadata layer.

Migration journey

Avanse followed a five-phase approach. The timelines can be adapted to your environment, but the systematic progression from validation through production deployment is key.

Phase 1: Technical validation (72-hour workshop)

Avanse started with a focused 72-hour workshop using isolated SageMaker environments where developers could experiment without impacting production. Their team tested SQL analytics against existing Athena tables and validated that Python and PySpark could replicate their existing analytics workflows.

The team confirmed that querying data directly in Amazon S3 addressed their synchronization bottleneck entirely. The 4-hour daily data copy was no longer necessary, which validated the migration approach.

Phase 2: Data migration and storage optimization

Avanse migrated datasets from NFS storage and legacy analytics formats into Amazon S3, consolidating the data into a single location. They implemented S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns, optimizing costs without impacting retrieval performance.

They replaced legacy analytics connectors with native Athena workgroups within SageMaker Unified Studio, avoiding data synchronization entirely. Source data remained in Amazon S3, queryable by both Athena SQL and SageMaker notebooks, establishing a single source of truth.

Phase 3: Compute modernization

Avanse moved from a shared analytics server to project-based isolation in SageMaker Unified Studio. Each business function (Risk Reporting, Collections, MIS) received its own project with dedicated compute spaces running JupyterLab. Project-specific IAM execution roles provided access controls and cost allocation per business unit.

A single browser-based URL with multi-factor authentication (MFA) now provides access to SQL analytics using the built-in query editor, ML development in JupyterLab notebooks, and big data processing through Amazon EMR Serverless. This replaced the need for local analytics client installations.

Phase 4: Governance implementation

Avanse deployed SageMaker Catalog as their central business data catalog. Analysts now discover approved datasets through semantic search rather than navigating folder structures or relying on manual documentation. They mapped technical Athena table names to business terms. For example, analysts search for “collection efficiency” and find the relevant tables with descriptions, schemas, and lineage.

Lineage capture traces each metric in risk reports back to source tables, transformations, and intermediate datasets. Every action (notebook execution, SQL query, data access) is tied to IAM Identity Center users, creating the comprehensive audit trail their compliance team needed.

Phase 5: Use case migration

Rather than attempting a big-bang migration, Avanse moved critical workflows one at a time:

Portfolio MIS (Monthly/Fortnightly)
Previously required the daily 4-hour data copy from Amazon S3 into the external analytics application before report generation could begin. Avanse avoided the data synchronization step entirely and now generates MIS reports by querying existing Athena tables directly in Amazon S3. Because the source data was already on AWS, there was no need to involve the external application for this activity. Report generation dropped from hours to under 30 minutes.
Collection Efficiency and Bounce Calculation
Ported complex legacy analytics procedures for calculating metrics like collection efficiency and bounce rates to event-driven processing using AWS Glue ETL, AWS Lambda, and PySpark jobs for high-volume data aggregation. The serverless execution model charges only for compute time consumed.
EDW Risk Reporting
Large-scale regulatory joins of Enterprise Data Warehouse assets previously ran as legacy scheduled procedures. These now run as SQL queries in the SageMaker Unified Studio query editor, where analysts execute them on-demand or schedule them through Athena workgroups. The distributed query engine handles complex multi-table joins spanning millions of rows.
Scorecard Generation
Model building shifted from the external analytics application to SageMaker AI workflows. Data scientists use JupyterLab with Python libraries and deploy models directly to SageMaker endpoints, avoiding data movement between separate environments.

Overcoming technical challenges

One technical challenge was code migration. Avanse’s analytics code base contained years of accumulated proprietary scripts and procedures. Direct line-by-line translation was not practical. Instead, they took a pragmatic approach: basic data transformations moved to SQL in Athena, complex business logic was rewritten in PySpark for scalability, and statistical procedures were replaced with Python libraries like pandas and scikit-learn. The approach was to focus on what the code accomplishes, then implement it using cloud-native patterns.

The other technical challenge was performance validation. The team needed to confirm that querying data in Amazon S3 would deliver acceptable performance compared to the external analytics application’s in-memory processing. Queries against Parquet-formatted data in Amazon S3 using Athena delivered comparable performance for standard reporting workloads, while avoiding the 4-hour daily data synchronization step entirely. For large-scale regulatory joins spanning millions of rows, Amazon EMR Serverless provided distributed Spark processing that completed in minutes rather than the hours required in the external application.

Key outcomes

Area Result
Licensing costs Avoided external analytics application fees entirely
Storage costs Reduced through S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns
Report generation From over 4 hours (including data synchronization from Amazon S3 to the external analytics application) to under 30 minutes with direct Amazon S3 querying
Compliance audits From weeks of manual investigation to days with automated lineage reports
Compute costs Usage-based serverless model replaced always-on external analytics infrastructure
Collaboration Unified browser-based environment for data scientists, analysts, and engineers

“By adopting SageMaker Unified Studio, we as the Data Team eliminated legacy licensing costs, reduced storage and compute expenses with a serverless, usage-based model, and accelerated our periodic report generation. At the same time, we transformed compliance and collaboration by cutting audit timelines while unifying our teams in a single, efficient data environment.” – Komal Thakkar, AVP – Lead, Data Engineering, Avanse Financial Services

Best practices

Based on their experience, Avanse recommends:

  • Start with a workshop. Validate your specific use cases in a 72-hour technical validation before committing to full migration.
  • Migrate use cases, not code. Focus on what your analytics accomplish, then implement using cloud-native patterns rather than translating legacy scripts line by line.
  • Invest in governance early. Implement the data catalog and lineage tracking from day one.
  • Embrace project-based isolation. Organize around business functions for clear cost allocation and security boundaries.
  • Document business logic. Use migration as an opportunity to capture undocumented knowledge in the business glossary and dataset descriptions.

Conclusion

Avanse’s migration from an external analytics application to Amazon SageMaker Unified Studio consolidated their analytics stack into a single integrated environment on AWS. By querying data directly in Amazon S3 instead of copying it into the external application, they alleviated their biggest operational bottleneck. Project-based isolation replaced a shared server model, giving each business unit independent compute and clear cost visibility. And centralized governance through SageMaker Catalog and IAM Identity Center gave their compliance team the audit trails they had been missing.

The serverless, usage-based model means Avanse no longer pays for idle capacity. The lakehouse architecture supports new analytics patterns as they emerge, and native integration with AWS services, including generative AI through Amazon Bedrock, positions them to adopt new capabilities as their needs evolve.

Next steps

Start your analytics modernization journey by scheduling a 72-hour technical validation workshop. Contact your AWS account team to discuss your migration approach.

For more information, see:

AWS Weekly Roundup: NY Summit recap, Local Zone in Hanoi, Grok 4.3 in Bedrock, price reductions, and more (June 22, 2026)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-ny-summit-recap-local-zone-in-hanoi-grok-4-3-in-bedrock-price-reductions-and-more-june-22-2026/

Last week AWS Summit New York City brought together thousands of customers, partners, and builders for a free, one-day event showcasing the latest in cloud and AI innovation. Dr. Swami Sivasubramanian, VP of Agentic AI at AWS unveiled a stack of AI launches in his keynote, all built around one thesis: agents that compound value over time.

  • Agents for working – You can launch autonomous agents and access a smarter activity feed with new Amazon Quick features, which now let you create and run multi-step agents directly in the desktop app and consolidates email, Slack, calendar, and tasks into a single prioritized view with personalized rules.
  • Agents for securing – You can shift from reactive to proactive security with AWS Continuum, a new AI-native security service that reasons, validates, and acts at machine speed across the full code vulnerability lifecycle. AWS Security Agent (now part of AWS Continuum) adds new features: threat modeling; pull request code scanning with remediation across major Git platforms; and IDE integrations via Kiro power, Claude Code plugin, and MCP.
  • Agents for building – You can write, ship, and modernize code in one continuous loop with Kiro, AWS DevOps Agent, and AWS Transform. Kiro introduces a native iOS app; AWS DevOps Agent adds release management capabilities to assess code changes before production; and AWS Transform continuous modernization reduces tech debt autonomously.
  • Agents customers create – You can go from agent idea to production in minutes with Amazon Bedrock AgentCore, which now includes a GA harness for infrastructure and orchestration, Web Search, Managed Knowledge Base, policy integrations with Guardrails, and the new AWS Context service for mapping organizational data relationships.

To learn more, visit the Summit recap from our top announcements blog post and Amazon News post.

Last week’s launches
Here are last week’s launches that caught my attention:

  • AWS Local Zone in Hanoi, Vietnam  —This new Local Zone is one of the first AWS Local Zones in the Asia Pacific with support for Amazon S3 and Amazon EBS Local Snapshots, enabling customers to meet data residency requirements by storing and backing up data locally. To get started, enable the Hanoi Local Zone (ap-southeast-1-han-1a) from the Regions and Zones tab in the AWS Global View or by using the ModifyAvailabilityZoneGroup API.
  • AWS Blocks, an open-source TypeScript framework for application developers (preview) — AWS Blocks runs a fully functional local environment with Postgres, authentication, and real-time messaging, no AWS account required. When you’re ready to deploy, the same application code runs on production AWS services with zero changes, and you can drop into AWS CDK at any point for direct resource configuration.
  • Grok 4.3 from xAI in Amazon Bedrock —You can use the Grok 4.3 model on Amazon Bedrock, giving you even more choice as you build generative AI applications across reasoning, agentic, and enterprise workflows. Grok 4.3 runs on a new inference engine in Bedrock designed for price performance, with support for tool calling, structured output, and response streaming.
  • Amazon S3 annotations: attach rich, queryable context directly to your objects — Amazon S3 now lets you attach up to 1 GB of rich, mutable, and queryable context directly to your objects using annotations, purpose-built for AI agents and autonomous workflows that need to discover, understand, and act on data at scale without maintaining separate metadata systems.
  • Amazon ECS announces faster service auto scaling — Amazon ECS service auto scaling now detects and responds to load changes faster with support for high resolution (20-second) metrics and metric publishing optimizations. In AWS benchmarking tests, time to trigger scale-out improved from 363 seconds to 86 seconds (76% faster), and total time to scale and provision new tasks improved from 386 seconds to 109 seconds (72% faster).
  • Amazon EC2 G7 instances accelerated by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs — AWS is the first major cloud provider to support NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. G7 instances are accelerated by these GPUs with custom sixth-generation Intel Xeon Scalable processors, delivering up to 4.6x AI inference performance and up to 2.1x graphics performance compared to G6 instances.
  • Strands Agents introduces new capabilities — Strands is an open source toolkit for building production agents. You can now use better context management in Harness SDK, a new isolated execution environment with Strands Shell, and chaos testing and red teaming in Strands Evals.
  • AWS Management Console Private Access – You can access the AWS Console from VPCs without internet connectivity, allowing enterprises to manage their AWS infrastructure through the console while maintaining strict network security controls in air-gapped environments.
  • AWS Marketplace Storefront is now generally available – AWS Partners can create and deploy their own branded catalog of solutions and services on their website or application in hours. Channel Partners and Independent Software Vendors can now simplify how they manage their cloud marketplace business and make it easier for customers to discover and purchase their solutions from AWS Marketplace.
  • Palo Alto Networks (PANW) Advanced DNS Security on Amazon Route 53 Resolver DNS Firewall (preview) – You can now enforce DNS threat protections from Palo Alto Networks directly on Route 53 DNS Firewall rules, without deploying separate firewalls or modifying VPC configurations — by subscribing to PANW from the DNS Firewall console through the embedded AWS Marketplace widget.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Price reductions 
AWS continues to look for ways to increase performance and lower prices for our customers. I noticed a few such efforts last week, so I’d like to share them:

Learn more about AWS, browse and join upcoming AWS-led in-person and virtual events, startup events, and developer-focused events as well as AWS Summits and AWS Community Days. Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Channy

Top announcements of the AWS Summit in New York, 2026

Post Syndicated from AWS News Blog Team original https://aws.amazon.com/blogs/aws/top-announcements-of-the-aws-summit-in-new-york-2026/

Today at the AWS Summit in New York City, Swami Sivasubramanian, AWS VP of Agentic AI, provided the day’s keynote. Here’s our roundup of the biggest announcements from the event:

New in Amazon Bedrock AgentCore
We’re introducing new capabilities on Amazon Bedrock AgentCore: connecting AI agents to organizational, web, and paid knowledge, helping teams find and fix what’s going wrong in production, and enforcing controls that scale as agents grow more capable.

Together, these capabilities help you build more capable agents faster, govern those agents with controls that scale, and improve them continuously. To learn more, read our blog post covering all the new features.

New in AI-based security tools

New in building AI-based applications 

  • Introducing Kiro for iOS — Kiro introduces a native iOS app, available in a gated preview, built for real engineering work that gives developers a new surface to kick off, monitor, steer, and interact with their Kiro sessions directly from their phone. That means you can now start sessions, check back when they’re done, review diffs, and approve changes all while staying connected to your work with no laptop running.
  • AWS DevOps Agent adds release management capabilities to assess code changes before production — You can use a new release readiness review of code changes and autonomous release testing. These new features verify every change against the natural language standards you give to the DevOps Agent and run change-specific tests in production-like environments.
  • Proactively reduce tech debt autonomously with AWS Transform – continuous modernization — You can use continuous analysis (preview) to automatically scan your code repositories against configurable baselines and generates findings in hours, not weeks. Once you’ve identified and prioritized findings, you can configure autonomous remediations that generate pull requests for affected repositories automatically.

In addition to the keynote announcements, we have other important launches this week:

Amazon S3 annotations: attach rich, queryable context directly to your objects

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/amazon-s3-annotations-attach-rich-queryable-context-directly-to-your-objects/

Today, we’re announcing a new metadata capability for Amazon Simple Storage Service (Amazon S3) called annotations, enabling you to attach rich, large-scale business context directly to your objects. You can store up to 1,000 named annotations per object, each up to 1 MB in size, totaling up to 1 GB per object, in flexible formats like JSON, XML, YAML, or plain text. You can modify or delete an annotation at any time, without re-writing your objects, making it easy to keep your object context current.

Organizations are building AI agents and autonomous workflows that need to find, understand, and act on data without human intervention. To support these agentic workflows, you need metadata that can evolve alongside the data, scale to petabytes of objects, and remain queryable without expensive retrieval.

With S3 annotations, you can store context such as AI-generated transcripts, content ratings, or technical specifications directly alongside your objects. Your context moves automatically with the object during copy, replication, and cross-region transfers, and S3 removes it when you delete the object. When you enable S3 Metadata, annotations automatically flow into fully managed annotation tables that you can query with Amazon Athena and other analytics engines.

Common use cases
Annotations solve complex metadata challenges across industries:

  • Media & Entertainment: Track transcripts, content moderation results, subtitle files, and licensing metadata as separate annotations on video assets, eliminating the need to synchronize metadata across multiple media asset management systems.
  • Financial Services: Attach AI-generated investment summaries and sentiment analysis to research documents, enabling autonomous research agents to discover relevant datasets through natural-language queries without maintaining separate metadata databases.
  • Life Sciences: Annotate clinical trial data with regulatory status, patient cohort details, and approval chains, making compliance audits faster while keeping full context accessible for archived data in Amazon S3 Glacier storage classes without retrieval charges.

How annotations address metadata challenges
Amazon S3 already supports several ways to describe your objects. System-defined metadata captures properties like size and storage class. Object tags support operational tasks like access control and lifecycle management. User-defined metadata lets you add small amounts of custom information at upload time.

While these capabilities work well for their intended purposes, they have limitations when you need to attach much richer context without building and maintaining separate metadata systems. Annotations address these needs by providing metadata capabilities at a fundamentally different scale and flexibility, offering mutable, queryable context per object compared to 10 immutable tags or 2 KB of headers.

Capability Max size Mutable? Best for
System-defined metadata Fixed No Object properties (size, storage class, creation time)
User-defined metadata 2 KB No (set at upload) Small custom key-value pairs
Object tags 10 tags, 128/256 characters per key/value Yes Access control, lifecycle rules, cost allocation
Annotations 1 GB (1,000 × 1 MB) Yes Rich business context (JSON, XML, YAML, plain text)

Today, metadata describing S3 objects often lives in separate databases or sidecar files, requiring complex synchronization workflows that can exceed data storage costs. When you enable S3 Metadata annotation tables, this context becomes queryable at scale through Amazon Athena. AI agents can discover your data through natural language with the S3 Tables MCP server, which provides a standardized interface for AI models to query your annotations. You can query annotations for objects in any storage class, without restoring the objects or paying retrieval charges.

Getting started with annotations
To start using annotations, make sure your AWS Identity and Access Management (IAM) policy or bucket policy grants permissions for the s3:PutObjectAnnotation and s3:GetObjectAnnotation actions. You can then add annotations to any existing or new S3 object using the PutObjectAnnotation API.

For example, a media company can attach technical specifications and AI-produced summaries to a video asset using the AWS Command Line Interface (AWS CLI):

# Create a JSON file with technical metadata
cat > mediainfo.json << 'EOF'
{"codec":"H.265","resolution":"3840x2160","audio_tracks":8,"frame_rate":29.97}
EOF

# Attach it as an annotation
aws s3api put-object-annotation \
  --bucket my-media-bucket \
  --key videos/documentary-2026.mp4 \
  --annotation-name mediainfo \
  --annotation-payload ./mediainfo.json
# Attach a plain-text AI-generated summary as a separate annotation
echo "A 90-minute nature documentary covering wildlife migration patterns across three continents, featuring aerial footage and underwater sequences. Languages: English, Spanish, Portuguese." > ai_summary.txt

aws s3api put-object-annotation \
  --bucket my-media-bucket \
  --key videos/documentary-2026.mp4 \
  --annotation-name ai_summary \
  --annotation-payload ./ai_summary.txt

These commands attach two separate annotations to the same video object. The mediainfo annotation stores structured technical specifications as JSON, while the ai_summary annotation stores a text description. Each annotation is identified by a unique name, and you can read and modify each one independently. With unique names for each annotation, you can use different annotations to support multiple concurrent enrichment workflows, for example, one team adding technical metadata while another team adds content classifications, without interfering with each other.

Retrieve a specific annotation using the GetObjectAnnotation API:

aws s3api get-object-annotation \
  --bucket my-media-bucket \
  --key videos/documentary-2026.mp4 \
  --annotation-name mediainfo \
  ./mediainfo-output.json

To see all annotations attached to an object, use the ListObjectAnnotations API:

aws s3api list-object-annotations \
  --bucket my-media-bucket \
  --key videos/documentary-2026.mp4

When you no longer need a specific annotation, remove it using the DeleteObjectAnnotation API:

aws s3api delete-object-annotation \
  --bucket my-media-bucket \
  --key videos/documentary-2026.mp4 \
  --annotation-name mediainfo

You can update an existing annotation at any time by calling PutObjectAnnotation again with the same annotation name. For large objects uploaded using multipart upload, attach annotations after completing the multipart upload using the PutObjectAnnotation API.

Querying annotations at scale with S3 Metadata tables
Attaching annotations to individual objects is useful, but the real power comes when you query across all your annotations at scale. When you enable S3 Metadata annotation tables on your bucket, S3 automatically indexes your annotations into a fully managed Apache Iceberg table, called an annotation table. You can query annotation tables with Amazon Athena or any Iceberg-compatible engine.

To enable annotation tables, use the S3 console or the CreateBucketMetadataConfiguration API. The following example creates a new metadata configuration with annotation tables enabled while keeping journal tables for change tracking and disabling the live inventory table:

{
  "JournalTableConfiguration": {
    "RecordExpiration": { "Expiration": "DISABLED" }
  },
  "InventoryTableConfiguration": { "ConfigurationState": "DISABLED" },
  "AnnotationTableConfiguration": {
    "ConfigurationState": "ENABLED",
    "Role": "arn:aws:iam::123456789012:role/S3MetadataAnnotationRole"
  }
}

This configuration tells S3 to automatically capture all your annotations in a queryable table. Once applied, any annotation you attach to objects in this bucket will appear in the table within approximately one hour.

If the bucket already has a metadata configuration, use the UpdateBucketMetadataAnnotationTableConfiguration API:

aws s3api update-bucket-metadata-annotation-table-configuration \
  --bucket my-media-bucket \
  --annotation-table-configuration '{"ConfigurationState":"ENABLED","Role":"arn:aws:iam::123456789012:role/S3MetadataAnnotationRole"}'

Once enabled, your annotations automatically flow into the annotation table. Journal tables update in near real time, while annotation tables refresh within an hour. Unlike traditional metadata tables that require predefined schemas, annotation tables automatically adapt to any JSON, XML, or YAML structure you write. Each annotation becomes a row in the table with its content stored in a text_value column, letting you query across all annotations without schema migrations.

If you enable annotation tables on a bucket that already has annotated objects, S3 automatically backfills existing annotations into the table. The backfill process runs in the background and can take several hours to days depending on the number of objects.

For example, to find all video assets with more than 8 audio tracks across your entire bucket using Amazon Athena:

SELECT DISTINCT bucket, object_key
FROM "s3tablescatalog/aws-s3"."b_my_media_bucket"."annotation"
WHERE name = 'mediainfo'
AND CAST(json_extract_scalar(text_value, '$.audio_tracks') AS INTEGER) > 8

This query scans the annotation table for all annotations named mediainfo, extracts the audio_tracks field from the JSON content, and returns objects where the count exceeds 8.

Or to find all objects that received new annotations in the last 24 hours through the journal table:

SELECT bucket, key, version_id, record_timestamp, annotation.name
FROM "s3tablescatalog/aws-s3"."b_my_media_bucket"."journal"
WHERE record_timestamp >= (current_date - interval '1' day)
AND annotation.name IS NOT NULL
AND record_type IN ('CREATE_ANNOTATION', 'DELETE_ANNOTATION')

This query uses the journal table to track annotation changes in near real time, which is ideal for building event-driven workflows that respond to new or deleted annotations.

You can also use natural language to search objects by their annotations using agents in Amazon SageMaker Unified Studio or any IDE with the S3 Tables MCP server. For example, asking “find all PG-rated movies with Spanish subtitles from 2023” returns results in seconds instead of the hours it would take querying multiple disconnected systems.

Get started today
You can start using Amazon S3 annotations today in all AWS Regions, including the AWS China Regions. Annotation tables are available in all AWS Regions where S3 Metadata is available.

Whether you’re building AI agents that need to discover data autonomously, managing petabytes of media assets with complex metadata, or tracking compliance context for archived datasets, annotations give you the scale and flexibility to attach rich metadata directly to your objects without managing separate systems.

Annotation storage is always billed at S3 Standard rates, even if the parent object is in S3 Glacier or another storage class. For full pricing details, visit the Amazon S3 pricing page.

To learn more and get started, visit the Amazon S3 Metadata overview page and the Amazon S3 documentation. Send feedback to AWS re:Post for S3 or through your usual AWS Support contacts.

Daniel Abib

Access Amazon S3 data files directly using AWS Lake Formation permissions

Post Syndicated from Aarthi Srinivasan original https://aws.amazon.com/blogs/big-data/access-amazon-s3-data-files-directly-using-aws-lake-formation-permissions/

Data scientists and ML engineers often need to access raw data files in Amazon Simple Storage Service (Amazon S3) for machine learning training, data exploration, and generative AI workflows. However, when table-level access is governed by AWS Lake Formation, accessing the underlying S3 files has required maintaining separate permission mechanisms. S3 bucket policies or AWS Identity and Access Management (IAM) role policies create operational overhead and risk of permission drift.

Lake Formation now supports direct access to S3 data file locations for tables whose permissions it manages. Previously, data scientists with Lake Formation permissions on AWS Glue Data Catalog tables could query them using spark.sql(). Now, they can also read and write the underlying S3 data files using spark.read.parquet() or spark.read.csv() from Amazon EMR Spark jobs, Amazon SageMaker Unified Studio notebooks with EMR compute, and custom applications. All access is governed by the same Lake Formation permissions.

This capability is powered by the new GetTemporaryDataLocationCredentials() API, which vends temporary credentials scoped to registered S3 locations when callers have appropriate Lake Formation permissions on the corresponding Data Catalog tables. This eliminates the need to manage separate S3 bucket policies for file-level access while maintaining fine-grained access control in Lake Formation for table-based access. It enables your data scientists to explore S3 datasets securely, accelerate machine learning pipelines, and build generative AI workflows without compromising governance.

In this post, we demonstrate reading from and writing to Lake Formation-managed S3 locations using Apache Spark jobs from EMR. Lake Formation credential vending for S3 location access is available in EMR release label 7.13 and later, Boto3 1.42.29 and later, AWS Java SDK 2.41.32 and later, and AWS Command Line Interface (AWS CLI) version 2.33.1 and later.

Key use cases for Lake Formation permissions to S3 locations

  • Unified permissions for Analytics and Machine Learning pipelines – Data scientists can access both structured tables through SQL queries and underlying data files through programmatic APIs for machine learning and AI workloads. They are empowered to use tools of their choice – for example, use Amazon Athena for SQL analytics with the table names while read and write to the underlying files in their SageMaker notebook or Spark application with spark.read.parquet(“s3://bucket/database_path/table_files/).
  • Enable AI ready data lakes – Machine learning pipelines can read training data directly from governed data lakes. Generative AI applications can access foundation model training datasets, and data exploration workflows to use native file APIs while maintaining centralized governance and compliance.
  • Reduced operational complexity – Operations teams don’t need to maintain separate permission policies – one in Lake Formation for table access and another in S3 bucket policies or AWS Identity and Access Management (IAM) roles for file access. This reduces the risk of permission mismatches and avoids inconsistent access control.
  • Unified audit capability – Auditors do not need to examine multiple log sources, such as S3 Access Logs, AWS CloudTrail events from different services, to understand who accessed what data and when. With this feature, you get a unified CloudTrail audit trail showing both table access through SQL engines and file access through direct APIs, with each access event linked to the Lake Formation permission grant.

What customers are saying

“Through our close collaboration with AWS, Lake Formation’s new S3 location-based permissions have transformed how we manage data governance at Intuit. By unifying two separate access mechanisms for the same data into one unified permission model, we’ve dramatically reduced complexity and streamlined our auditing process. This is exactly the kind of simplification that lets our teams move faster without compromising security, ensuring we maintain the strict compliance and governance standards our regulators expect.”

— Tapan Upadhyay, Group Engineering Manager, Intuit

Lake Formation Credential Vending Plugin for AWS SDK v2 for Java

Lake Formation has made available a specialized library AWS Lake Formation Credential Vending Plugin for AWS SDK V2 for Java. The Java plugin intercepts S3 requests for data, checks Lake Formation permissions for the requested location, and provides temporary scoped credentials to the client if permissions are granted in Lake Formation. If the S3 location access permissions are not managed by Lake Formation, the plugin checks for access in Amazon S3 Access Grants and lastly falls back to IAM permissions. The plugin is supported independently of Spark and comes as an enhancement to EMR Spark Full Table Access (FTA) mode, starting in EMR 7.13 and later. The plugin is integrated at the S3A level. Therefore, any client of S3A can enable it by setting the S3A configurations, in addition to the EMR Lake Formation Full Table Access (FTA) configuration as follows:

fs.s3a.lakeformation.access.grants.enabled = true
fs.s3a.lakeformation.access.grants.fallback.to.iam = true

With the Java plugin, you can enable governance for data lake resources in your custom applications with Lake Formation permissions – managing both fine grained access for users requiring restricted access on Data Catalog tables while providing direct S3 object level access to use-cases that require them.

Note: (1) The principal that will be accessing direct S3 locations of the tables will require full table access. That is, Lake Formation SELECT permission on all columns and rows of the table is required. (2) The Spark cluster needs FTA configuration. (3) Currently, Apache Iceberg table format is not supported with this plugin.

Solution overview

A financial services company runs daily ETL jobs using Spark in EMR. They process raw transaction records in S3 and store the processed records in another S3 location. The transformed Parquet data is registered with Lake Formation and cataloged as a table in Data Catalog. The ETL job will have direct IAM access to the raw data location, while it uses Lake Formation permissions to write to and read from the curated table location. Downstream, a data-analyst role will query the curated table, with restricted column access. The solution is shown in Figure 1.

Figure 1 – Architecture shows EMR Spark writing curated records to the S3 location of a table using Lake Formation permissions while Data-Analyst queries the same table with Lake Formation fine grained access control in Athena.

Architecture diagram showing EMR Spark writing curated records to the S3 location of a table using Lake Formation permissions while Data-Analyst queries the same table with Lake Formation fine-grained access control in Athena

Prerequisites

To get started exploring this feature, we recommend you have the following setup.

Solution walkthrough

First, we will get the setup ready with S3, sample database, table, and data. We will add a raw data set to S3 location, create a table with parquet data in another S3 location that represents the curated dataset for further downstream consumption. We will register the table data location with Lake Formation and grant permissions for the EMR run time role and Data-Analyst role.

Your S3 bucket will have the following structure.

Raw data – s3://<your-bucket-name>/raw/transactions/dt=2024-03-21/

Process data for table – s3://<your-bucket-name>/processed/transactions/

Spark script – s3://<your-bucket-name>/scripts/

Logs for the EMR cluster – s3://<your-bucket-name>/logs/

Step 1 – Create a parquet table in Data Catalog

From the Athena console query editor, create a table in Data Catalog.

-- Create a database
CREATE DATABASE finance_db;

-- Create an external table pointing to the S3 location
CREATE EXTERNAL TABLE IF NOT EXISTS finance_db.transactions_processed (
    transaction_id STRING,
    merchant_name STRING,
    amount DECIMAL(18,2),
    currency STRING,
    account_number STRING,
    card_type STRING,
    status STRING,
    region STRING
)
PARTITIONED BY (transaction_date DATE)
STORED AS PARQUET
LOCATION 's3:///processed/transactions/'
TBLPROPERTIES (
    'parquet.compress'='SNAPPY'
);

Step 2 – Register S3 location and grant table permission to IAM roles in Lake Formation

2.1 Register the table data location s3://<your-bucket-name>/processed/transactions/ with Lake Formation in Lake Formation mode using the custom S3 registration IAM role. For details on how to register locations with Lake Formation, refer Adding an Amazon S3 location to your data lake.

2.2 Grant DESCRIBE permission on the database finance_db and ALL permission on the table transactions_processed to your EMR runtime role.

2.3 Grant Data location permission to EMR runtime role on the curated table’s location. This is to allow writing to that location.

2.4 Grant DESCRIBE permission on the database finance_db and SELECT permission on the table transactions_processed to your Data-Analyst role. Exclude the columns transaction_id and account_number while granting SELECT permissions on the table to the Data-Analyst role.

For details on how to grant Lake Formation permissions, refer Granting database permissions using the named resource method; Granting table permissions using the named resource method and Granting data location permissions.

Step 3 – Run ETL script in EMR

3.1 Download the script bdb-5860-script.py.

3.2 Edit the S3 bucket name placeholder in the script (RAW_PATH and TABLE_PATH) to your resource names and upload to your S3 path s3://<your-bucket-name>/scripts/.

3.3 Make sure your EMR runtime role has access to the script location in its IAM policy permissions.

3.4 Submit and run the script as a step to the EMR cluster, following instructions at Add a Spark step.

What does the script do?

It populates raw records of transaction data into a Spark data frame, writes to the raw data bucket location using IAM permissions on the EMR runtime role. We apply some transformations and write directly to the S3 location of the table that is registered with Lake Formation, from the data frame using Spark’s native Parquet writer.

The following figure shows the stdout of the step.

EMR step stdout showing successful Spark job execution with data written to the Lake Formation-managed S3 location

The Java plugin integrated into EMR 7.13 automatically handles the access for the table’s data location registered with Lake Formation, so you don’t need to manually call the GetTemporaryDataLocationCredentials() API. In this example, the table data location s3://<your-bucket-name>/processed/transactions/ is registered with Lake Formation, for which EMR runtime role is granted ALL permissions. The direct S3 location access support by Lake Formation allows reading and writing to the location directly using Spark data frame.

Step 4 – Run query as Data-Analyst using Athena

Log in as the Data-Analyst role to the Athena console. Run a select query on the table as follows.

SELECT * FROM finance_db.transactions_processed WHERE status = 'DECLINED' AND transaction_date=DATE '2024-03-21';

The Data-Analyst role should see all but two columns of the table.

Athena query results showing the Data-Analyst role can access all columns except transaction_id and account_number

With these steps complete, we’ve read from and written to direct S3 locations using Spark data frames with the syntax s3://bucketname/prefix/, and accessed the same data using database_name.table_name syntax with Lake Formation permissions. This shows fine-grained access at table level and coarse-grained access at the file path level.

Clean up

To avoid incurring costs, clean up the resources you created for this post.

  1. Delete the Data Catalog database and tables. This removes the related Lake Formation permissions too. Remove the S3 bucket registration from Lake Formation.
  2. Delete the data files, logs, and the PySpark script of this post from your S3 bucket.
  3. Terminate the EMR cluster.

Conclusion

In this post, we showed how to use Lake Formation’s direct S3 location access to read and write data files using Spark data frames from Amazon EMR, while maintaining unified governance through Lake Formation permissions. We walked through the GetTemporaryDataLocationCredentials() API and the AWS Lake Formation Credential Vending Plugin for AWS SDK v2 for Java, which is integrated into EMR release labels 7.13 and later.

This capability unifies permission management for both fine-grained table-based access and direct S3 file path access in Lake Formation. Your data scientists can now use spark.read.parquet() and spark.write alongside spark.sql(), governed by the same permissions, audited in the same CloudTrail logs, and managed from a single console.

To get started, launch an EMR 7.13 cluster and start exploring the feature. Here are some additional resources:

Acknowledgements: We would like to thank all the team members who worked to launch this feature successfully – Rajas Bhate, Akhil Yendluri, Kunal Parikh, Sharda Khubchandani, Dhananjay Badaya, Santhosh Padmanabhan, Nitin Agrawal and Sandeep Adwankar.


About the authors

Aarthi Srinivasan

Aarthi Srinivasan

Aarthi is a Senior Big Data Architect at Amazon Web Services (AWS). She works with AWS customers and partners to architect data lake solutions, enhance product features, and establish best practices for data governance.

Archana Inapudi

Archana Inapudi

Archana is a Senior Solutions Architect at Amazon Web Services (AWS). She works with strategic enterprise customers to drive cloud data modernization, architect data lake and analytics solutions, and establish best practices for data governance and security. With over 15 years of experience in cloud, data engineering, and AI/ML, Archana is passionate about using technology to accelerate growth and deliver business outcomes.

Srinivasan Krishnasamy

Srinivasan Krishnasamy

Srinivasan is a Principal Delivery Consultant at AWS with 25+ years of experience architecting data and analytics solutions at scale. He partners with enterprise customers to modernize data platforms, build robust data governance frameworks, and drive measurable business outcomes on AWS, using the full spectrum of data engineering, AI/ML, and generative AI. Outside of work, he enjoys hiking, swimming, and gardening.

Anandkumar Kaliaperumal

Anandkumar Kaliaperumal

Anandkumar is a Senior Delivery Consultant at AWS, bringing over 23 years of deep expertise in data and analytics. A specialist in architecting scalable data analytics, AI/ML, and generative AI solutions, he thrives on tackling complex data challenges spanning data engineering, analytics, machine learning, and generative AI workloads.

Mitali Sheth

Mitali Sheth

Mitali is a Streaming Data Engineer at Amazon Web Services (AWS) Professional Services. She works with strategic software customers to architect real-time analytics solutions, design event-driven architectures, and modernize streaming infrastructure using Amazon MSK, Amazon Managed Flink, AWS Glue, and AWS Lake Formation. She holds an M.S. in Computer Science from the University of Florida.

Automating contract intelligence with Doczy.ai™ on AWS

Post Syndicated from Sanket Nasre original https://aws.amazon.com/blogs/architecture/automating-contract-intelligence-with-doczy-ai-on-aws/

Extracting actionable insights from thousands of contracts and legal documents remains a challenge. For organizations, critical business information is locked in unstructured documents such as contracts, legal agreements, provider arrangements, and vendor invoices. Extracting and operationalizing this information has traditionally been a manual, error-prone, and resource-intensive process. This leads to missed savings opportunities, costly delays, and significant inefficiencies across the enterprise.

AArete, a global management and technology consulting firm specializing in healthcare, recognized this challenge and developed Doczy.ai™, an intelligent contract interpretation solution powered by generative AI on Amazon Web Services (AWS).

In this post, we show you how Doczy.ai™ uses generative AI on AWS to automate contract intelligence at scale, transforming unstructured documents into structured, actionable insights, so organizations can automate critical business processes and unlock the full value of their data.

The challenge: Data trapped in documents

For healthcare organizations, managing and interpreting contracts and documents represents a major operational bottleneck. Manual review processes require deploying teams to extract data from thousands of documents. This is an approach that is neither scalable nor sustainable, highly prone to error, and costly. Organizations relying on institutional knowledge face additional risks: critical information resides with a few key individuals, creating knowledge silos and succession planning challenges. Existing Contract Lifecycle Management (CLM) systems often prove inadequate for capturing the nuanced and complex terms unique to each agreement. These legacy systems can only configure predefined fields, missing the rich detail and contextual information that distinguishes contracts. The downstream impact is substantial: in healthcare, reimbursement terms must be manually translated into claims systems—a slow, error-prone process. Similarly, verifying vendor invoices against contract terms often requires manual effort, leading to payment processing delays and missed contractual savings opportunities. These inefficiencies ultimately leave significant value on the table.

This is where Doczy.ai™ provides significant value.

Doczy.ai™: An intelligent contract interpretation solution

Doczy.ai™ directly addresses these challenges using advanced AI and scalability on AWS. Developed by AArete, Doczy.ai™ pushes the boundaries of document intelligence. The solution automatically interprets complex documents and converts them into a structured, queryable information repository that allows organizations to unlock the full value of their data and drive smarter decisions.The evolution of Doczy.ai™ reflects rapid AI advancement. Prior to 2020, document processing required manual effort, with individuals processing approximately 100 documents per week. Between 2020–2023, the firm implemented rules-based contract processing, achieving approximately 55% accuracy. The breakthrough came in 2024 with an AI-based processing built on AWS achieved 99% accuracy—a dramatic improvement over the 55% accuracy of traditional rules-based systems.

Doczy.ai™ architecture

Doczy.ai™ is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Doczy.ai is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Architecture of Doczy.ai™

External users access the platform through a secure Next.js frontend, with Amazon Cognito managing authentication and authorization behind the scenes. After authentication, users upload documents directly to Amazon Simple Storage Service (Amazon S3), where durable, scalable object storage ensures nothing is lost and everything is accessible at scale. From there, the real intelligence begins.

An AWS Lambda function triggers Amazon Textract to extract text and metadata from documents in various formats. What sets Doczy.ai™ apart at this stage is its patented “smart chunking” algorithm, a proprietary approach that goes far beyond pulling words off a page. Rather than treating a document as a flat sequence of text, smart chunking preserves hierarchical structure and one-to-many relationships within documents. It uses a combination of semantic and keyword search to decompose text into meaningful, context-aware chunks, applying dynamic parameters to maintain logical relationships throughout. Sequential identifiers and metadata-driven grouping organize these chunks into field groups, detecting overlaps and removing duplications while keeping the document’s natural flow intact.

After chunking, the document enters the dual clustering engine of Doczy.ai™. This two-lens methodology analyzes every contract simultaneously from both a semantic and a structural perspective. On the semantic side, extracted text is converted into embeddings, numerical representations of meaning, and similar ideas are grouped together even when they’re expressed in different words. On the structural side, pattern-recognition algorithms identify clause types, formatting conventions, table layouts, and hierarchical organization, understanding. For example, that a three-nested-level exhibit carries fundamentally different implications than a straightforward attached schedule.These two analyses don’t operate in isolation. Projection algorithms compare the semantic and structural clusters side by side, synthesizing them into a unified, enriched document model that captures both meaning and context. It’s this convergence that drives the 99% accuracy rate of Doczy.ai™. The system doesn’t just read the words, it understands the contract. Advanced large language models (LLMs) then generate structured output grounded in this dual-clustered intelligence.Before output is finalized, the system determines each document’s file class and generates prompts tailored to the extracted text, cluster classification, and domain context. Through few-shot and multi-shot prompting, the platform continuously edits the prompt on domain-specific examples and based on real outputs, creating a feedback loop that compounds accuracy improvements over time.

The resulting structured data flows into Snowflake, forming a centralized repository that powers intelligent dashboards with actionable insights and visualizations. Throughout the entire pipeline, Amazon CloudWatch monitors performance in real time and proactively surfaces issues before they escalate, while AWS Secrets Manager safeguards sensitive information, ensuring that security is not an afterthought, but a foundational layer woven into every stage of the system.

The transformative impact of Doczy.ai™

The results of this AI-powered approach are transformative and measurable. By automating contract interpretation and document processing, Doczy.ai™ has demonstrated significant impact at scale for multiple organizations across healthcare and financial services. The scale of operations over the last 22 months demonstrates the maturity and production readiness of Doczy.ai™. This solution has processed 2.5 million contract documents (50 million pages) with 137 million API calls to Amazon Bedrock and 442 billion tokens—a level of automation and accuracy previously unattainable through manual or traditional document processing approaches. Over this same period, Doczy.ai™ has helped clients achieve approximately 330 million dollars in cumulative direct and indirect savings.The 99% accuracy rate represents significant improvement over the approximately 55% accuracy of rules-based systems and far exceeds manual processing, which is typically affected by fatigue and human error. The 97% reduction in manual processing time translates directly to cost savings and enables organizations to reallocate human resources to higher-value activities that require judgment and strategic thinking.

A use case in action: Business process automation for health plans

For health plans, Doczy.ai™ provides a powerful solution to automate and improve contract management across the entire lifecycle. It ingests existing contracts in both paper and digital formats, integrates with contract management systems such as Coupa and Icertis, and processes new contracts and amendments as they’re executed. It then creates a centralized metadata repository that feeds directly into downstream systems, enabling end-to-end business process automation.This automation unlocks critical capabilities: Organizations can continuously analyze and improve contract terms, identifying opportunities to improve financial performance and operational efficiency. The architecture feeds accurate, up-to-date contract data directly into claims systems, automating the configuration process that previously required manual translation of reimbursement terms and removing manual data entry, configuration errors, and delays. Additionally, the platform helps maintain claim payment accuracy by assessing payments against contract terms, identifying discrepancies, and flagging potential overpayments or underpayments before they occur.By automating manual processes, health plans can adapt quickly to new contract terms and regulatory requirements. The intelligent dashboards and actionable insights provided by Doczy.ai™ enable decision-makers to understand contract performance, identify trends, and take proactive action to optimize financial outcomes.

Getting started with Doczy.ai™

Organizations interested in using Doczy.ai™ to transform document processing and contract management can engage with AArete to discuss their specific use cases and requirements. AArete offers the platform as a Software as a Service (SaaS) solution, enabling rapid deployment without significant infrastructure investment. AArete’s team of experts will configure this solution for your specific document types, domain terminology, and business processes, supporting maximum value from day one.

Conclusion

The challenge of unlocking data from unstructured documents is a major hurdle for many businesses, particularly in healthcare and financial services where contracts and agreements govern critical operational and financial relationships. By embracing intelligent document intelligence on AWS, organizations can solve this long-standing operational challenge and unlock a new frontier of strategic advantage, turning their data into their most valuable asset.

Built on a sophisticated architecture that orchestrates Amazon Cognito, Amazon S3, AWS Lambda, Amazon Textract, Amazon Elastic Container Service (Amazon ECS), Amazon Bedrock, Amazon CloudWatch, and AWS Secrets Manager, Doczy.ai™ demonstrates how modern cloud services can solve complex document-heavy business problems. Its advanced hybrid smart chunking, dual clustering, and prompt optimization techniques form the core of a patented contract intelligence engine.

Doczy.ai™ delivers tangible impact, processing up to 250,000 contract documents per week with 99% accuracy, reducing manual processing time by 97%, and helping clients unlock roughly 330 million dollars in cumulative savings over 22 months. By embracing this intelligent document processing, organizations can turn contracts into a strategic data asset, improving efficiency, accuracy, and profitability while freeing teams to focus on higher-value work.

To learn more about how AArete and Doczy.ai™ can help your organization transform document processing and unlock the value of your unstructured data, visit the AArete website.


About the authors

Securing client confidentiality at scale: Automated data discovery and governed analytics for legal workloads

Post Syndicated from Rohan Kamat original https://aws.amazon.com/blogs/big-data/securing-client-confidentiality-at-scale-automated-data-discovery-and-governed-analytics-for-legal-workloads/

Automating data security and analytics for legal documents presents a unique challenge when your legal team stores documents with strong access controls, organized by client and matter, encrypted at rest, and governed by well-defined policies. But what happens when you want to run analytics across those repositories? The typical path is extracting content into separate data pipelines or third-party tools, which fragments your governance model and introduces new risks. Law firms and corporate legal departments operate under distinct obligations that make data governance non-negotiable. Attorney-client privilege, work product doctrine, and professional conduct rules impose strict duties around how client information is handled, accessed, and disclosed. Governance failure in this context isn’t just a compliance gap, it can result in privilege waiver, disqualification from representation, or disciplinary action.

Legal professionals use ethical walls, also called information barriers, as structural safeguards that prevent the flow of confidential information between teams within a firm that represent adverse or potentially conflicting interests. Professional conduct rules mandate these barriers, and failure to maintain them can result in firm disqualification, malpractice liability, or regulatory sanctions.

Privilege boundaries are equally critical. Attorney-client privilege and work product protection apply only when you properly control access to the underlying material. If you expose privileged documents or metadata about their contents to unauthorized individuals, you risk losing your privilege protection. When organizations fail to maintain reasonable controls over privileged material, courts might find that they have waived their privilege. You should therefore actively manage your access governance, not only as a security concern but as a legal preservation requirement.When you extract content into separate analytics systems or grant broader access than your matter structures support, you create pressure on both protections. You gain visibility but lose confidence in your controls.

In this post, we show you a reference architecture that automates sensitive data discovery across legal document repositories on Amazon Web Services (AWS), demonstrate how to capture structured findings as a compliance dataset, and guide you through building a governed analytics workspace that maintains your security boundaries. You walk away with a practical model for building security and analytics into the same lifecycle, without moving documents outside their system of record.

Analytics shouldn’t weaken governance

Most legal organizations have invested heavily in securing their document repositories. You store documents in structured storage, organized by client and matter. You access controls map to matter boundaries (the organizational and access structures that separate one client engagement from another). You establish retention and hold policies.The difficulty starts when teams want to analyze what’s inside those repositories. Running analytics typically means copying content into a separate system, standing up a new data pipeline, or granting broader access than existing matter structures support. Each of these steps introduces governance gaps. Manual reporting fills some of the void, but it doesn’t scale and can’t provide continuous visibility. What’s missing is a model where security controls and analytics reinforce each other, where the act of discovering sensitive data also produces the dataset that you use for reporting, and where governance applies once and carries through every downstream operation.

Automation addresses this by combining continuous sensitive data discovery with governed analytics, built on discovery metadata rather than document copies. This automated approach delivers four key advantages:

  • No document movement. Your files stay in their system of record. Analytics runs against structured discovery metadata, not document content, so governance boundaries remain intact.
  • Continuous discovery instead of manual scanning. Automated classification identifies regulated and sensitive information on an ongoing basis, replacing periodic manual reviews with on demand visibility.
  • Unified governance. You define matter-aligned access policies once, and they carry through from document storage to findings analytics and compliance reporting.
  • Built-in audit readiness. A durable record of discovery findings and remediation actions accumulates automatically over time, giving you structured evidence for client reviews and regulatory inquiries.

Reference Architecture

The following architecture shows how continuous discovery, governance, and compliance operations can work together without copying legal documents into analytics systems.

This reference architecture illustrates how law firms and corporate legal departments can automate sensitive data discovery and compliance analytics on AWS without moving documents outside their system of record

Architecture walkthrough

Store and protect documents in Amazon Simple Storage Service (Amazon S3)

Store your legal documents in Amazon S3, which serves as the system of record for document content. Align your buckets and prefixes to client and matter structures so that access controls map directly to matter boundaries. Where your retention or legal hold requirements demand it, apply S3 Object Lock to enforce immutability. You can encrypt your data using AWS Key Management Service (AWS KMS), which gives you centralized control over encryption keys and policies.

Discover and classify sensitive data with Amazon Macie

You will configure Amazon Macie to continuously analyze your document repositories. Macie identifies regulated information such as personally identifiable information (PII), financial data, and other sensitive content and produces structured findings that describe what Macie identified and where it exists. This provides ongoing visibility into data exposure without requiring document movement or manual scanning.

Catalog and govern findings with AWS Glue and AWS Lake Formation

You will use AWS Glue to catalog the findings dataset and maintain its schema so it stays query-ready. Apply AWS Lake Formation tag-based policies to govern access, aligning tags to client, matter, and confidentiality tier. This approach enforces ethical walls and least-privilege access consistently across analytics and reporting activities.

AI-powered chat agent using Amazon Quick Suite

You can create custom chat agents to tailor conversational interfaces for specific legal business needs. These agents can be configured with legal-specific knowledge bases, connected to relevant document repositories, and customized with instructions appropriate for legal workflows. You can use this chat agent to interact with your legal documents through natural language conversation for capabilities like:

  • E-Discovery:Search and analyze large volumes of legal documents to quickly find relevant information across your document repository.
  • Contract Analysis:Review contracts and automatically extract key terms, clauses, and obligations to streamline your contract review process.

The chat agent can help you navigate complex document sets through conversational queries, making legal research and document review more efficient and accessible.

Analyze and report with Amazon Quick Sight

You will use Amazon Quick as your compliance operations workspace. Quick provides a unified environment where your teams can query findings, generate dashboards, track remediation actions, and produce audit-ready reports. The agentic AI capabilities of Amazon Quick can autonomously build analyses, surface anomalies across matters, generate executive summaries for client reviews, and proactively recommend remediation priorities based on finding severity and trends. Combined with built-in data stories for automated narrative generation and pixel-perfect paginated reports for regulatory submissions, Quick reduces the time from discovery to action while keeping your teams within a governed interface aligned to matter-based permissions. Rather than switching between separate visualization, workflow, and reporting tools, your legal and compliance teams can review findings, manage response activities, and collaborate all within a single workspace that respects ethical walls and privilege boundaries.

Escalate high-severity findings

For high-severity findings that demand immediate attention, route alerts through AWS Security Hub or Amazon Simple Notification Service (Amazon SNS) to trigger escalation workflows. This connects visibility directly to action when your teams identify sensitive data risks.

Why this approach works for legal

Documents stay where they belong. Your files remain in Amazon S3, aligned to client and matter boundaries. No content moves into separate analytics pipelines.Ethical walls remain intact. Because analytics is built on discovery findings and not document copies, you can govern access to findings using the same matter-aligned controls that apply to documents. Compliance and security teams gain visibility without expanding document access.Discovery runs continuously, not periodically. Rather than scheduling quarterly or annual scans, you maintain a current view of sensitive data across your repositories.

Governance applies once and carries through. Lake Formation tag-based policies govern findings access at the catalog level. You define your matter and confidentiality mappings once, and they carry through to every dashboard, query, and report.Audit readiness is built in. Instead of assembling reports manually before a client review or regulatory inquiry, you maintain a historical record of discovery findings and remediation actions. You can demonstrate your posture over time with consistent, structured evidence.

Security and analytics reinforce each other. Your analytics capability is built on top of your security controls, not alongside them. Strengthening one strengthens the other.

Cost considerations

The primary cost drivers for this architecture include:

  • Amazon Macie: You pay based on the number of S3 buckets evaluated and the volume of data inspected for sensitive data discovery. Review Amazon Macie pricing for current rates.
  • Amazon S3: Storage costs for both your document repositories and the compliance intelligence bucket. Consider S3 lifecycle policies to tier older findings into lower-cost storage classes.
  • AWS Glue and AWS Lake Formation: Charges for crawlers and catalog storage. For most implementations, these costs are modest.
  • Amazon QuickSight: Per-user pricing based on the edition that you select (Standard or Enterprise). Enterprise edition supports row-level and column-level security, which aligns well with matter-based governance.
  • Amazon EventBridge, AWS Security Hub, and Amazon SNS: Charges based on event volume and notifications delivered. For findings-based workflows, these costs are generally low.

Use the AWS Pricing Calculator to estimate costs based on your repository size, user count, and discovery frequency.

Getting started

Start by identifying a representative set of document repositories in Amazon S3. We recommend that you start with two or three matters that span different practice areas and confidentiality tiers.

  1. Turn on Amazon Macie for those repositories and configure automated sensitive data discovery.
  2. Catalog the findings dataset with AWS Glue and apply Lake Formation tag-based access policies aligned to your matter structure.
  3. Build your first Amazon Quick Sight dashboard to visualize findings by matter, sensitivity type, and severity.
  4. Define escalation rules in AWS Security Hub or Amazon SNS for high-severity findings.

After you validate this workflow against your initial repositories, expand gradually. Add more repositories to Macie discovery. Refine your governance tags to reflect practice areas and confidentiality tiers. Extend your dashboards from basic posture visibility to trend analysis and remediation tracking.The goal isn’t to build a comprehensive analytics solution all at once. Start with a secure foundation where discovery findings, governance, and reporting operate together in a way that aligns with your legal workflows, and then expand from there.

Conclusion

You don’t have to choose between protecting client data and understanding it. By building analytics on top of governed discovery findings and using a unified compliance workspace, you gain visibility into your data posture without weakening confidentiality boundaries.This approach brings security, governance, and analytics together in a way that reflects how legal work is actually structured. It provides continuous visibility, supports audit readiness, and delivers insight without requiring documents to move outside their system of record.

Next steps

Review the Amazon Macie User Guide to understand sensitive data discovery configuration options and Amazon Quick Sight documentation to evaluate dashboard and row-level security capabilities.

Contact your AWS account team to discuss implementation support for legal and compliance workloads.


About the authors

Photo of Author - Rohan Kamat

Rohan Kamat

Rohan Kamat is a Solutions Architecture Leader within HCLS with extensive experience in cloud architecture, cybersecurity, Identity and Access Management, and enterprise networking. Rohan focuses on helping architects build both depth in cloud technologies and strength in executive communication, making sure they can confidently guide organizations through business and technical transformation. Outside of his professional work, Rohan enjoys time with his family, organizing community cricket events, and exploring fitness and wellness activities like pickleball and ping pong. He also enjoys planning travel experiences that bring people together and create lasting shared memories.

Photo of Author- Miguel Lopez Luis

Miguel Lopez Luis

Miguel Lopez Luis is an AWS Solutions Architect who works with small and medium businesses across the United States. He graduated with a Bachelor’s degree in Cybersecurity from Bellevue University in Nebraska and is a member of the Omega Nu Lambda Honor Society. Leveraging his extensive expertise in business management, Miguel is passionate about planning strategic initiatives, leading cross-functional teams, and mentoring others. In his personal time, he enjoys activities that involve travel, sports, and cooking.

Photo of Author - Pranali Khose

Pranali Khose

Pranali Khose is an AWS Solutions Architect based in Seattle. She works directly with small and medium business (SMB) customers across the United States, to design and implement cloud solutions that address their unique business challenges and accelerate digital transformation. Pranali holds a Master of Science in Computer Science from the University of Texas at Arlington.

How to use streamlined permissions for Amazon S3 Tables and Iceberg materialized views

Post Syndicated from Srividya Parthasarathy original https://aws.amazon.com/blogs/big-data/how-to-use-streamlined-permissions-for-amazon-s3-tables-and-iceberg-materialized-views/

Apache Iceberg has emerged as the open table format for data lakes. It handles petabyte-scale datasets, lets teams evolve schemas and partitions in place, and supports time travel and incremental processing for data lake management at scale. Amazon S3 Tables provide a fully managed Apache Iceberg table experience in Amazon S3, optimized for analytics workloads, and integrate with the AWS Glue Data Catalog so AWS analytics services such as Amazon Redshift, Amazon EMR, Amazon Athena, Amazon SageMaker, and AWS Glue query your data. Together, they form the foundation of a modern data lake architecture on AWS.

S3 Tables integrate with the AWS Glue Data Catalog using AWS Identity and Access Management (IAM) – based authorization. If you manage analytics workloads across these services, you can now define permissions across storage, catalog, and compute in a single IAM policy. This gives teams already using IAM a straightforward path to govern access to S3 Tables resources without changing their existing permission model. For fine-grained access controls, you can opt in to AWS Lake Formation at any time through the AWS Management Console, AWS Command Line Interface (AWS CLI), API, or AWS CloudFormation.

Iceberg materialized views created in the Glue Data Catalog extend this foundation by letting you store pre-computed query results as Iceberg data on Amazon S3. When a query repeats aggregations or joins across large datasets, the engine reads directly from the materialized view’s S3 location rather than reprocessing the base tables. A materialized view can reside in S3 Tables or in an S3 general purpose bucket, independent of where its base tables live, which lets you place pre-computed results wherever fits your access patterns and cost model best.

In this post, we walk through how to set up and manage S3 Tables in the AWS Glue Data Catalog, create and query Iceberg materialized views, and configure access controls that work across your analytics stack with IAM-based authorization.

 Solution overview

Architecture diagram showing AWS Glue Data Catalog integration with Amazon Athena, AWS Glue, Amazon Redshift, and Amazon EMR through IAM roles and policies, with Amazon S3 storage and optional AWS Lake Formation governance.

The above architecture illustrates how S3 Tables integrate with AWS Glue Data Catalog using IAM-based authorization, so you can define the necessary permissions across storage, catalog, and query engines in a single IAM policy. This permission model accelerates onboarding for new teams and workloads.

Key architecture components include:

Storage Layer: Data stored as Iceberg tables in Amazon S3 Tables

Catalog Layer: AWS Glue Data Catalog serves as the single metadata repository.

Compute Layer – Amazon Athena, AWS Glue, Amazon Redshift, and Amazon EMR connect to a single data Catalog to access Iceberg tables.

Security: AWS IAM authorizes access to resources in storage, catalog, and compute layers.

Prerequisites:

To follow along with this post, you must have an AWS account and an IAM role or user with appropriate permissions and familiarity to the following services:

  • IAM
  • AWS Glue Data Catalog
  • Amazon S3
  • Amazon Athena
  • Amazon Redshift
  • Amazon EMR

For the minimum permissions required for the role/user for metadata and data access, refer to required IAM permissions documentation.

Solution walkthrough

In this walkthrough, you will integrate S3 Tables with the AWS Glue Data Catalog, create Iceberg materialized views, and query data using multiple analytics engines. You will also learn to use materialized views when you have complex aggregations queried frequently but underlying data changes. You can follow these steps to implement the solution. It will take about 45–60 minutes to complete this walkthrough.

Setup S3 Tables and integrate with Glue Data Catalog

Navigate to Amazon S3 console:

  1. On the left menu, select Table buckets.
  2. Choose the Create table bucket button.

Amazon S3 console showing the Table buckets management page in the US West (N. California) us-west-1 Region with zero table buckets, integration status disabled, and the Create table bucket button highlighted.

  1. In the next screen, we will fill the name of the bucket as salesbucket. Please ensure the Enable Integration configuration is checked. This step integrates S3 Tables with AWS Glue Data Catalog.

AWS S3 Create table bucket form with General configuration showing bucket name "salesbucket" and Integration with AWS analytics services section with Enable integration checkbox selected.

  1. Keep the other options as default and choose Create table bucket.
  2. After it is created, you will be redirected back to the list of table buckets. Choose the table bucket salesbucket.
  3. Select the Create table with Athena button.
  4. Create a namespace in S3 Tables which is equivalent to a database in AWS Glue Data Catalog. Enter namespace (database) name as “sales” and click Create namespace.

Create table with Athena dialog in the Amazon S3 salesbucket console showing namespace configuration with "Create a namespace" selected and namespace name set to "sales."

  1. Choose Create table with Athena, and a new tab will be open with the Amazon Athena console.
  2. When the Amazon Athena console opens, you will see an example of a query to create a table and examples to insert rows in that table. You could use this query block by uncommenting the code and executing each statement individually by highlighting it. At the end, you will have data in the table.

Amazon Athena query editor showing a SQL analytics query on the daily_sales table with results displaying product categories, units sold, total revenue, and average price for February 2024 sales data.

Query S3 Tables and create materialized view using Amazon EMR:

To run the instruction on Amazon EMR, complete the following steps to configure the cluster:

  1. Create an IAM role for the Amazon EMR instance profile following the Amazon EMR Management Guide. Add the following as policies and trust relationship for working on materialized views.

Replace ACCOUNT_ID with your AWS account ID, Instance_profile_role to the Amazon EMR instance profile role, and REGION with your AWS Region.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"GlueDataCatalogPermissions",
         "Effect":"Allow",
         "Action":[
            "glue:GetCatalog",
            "glue:GetDatabase",
            "glue:CreateTable",
            "glue:GetTable",
            "glue:GetTables",
            "glue:UpdateTable",
            "glue:DeleteTable"
         ],
         "Resource":[
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:catalog",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:catalog/s3tablescatalog",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:catalog/s3tablescatalog/*",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:database/salesdb",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:database/salesdb/*",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:database/s3tablescatalog",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:database/s3tablescatalog/*",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:table/s3tablescatalog/*",
            "arn:aws:glue:<REGION>:<ACCOUNT ID>:table/*/*"
         ]
      },
      {
         "Sid":"S3TablesDataAccessPermissions",
         "Effect":"Allow",
         "Action":[
            "s3tables:GetTableBucket",
            "s3tables:GetNamespace",
            "s3tables:GetTable",
            "s3tables:GetTableMetadataLocation",
            "s3tables:GetTableData",
            "s3tables:ListTableBuckets",
            "s3tables:CreateTable",
            "s3tables:PutTableData",
            "s3tables:UpdateTableMetadataLocation",
            "s3tables:ListNamespaces",
            "s3tables:ListTables",
            "s3tables:DeleteTable"
         ],
         "Resource":[
            "arn:aws:s3tables:<REGION>:<ACCOUNT ID>:bucket/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":"iam:PassRole",
         "Resource":"arn:aws:iam::<ACCOUNT ID>:role/service-role/<Instance_profile_role>"
      }
   ]
}

Add the following to the trust policy in addition to existing:

 {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "glue.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
  1. Launch an Amazon EMR cluster 7.12.0 or higher with instance profile role created in the previous step and with Iceberg enabled. For more information, refer to Use an Iceberg cluster with Spark.
  2. Connect to the primary node of your Amazon EMR cluster by using SSH, and run the following command to start a Spark application with the required configurations:

Replace bucket_name with your bucket name.

spark-sql \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.glue_catalog.type=glue \
  --conf spark.sql.catalog.glue_catalog.warehouse=s3://<bucket_name> \
  --conf spark.sql.catalog.glue_catalog.glue.region=<region> \
  --conf spark.sql.catalog.glue_catalog.glue.id=<accountid>:s3tablescatalog/salesbucket \
  --conf spark.sql.catalog.glue_catalog.glue.account-id=<accountid> \
  --conf spark.sql.catalog.glue_catalog.client.region=<region> \
  --conf spark.sql.optimizer.answerQueriesWithMVs.enabled=true \
  --conf spark.sql.defaultCatalog=glue_catalog
  1. Run the following queries to query the daily_sales table.
spark-sql ()> use sales;
spark-sql (sales)> select * from daily_sales;
2024-01-15 Laptop 900.0
2024-01-15 Monitor 250.0
2024-01-16 Laptop 1350.0
2024-02-01 Monitor 300.0
2024-02-01 Keyboard 60.0
2024-02-02 Mouse 25.0
2024-02-02 Laptop 1050.0
2024-02-03 Laptop 1200.0
2024-02-03 Monitor 375.0
  1. Create Materialized view.
CREATE MATERIALIZED VIEW sales_mv as 
SELECT 
    product_category,
    COUNT(*) as units_sold,
    SUM(sales_amount) as total_revenue, 
    AVG(sales_amount) as average_price 
FROM 
    glue_catalog.sales.daily_sales 
GROUP BY 
    product_category;

A newly created materialized view is populated with the initial query results but does not update automatically as base table data changes. To keep it current, specify a REFRESH EVERY clause when creating the view. This accepts a time interval and unit, so you can define how often the materialized view is recomputed from the base tables.

  1. Add refresh interval.
CREATE MATERIALIZED VIEW sales_mv 
SCHEDULE REFRESH EVERY 2 HOURS as 
SELECT 
    product_category,
    COUNT(*) as units_sold,
    SUM(sales_amount) as total_revenue, 
    AVG(sales_amount) as average_price 
FROM 
    glue_catalog.sales.daily_sales 
GROUP BY 
    product_category;
  1. Alternatively, you can refresh them manually.

For manual full refresh, you can use the following command:

REFRESH MATERIALIZED VIEW sales_mv FULL;

For manual incremental refresh, you can use the following command:

REFRESH MATERIALIZED VIEW sales_mv;

For more details, refer to Refreshing materialized views.

  1. Query the MV.
spark-sql (sales)> select * from sales_mv
Keyboard 1 60.0 60.0
Laptop 4 4500.0 1125.0
Mouse 1 25.0 25.0
Monitor 3 925.0 308.3333333333333

After the Iceberg materialized views are created, you can access them using IAM principals that have required IAM permissions to Glue Data Catalog resource and its underlying storage.

Iceberg materialized views are flexible in how they combine base tables and access control modes. Base tables can reside in S3 general-purpose buckets (with IAM or Lake Formation access control), in S3 Tables (through the s3tablescatalog catalog), or a combination of these—all within a single materialized view definition. The materialized view itself can use either IAM or AWS Lake Formation access control, independently of its base tables.

For more details, refer to How materialized views work with AWS Glue.

Query using Athena:

Additionally, you can query the same materialized view from Athena SQL. The following image shows the same query run on Athena and the resulting output.Amazon Athena query editor showing SELECT query results from the sales_mv materialized view with product category aggregations including Keyboard and Laptop sales data.

Query using Amazon Redshift:

To query the S3 Tables in AWS Glue Data Catalog using Amazon Redshift, you must create a database in the default catalog in Glue Data Catalog that points to the S3 Tables catalog.

  1. On the AWS Glue console, choose Databases, and then choose Add Database.

AWS Glue Data Catalog Databases page showing one default database in catalog 466053964652, with the Add database button highlighted.

  1. Choose the Glue Database resource link option, add a name for the database, choose salesbucket on the target catalog and sales as the target database. Then select Create database.

AWS Glue Create a database form with Glue Database Resource Link selected, name set to "salesdb," target catalog "salesbucket," and target database "sales."

After creating the database, we will see the “salesdb” resource link under Databases on AWS Glue Data Catalog.

AWS Glue Data Catalog Databases page showing two databases: "default" and the newly created "salesdb" resource link with source catalog pointing to s3tablescatalog.

Create IAM role with the following policy for the Amazon Redshift schema creation. Replace the AWS Region and account ID for your account.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"GlueDataCatalogPermissions",
         "Effect":"Allow",
         "Action":[
            "glue:GetCatalog",
            "glue:GetDatabase",
            "glue:CreateTable",
            "glue:GetTable",
            "glue:GetTables",
            "glue:UpdateTable",
            "glue:DeleteTable"
         ],
         "Resource":[
            "arn:aws:glue:<REGION>:<ACCOUNTID>:catalog",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:catalog/s3tablescatalog",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:catalog/s3tablescatalog/*",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:database/salesdb",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:database/salesdb/*",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:database/s3tablescatalog",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:database/s3tablescatalog/*",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:table/s3tablescatalog/*",
            "arn:aws:glue:<REGION>:<ACCOUNTID>:table/*/*"
         ]
      },
      {
         "Sid":"S3TablesDataAccessPermissions",
         "Effect":"Allow",
         "Action":[
            "s3tables:GetTableBucket",
            "s3tables:GetNamespace",
            "s3tables:GetTable",
            "s3tables:GetTableMetadataLocation",
            "s3tables:GetTableData",
            "s3tables:ListTableBuckets",
            "s3tables:CreateTable",
            "s3tables:PutTableData",
            "s3tables:UpdateTableMetadataLocation",
            "s3tables:ListNamespaces",
            "s3tables:ListTables",
            "s3tables:DeleteTable"
         ],
         "Resource":[
            "arn:aws:s3tables:<REGION>:<ACCOUNTID>:bucket/*"
         ]
      }
   ]
}

Create an Amazon Redshift provisioned cluster or Amazon Redshift Serverless, attaching the IAM role created in previous step.

To access the AWS Glue Catalog and the resource link, you can now log in to Amazon Redshift as a local user. We use the admin user and Amazon Redshift Query Editor v2.

Amazon Redshift Query Editor v2 interface connected to Serverless workgroup "s3tablesblog" showing 2 native databases and 1 external database with an empty query editor ready for input.

To create the external schema, you must run the following command: Replace ACCOUNT_ID with your AWS Account ID, IAM_ROLE to IAM role created for schema access, and REGION with your AWS Region.

CREATE EXTERNAL SCHEMA salesdb
FROM DATA CATALOG DATABASE 'salesdb'
IAM_ROLE 'arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE>'
REGION '<REGION>'
CATALOG_ID '<ACCOUNT_ID>';

After you have created the external schema, it will show up on the left side, under the dev database. The table that we created, daily_sales, is available and we can query directly from Amazon Redshift using a local user.

Amazon Redshift Query Editor v2 showing a SELECT query on the daily_sales table in the salesdb schema with 9 rows of results displaying sale dates, product categories, and sales amounts from January–February 2024.

Cleanup:

After completing the walkthrough, follow these steps to remove the resources and avoid ongoing charges. These cleanup steps will permanently delete the data, including the daily_sales table and sales_mv materialized view. Make sure that you have backed up the data that you need to retain before proceeding.

To avoid incurring future charges, clean up the resources that you created during this walkthrough:

  • Remove the Glue Data Catalog resources
  • Delete the table bucket
  • Terminate and Delete the Amazon Redshift cluster
  • Terminate and Delete the Amazon EMR cluster
  • Delete the IAM roles/policies created

Conclusion

Amazon S3 Tables now integrate with AWS Glue Data Catalog through IAM-based authorization via a single IAM policy. By consolidating permissions for storage, catalog, and query engines into one IAM policy, you can streamline authorization with AWS analytics services like Amazon Athena, Amazon EMR, and AWS Glue. You can use this streamlined IAM authorization model to build your data lake faster while maintaining enterprise-grade security. For organizations with additionally granular data access requirements, AWS Lake Formation remains available to layer fine-grained access controls on top of this foundation. This is configurable through the AWS Management Console, CLI, API, or CloudFormation. This integration allows AWS analytics users to use IAM and scale their analytics capabilities with reduced operational complexity.

To learn more about to S3 Tables and integration with Glue Data catalog, visit: Amazon S3 Tables integration with AWS analytics services overview and Integrating with Amazon S3 Tables.


About the authors

Ricardo Serafim

Ricardo is a Senior Analytics Specialist Solutions Architect at AWS. He has been helping companies with Data Warehouse solutions since 2007.

Milind Oke

Milind is a Data Warehouse Specialist Solutions Architect based out of New York. He has been building data warehouse solutions for over 15 years and specializes in Amazon Redshift.

Pratik Das

Pratik is a Senior Product Manager with AWS Lake Formation. He is passionate about all things data and works with customers to understand their requirements and build delightful experiences. He has a background in building data-driven solutions and machine learning systems.

Srividya Parthasarathy

Srividya is a Senior Big Data Architect on the AWS Lake Formation team. She works with the product team and customers to build robust features and solutions for their analytical data platform. She enjoys building data mesh solutions and sharing them with the community.

PACIFIC enables multi-tenant, sovereign product carbon footprint exchange on the Catena-X data space using AWS

Post Syndicated from Kevin S. Ridolfi original https://aws.amazon.com/blogs/architecture/pacific-enables-multi-tenant-sovereign-product-carbon-footprint-exchange-on-the-catena-x-data-space-using-aws/

This post is cowritten by Anil Akarsu and Dr. Renè Holschuh from BASF.

BASF is a global chemical industry leader and active member of the Catena-X Automotive Network. It pioneers sustainable solutions that enable automotive organizations to track carbon emissions across complex supply chains. CircularTree transforms sustainability reporting through innovative digital solutions that systematically identify and control Scope 3 greenhouse gas (GHG) emissions across global supply networks. They establish standardized data exchange protocols through participation in forward-thinking associations including PACT, Catena-X, and ESTAINIUM. BASF and CircularTree created PACIFIC, a product powered by AWS that streamlines transparent product carbon footprint (PCF) reporting across the value chain by automating PCF data exchange, reducing manual effort, and ensuring trustworthy data sharing. Through this unique relationship, AWS helps customers integrate software, services, and processes to accelerate business transformation. This post explores how PACIFIC enables multi-tenant, sovereign PCF exchange on the Catena-X data space using Amazon Elastic Container Service (Amazon ECS) on AWS Fargate, Amazon Cognito, and AWS Identity and Access Management (IAM) to deliver measurable environmental impact and competitive advantage in a carbon-conscious marketplace.

Carbon data at scale, across company borders

Sustainability is now an operational requirement, driven by growing regulatory pressure in the European Union and increasing customer expectations for credible and auditable emissions data. For manufacturers in the automotive supply chain, this is especially challenging because emissions data does not live in one place. It is distributed across fragmented tiers of suppliers, different internal systems, and partner-to-partner handoffs that still happen through spreadsheets, emails, and one-off integrations.

At the same time, the industry is converging on shared ways to exchange data, with Catena-X setting expectations for interoperability and trust in cross-company collaboration. PACIFIC was built for this reality as a multi-tenant SaaS product that enables companies to manage and exchange PCFs while maintaining data sovereignty. Its Catena-X certification signals alignment with industry standards, and the partnership with BASF grounds the platform in real supply chain requirements.

Data security, sovereignty, and interoperability

To make PCF exchange work in the real world, PACIFIC needed to solve two problems at the same time: enable frictionless collaboration across companies, while guaranteeing that each company stays in full control of their data and credentials. The platform had to operate as a multi-tenant software as a service (SaaS) for the supply chain, serving organizations on shared infrastructure without introducing any possibility of cross-tenant access.

They had to build an interoperable solution that could communicate with other solution providers on the Catena-X data space, using Eclipse Dataspace Components (EDC) connectors as a standard mechanism for cross-company data exchange. That meant enforcing strict data sovereignty, not only for PCF records but also for sensitive Catena-X integration configuration such as EDC and Digital Twin Registry (DTR) credentials. At the exchange layer, PACIFIC needed end-to-end authorization aligned with Catena-X expectations, where PCF data is shared only after explicit agreement and policy negotiation through EDC. Finally, the solution had to be practical to run and scale, so the following had to happen:

  • Onboard new companies without spinning up separate AWS accounts per tenant
  • Integrate suppliers’ PCF systems like BASF without tight coupling to the exchange workflow
  • Keep the platform secure, auditable, and operable as usage grows

Solution overview

Figure 1 gives a high-level view of how PACIFIC is built and deployed to enable secure, multi-tenant PCF exchange on the Catena-X data space. It shows the main building blocks of the product, how user traffic reaches the application, how tenant-aware identity and authorization are enforced, and how PACIFIC separates core platform features from integrations and exchange endpoints. The diagram also highlights the external connections enabling interoperability, including supplier PCF data sources like BASF services, and EDC and DTR “enablement service providers” for Catena-X based data sharing.

AWS Cloud architecture diagram for the PACIFIC platform showing a multi-layered system. At the top, a PACIFIC Web Client connects to an Identity & Authorization layer containing Amazon Cognito, AWS IAM, and AWS Secrets Manager. Traffic flows through AWS WAF to an Application Load Balancer within a VPC, which distributes requests to Amazon ECS (AWS Fargate) hosting four containerized microservices: core-modules, integration-module, pcf-exchange-module, and edc-dtr-module. These modules connect to Amazon RDS for relational database storage and Amazon S3 for object storage. External integrations at the bottom include BASF Product Carbon Footprint Services, an EDC/DTR Service Provider, and the Catena-X Automotive Network. The diagram illustrates a secure, microservices-based architecture for automotive industry carbon footprint data exchange.

Figure 1: PACIFIC high-level service architecture

Data protection through IAM-based tenant isolation

A core requirement for PACIFIC is maintaining data protection and security. Each company must have exclusive control over their PCF data, EDC connector, and DTR management credentials, without any possibility of cross-tenant access. Rather than provisioning separate AWS accounts per tenant PACIFIC implements a fine-grained IAM-based isolation model built on Amazon Cognito and AWS Secrets Manager. When a company joins the platform, PACIFIC automatically provisions a dedicated IAM role with a scoped policy that permits access only to that company’s secrets in Secrets Manager. Users are assigned to an Amazon Cognito user pool group linked to their company’s IAM role. When a user authenticates, the Amazon Cognito identity pool maps their group membership to the corresponding IAM role, and AWS Security Token Service (AWS STS) issues temporary credentials for that role. This means, a user’s credentials can only retrieve their own company’s EDC secrets and access to other tenants’ configuration is denied at the IAM policy level. This architecture delivers true multi-tenant isolation using native AWS identity services, without the overhead of managing dedicated accounts or Amazon Virtual Private Cloud (Amazon VPC) per customer.

Securing PCF exchange with EDC-issued authorization tokens

Beyond tenant isolation within PACIFIC, CircularTree enforces authorization at the data exchange layer through the pcf-exchange-module, a per-tenant endpoint that serves PCF data to authorized trading partners. When a consumer’s EDC connector requests PCF data from a supplier’s EDC, the two connectors negotiate and agree on usage policies governing how the exchanged information can be used. After this agreement is established, the supplier’s EDC issues a special authorization token to the consumer’s EDC. The token derives from the supplier company’s Cognito app client credentials stored within their EDC and grants access specifically to that supplier’s pcf-exchange-module endpoint in PACIFIC. The consumer’s EDC then uses this token to call the supplier’s dedicated endpoint and retrieve the authorized PCF data. Because each tenant’s pcf-exchange-module is published as an individual endpoint, which only accepts tokens issued through the EDC handshake process, unauthorized access is prevented at multiple levels. One level is through EDC policy negotiation, and the other is through company-scoped OAuth2 token validation. This ensures that PCF data is transmitted only after explicit consent and only to the specific trading partner authorized in the data exchange agreement. Figure 2 gives an overview of the communication flow.

Data flow diagram showing a six-step secure token exchange process between Consumer EDC and Supplier EDC systems via the PACIFIC platform on AWS. The flow proceeds as follows: (1) Consumer EDC sends a Request PCF to Supplier EDC, (2) Policy Negotiation occurs between the two EDCs (shown as a dashed line), (3) a token is issued (marked with a key icon), (4) Supplier EDC provides a Data Space URL to the pcf-exchange-module, (5) Token Validation occurs between the pcf-exchange-module and the supplier-oauth2-client component (marked with a key icon), and (6) PCF Data flows back to the Consumer EDC. The PACIFIC layer at the bottom contains two AWS-hosted components: the pcf-exchange-module (orange icon) and the supplier-oauth2-client (red icon with checkmark), demonstrating OAuth2-based secure authentication for Product Carbon Footprint data exchange.

Figure 2: EDC-to-EDC communication with Cognito Oauth2 Tokens

Integrating supplier PCF systems through the integration module

While the pcf-exchange-module handles secure data exchange between trading partners using the Catena-X data space, PACIFIC also needs to ingest PCF data from suppliers’ internal systems. Running on AWS Fargate, the integration-module provides a flexible, scalable integration layer that connects to proprietary supplier PCF systems, such as BASF’s internal PCF services. Each supplier integration requires handling unique authentication flows. This ranges from OAuth2 client credentials to certificate-based authentication or API key mechanisms, all of which are securely managed through AWS Secrets Manager. The integration-module expects incoming PCF data to already conform to the standardized Catena-X PCF JSON format, ensuring consistency at the point of ingestion. After received, PCF data is stored in Amazon Simple Storage Service (Amazon S3) under company-specific prefixes. This is where IAM policies make sure that only the PCF owner company can access their respective data. By decoupling supplier system integration from the data exchange layer, PACIFIC can onboard new supplier PCF data sources without impacting the downstream Catena-X data sharing workflows. This can be done while the S3-based storage model helps maintain strict data sovereignty, and each company’s PCF data remains isolated and accessible only to its rightful owner.

Conclusion

PACIFIC turns Catena-X PCF exchange from a specification into an interoperable, scalable workflow running on Amazon ECS and AWS Fargate without requiring companies to give up control of their data and credentials. The impact is measurable in both speed and operational scalability. From a business perspective, BASF highlights the most tangible improvement: when a requested PCF dataset is already available, a manual exchange can take up to around seven days, whereas PACIFIC responds in seconds, and can deliver automated updates when PCFs change. This capability delivers up to 75% time savings for both customers and BASF. It also reduces the time-to-data from days to seconds and making emissions information more visible and usable in day-to-day supply chain operations. PACIFIC’s multi-tenant architecture scales onboarding and operations without managing individual AWS accounts per company, while still enforcing strong tenant isolation through IAM-scoped access control and per-tenant exchange endpoints. This scalability translates into faster onboarding, enabling BASF to integrate significantly more partners into the data space. The results are measurable: an 80% increase in newly onboarded companies between 2024 and 2025, and 55% growth in requested products and shared PCFs over the same period. This provides a scalable baseline for expanding the number of onboarded organizations and increasing the volume of PCF data exchanged as adoption grows—without weakening data sovereignty or interoperability. These results underline PACIFIC’s role as a catalyst for accelerating decarbonization across supply chains.

We encourage you to join BASF, CircularTree, and AWS in industry data sharing through emerging data spaces and transparent, trusted PCF exchange across global supply networks. To explore more sustainability solutions and AWS architecture patterns, visit the AWS Architecture Blog and get started with PACIFIC through the Cofinity-X App Marketplace.


About the authors

AWS Weekly Roundup: Claude Mythos Preview in Amazon Bedrock, AWS Agent Registry, and more (April 13, 2026)

Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-mythos-preview-in-amazon-bedrock-aws-agent-registry-and-more-april-13-2026/

In my last Week in Review post, I mentioned how much time I’ve been spending on AI-Driven Development Lifecycle (AI-DLC) workshops with customers this year. A common theme in those sessions is the need for better cost visibility. Teams are moving fast with AI, but as they go from experimenting to full production, finance and leadership really need to know who is using which resources and at what cost. That’s why I was so excited to see the launch of Amazon Bedrock new support for cost allocation by IAM user and role this week. This lets you tag IAM principals with attributes like team or cost center and then activate those tags in your Billing and Cost Management console. The resulting cost data flows into AWS Cost Explorer and the detailed Cost and Usage Report, giving you a clear line of sight into model inference spending. Whether you’re scaling agents across teams, tracking foundation model use by department, or running tools like Claude Code on Amazon Bedrock, this new feature is a game changer for tracking and managing your AI investments. You can get all the details on setting this up in the IAM principal cost allocation documentation.

Now, let’s get into this week’s AWS news…

Headlines
Amazon Bedrock now offers Claude Mythos Preview Anthropic’s most sophisticated AI model to date is now available on Amazon Bedrock as a gated research preview through Project Glasswing. Claude Mythos introduces a new model class focused on cybersecurity, capable of identifying sophisticated security vulnerabilities in software, analyzing large codebases, and delivering state of the art performance across cybersecurity, coding, and complex reasoning tasks. Security teams can use it to discover and address vulnerabilities in critical software before threats emerge. Access is currently limited to allowlisted organizations, with Anthropic and AWS prioritizing internet critical companies and open source maintainers.

AWS Agent Registry for centralized agent discovery and governance now in preview AWS launched Agent Registry through Amazon Bedrock AgentCore, providing organizations with a private catalog for discovering and managing AI agents, tools, skills, MCP servers, and custom resources. The registry helps teams locate existing capabilities rather than duplicating them, with semantic and keyword search, approval workflows, and CloudTrail audit trails. It is accessible via the AgentCore Console, AWS CLI, SDK, and as an MCP server queryable from IDEs.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Announcing Amazon S3 Files, making S3 buckets accessible as file systems — Amazon S3 Files transforms S3 buckets into shared file systems that connect any AWS compute resource directly with your S3 data. Built on Amazon EFS technology, it delivers full file system semantics with low latency performance, caching actively used data and providing multiple terabytes per second of aggregate read throughput. Applications can access S3 data through both file system and S3 APIs simultaneously without code modifications or data migration.
  • Amazon OpenSearch Service supports Managed Prometheus and agent tracing —Amazon OpenSearch Service now provides a unified observability platform that consolidates metrics, logs, traces, and AI agent tracing into a single interface. The update includes native Prometheus integration with direct PromQL query support, RED metrics monitoring, and OpenTelemetry GenAI semantic convention support for LLM execution visibility. Operations teams can correlate slow traces to logs and overlay Prometheus metrics on dashboards without switching between tools.
  • Amazon WorkSpaces Advisor now available for AI powered troubleshooting— AWS launched Amazon WorkSpaces Advisor, an AI powered administrative tool that uses generative AI to help IT administrators troubleshoot Amazon WorkSpaces Personal deployments. It analyzes WorkSpace configurations, detects problems automatically, and provides actionable recommendations to restore service and optimize performance.
  • Amazon Braket adds support for Rigetti’s 108 qubit Cepheus QPU — Amazon Braket now offers access to Rigetti’s Cepheus-1-108Q device, the first 100+ qubit superconducting quantum processor on the platform. The modular design features twelve 9 qubit chiplets with CZ gates that offer enhanced resilience to phase errors. It supports multiple frameworks including Braket SDK, Qiskit, CUDA-Q, and Pennylane, with pulse level control for researchers.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • What’s Next with AWS (April 28, Virtual) Join this livestream at 9am PT for a candid discussion about how agentic AI is transforming how businesses operate. Featuring AWS CEO Matt Garman, SVP Colleen Aubrey, and OpenAI leaders discussing emerging agent capabilities, Amazon’s internal experiences, and new agentic solutions and platform capabilities.

Browse here for upcoming AWS led in person and virtual events, startup events, and developer focused events.


That’s all for this week. Check back next Monday for another Weekly Roundup!

~ micah

Building Memory-Intensive Apps with AWS Lambda Managed Instances

Post Syndicated from Guy Haddad original https://aws.amazon.com/blogs/compute/building-memory-intensive-apps-with-aws-lambda-managed-instances/

Building memory-intensive applications with AWS Lambda just got easier. AWS Lambda Managed Instances gives you up to 32 GB of memory—3x more than standard AWS Lambda—while maintaining the serverless experience you know. Modern applications increasingly require substantial memory resources to process large datasets, perform complex analytics, and deliver real-time insights for use cases such as in-memory analytics, Machine Learning (ML) model inference, and real-time semantic search. AWS Lambda Managed Instances gives you a familiar serverless programming model and experience combined with the flexibility of being able to choose the underlying Amazon EC2 instance types and providing developers with access to large memory configurations.

In this post, you will see how AWS Lambda Managed Instances enables memory-intensive workloads that were previously challenging to run in serverless environments, using an AI-powered customer analytics application as a practical example. You’ll see cost savings of up to 33% compared to standard Lambda for predictable workloads, while eliminating the operational overhead of managing EC2 instances.

Understanding AWS Lambda Managed Instances

AWS Lambda Managed Instances runs your AWS Lambda functions on the Amazon EC2 instance types of your choice in your account, including Graviton4 and memory-optimized instance types. AWS handles underlying infrastructure lifecycle including provisioning, scaling, patching, and routing, while you benefit from Amazon EC2 pricing advantages like Savings Plans and Reserved Instances.

Key benefits include:

  • Flexible instance selection: Choose from compute-optimized (C), general-purpose (M), and memory-optimized (R) instance families
  • Configurable memory-CPU ratios: Optimize resource allocation for your workload
  • Multi-concurrent invocations: One execution environment handles multiple invocations simultaneously, improving utilization for I/O-heavy applications
  • Dynamic scaling: Instances scale based on CPU utilization without cold starts

AWS Lambda Managed Instances is best suited for high-volume, predictable workloads that benefit from sustained compute capacity and larger memory configurations.

Memory-Intensive Workloads Work Best with AWS Lambda Managed Instances

This blog focuses on one of AWS Lambda Managed Instances’ most powerful capabilities: running memory-intensive workloads that require more than the standard AWS Lambda’s 10 GB memory and 250MB ZIP limits. Here are the use cases where AWS Lambda Managed Instances helps:

  • In-Memory Analytics — Load gigabytes of structured data into memory at initialization and serve sub-millisecond analytical queries across thousands of invocations
  • ML Model Inference — Keep large model weights resident in memory across invocations for consistent, low-latency inference without a dedicated endpoint.
  • Real-Time Semantic Search — Build vector similarity search over large embedding indexes held entirely in memory, enabling natural language queries over millions of records without an external vector database.
  • Graph Processing — Hold large graph structures in memory for traversal algorithms that require the full graph to be accessible at once.
  • Scientific & Numerical Computing — Run simulations, Monte Carlo methods, and large matrix operations that require substantial working memory and benefit from memory-optimized Amazon EC2 instance families.
  • Large-Scale Report Generation — Aggregate and transform multi-gigabyte datasets in memory to generate complex reports or dashboards on demand, without staging data through intermediate storage.

Use Case: AI-Powered Customer Analytics with AWS Lambda Managed Instances

To demonstrate the power of AWS Lambda Managed Instances for memory-intensive applications, we built an AI-Powered Customer Analytics application that combines in-memory data processing with ML-based semantic search. The application loads in memory 1 million customer behavioral records (sessions, purchases, browsing patterns) from a Parquet file in S3 into a Pandas DataFrame and an embeddings cache consuming 200MB, then responds for analytics queries:

  1. Customer Analysis — Deep-dive into individual customer behavior: engagement scores, conversion rates, purchase patterns, and AI-generated customer segments
  2. Semantic Search — Natural language queries powered by FastEmbed (sentence-transformers/all-MiniLM-L6-v2) that find similar customers using vector similarity
  3. Cohort Analysis — Real-time segmentation by device, country, age group with aggregated metrics

Architecture Overview

Our AI-powered customer analytics application demonstrates this in practice: 1 million records in memory (200MB), a compact sentence transformer model for semantic search, sub-second query performance, and zero infrastructure to manage. The solution uses a simple, serverless architecture:

  • Customer transaction data (Parquet format) is stored in Amazon S3
  • Amazon Cognito User Pool authenticates users and issues JWT tokens for API access
  • Amazon API Gateway routes requests with Cognito authorizer validation, rate limiting (5 requests/second, burst 10), X-Ray tracing, and access logging
  • AWS Lambda function with AWS Lambda Managed Instances loads the entire dataset (200MB) and all-MiniLM-L6-v2 model (900MB) into memory during initialization while also performing a threaded embeddings cache generation. This step can consume about 14GB of the allocated memory, exceeding standard AWS Lambda’s 10 GB limit
  • Analytics queries execute against the in-memory data using the model
  • Results are returned in milliseconds for interactive analysis

Architecture diagram

Deploy the Application

The below steps walk you through deploying the application to AWS using the AWS Serverless Application Model (SAM). The deployment process packages your Lambda function code, uploads artifacts to Amazon S3, and provisions all required AWS resources including Lambda functions, IAM roles, and any configured VPC networking via AWS CloudFormation.

Prerequisites

Make sure you have the following tools installed locally:

  • AWS CLI configured with credentials
  • SAM CLI installed
  • Python 3.13+ installed locally
  • Docker or Finch (required for container builds)
  • AWS account with appropriate permissions
  • A VPC with at least 2 subnets (across different Availability Zones) and a security group — required for the Lambda Managed Instances capacity provider
  • Supported regions: Check AWS Capabilities by Region for supported regions

Getting Started

The complete source code for this application is available in our GitHub repository. To deploy it yourself follow the below steps and refer to the full deployment instructions hosted on GitHub.

1. Clone the repository

git clone https://github.com/aws-samples/sample-lambda-managed-instances-analytics.git

2. Navigate to the project folder

cd sample-lambda-managed-instances-analytics

chmod +x setup-data.sh deploy-lambda.sh

3. Generate sample data and upload to S3

./setup-data.sh

This script will create an S3 bucket (if needed), generate 1M rows of sample data, and upload the data to S3.

4. Build and deploy the Lambda function

./deploy-lambda.sh

This script will build the container image with FastEmbed, push it to ECR, and deploy the Lambda function along with Capacity Provider, API Gateway, and Cognito User Pool. After deployment, it automatically generates the UI authentication configuration and prompts you to create a test user.

SAM template

Capacity provider configuration

Run the Application

1. Start the UI

The application includes a simple HTML-based UI through which you can test the AWS Lambda function using Amazon API Gateway:

cd ui && python3 -m http.server 8000

2. Open your browser at http://localhost:8000 and click ‘Sign In’ to authenticate via Cognito using the username/password that you created during deployment

Starting the UI

3. Enter your API endpoint URL. Test connection and click system Info.

Testing the connection

Test the Application

a. Customer Analysis — Enter one or more User IDs to get more information on the customer behavior: engagement scores, conversion rates, purchase patterns, and AI-generated customer segments

Running customer analysis

b. Semantic Search – Enter natural language queries like “list high value customers from USA” in the Semantic Search and verify the results. Note that the response is very fast as the analytics data and FastEmbed models are loaded into memory during init stage

Running semantic search

c. Cohort Analysis — Enter the query data to get Real-time segmentation by device, country, age group with aggregated metrics

Running cohort analysis

Observability

AWS Lambda Managed Instances automatically publishes metrics to Amazon CloudWatch, giving you visibility into function performance and capacity utilization. Monitor InitDuration to track dataset and model load time at startup, MaxMemoryUsed to confirm your data fits within configured memory, and ProvisionedConcurrencySpilloverInvocations to detect when AWS Lambda Managed Instances capacity is exhausted.

Enable AWS Lambda Insights for enhanced per-invocation metrics including CPU time and memory utilization over time. Use Amazon CloudWatch Log Insights to query INIT_START, INIT_END, and REPORT log entries for initialization and memory details per invocation.

AWS Lambda Insights

What Makes This Better with AWS Lambda Managed Instances

Without AWS Lambda Managed Instances, building this same application would require one of these alternatives:

  • Option A: EC2 with auto-scaling — Full control, full responsibility: patching, scaling policies, load balancing, and deployment pipelines — all on you.
  • Option B: Redesign for standard Lambda — Swap in-memory data for an external database and replace the ML model with Amazon SageMaker endpoint. More latency, more cost, more complexity.

With AWS Lambda Managed Instances, you write a single AWS Lambda function, define a Capacity Provider, and deploy with SAM. AWS Lambda handles the Amazon EC2 instances, scaling, and lifecycle, giving you the memory you need with the operational simplicity you want. The in-memory approach eliminates network latency and disk I/O, delivering consistent sub-200ms response times for complex analytics.

Cost Considerations

AWS Lambda Managed Instances uses Amazon EC2-based pricing with a management fee. For predictable workloads, you can leverage Amazon EC2 Savings Plans or Reserved Instances to reduce costs significantly.

Example cost comparison (us-east-1, 32 GB memory, 1M invocations/month):

  • AWS Lambda (standard): ~$267/month (on-demand pricing)
  • AWS Lambda Managed Instances: ~$180/month (with 1-year Compute Savings Plan)
  • Savings: 33% reduction

The cost benefits increase with higher memory configurations and sustained workloads that can take advantage of Amazon EC2 pricing discounts.

Best Practices

Based on experience building this solution, here are key recommendations:

  • Memory sizing: Start with your dataset size plus 50% overhead for processing. Monitor Amazon CloudWatch metrics to optimize.
  • Initialization strategy: Load large datasets during the init phase to amortize the cost across multiple invocations.
  • Concurrency configuration: Set PerExecutionEnvironmentMaxConcurrency based on your workload’s I/O characteristics. Higher values work well for I/O-bound analytics.
  • Data format: Use columnar formats like Parquet for efficient memory usage and fast loading.
  • Monitoring: Track initialization duration, memory utilization, and invocation latency in Amazon CloudWatch to identify optimization opportunities.

Cleanup

When you’re done exploring the solution, it’s good practice to remove all provisioned resources to avoid ongoing charges. For the full cleanup commands and exact steps, refer to the project’s README.md in GitHub repository.

Conclusion

AWS Lambda Managed Instances opens up a new class of serverless applications that support larger AWS Lambda layer packages and more memory. Memory-intensive workloads — in-memory analytics, ML inference, graph processing, scientific computing — can now run with the simplicity of AWS Lambda and the resources of Amazon EC2. The customer analytics example demonstrates how in-memory processing with AWS Lambda Managed Instances delivers performance improvements over traditional database queries while maintaining serverless benefits like automatic scaling and pay-per-use pricing.

Ready to get started? Explore the AWS Lambda Managed Instances documentation and try building your own memory-intensive serverless application. You can find the complete code for this example on GitHub.

A framework for securely collecting forensic artifacts into S3 buckets

Post Syndicated from Jason Garman original https://aws.amazon.com/blogs/security/a-framework-for-securely-collecting-forensic-artifacts-into-s3-buckets/

When customers experience a security incident, they need to acquire forensic artifacts to identify root cause, extract indicators of compromise (IoCs), and validate remediation efforts. NIST 800-86, Guide to Integrating Forensic Techniques into Incident Response, defines digital forensics as a process comprised of four basic phases: collection, examination, analysis, and reporting. This blog post focuses on the first phase—collection—and provides best practices for implementing least privilege during the forensic evidence collection processes that collect evidence and store the artifacts in Amazon Simple Storage Service (Amazon S3) buckets. The architecture presented in this post can be used to collect forensic evidence from both Amazon Web Services (AWS) and non-AWS compute resources.

It’s important to consider the security of the forensic artifact collection process because it involves communicating with potentially compromised resources. The collection methodology itself should be designed to avoid adding additional risks to infrastructure or other forensic investigation processes. At the same time, the collection of forensic artifacts requires the use of specialized tools that are difficult to change or adapt to new security requirements.

This post outlines factors that you should consider when creating an evidence collection capability and introduces an architecture that implements the best practices for least privilege and integrating with (instead of changing or adapting) existing forensic tools that support uploading artifacts to S3 buckets by using AWS security credentials.

Solution architecture

The architecture presented in this post demonstrates the following AWS best practices:

  1. Least privilege – Use AWS Identity and Access Management (IAM) policies to provide least privilege access to upload forensic artifacts to an S3 location dedicated to a specific forensic collection task. The locked down credentials cannot be used to view or modify any other forensic collections.
  2. Time-limited credentials – Use AWS Security Token Service (AWS STS) to provide time limited credentials, reducing the potential for an unauthorized user to abuse credentials while they’re visible on the target machine during the artifact collection process.
  3. Compatibility with third-party tools – Forensic tools are specialized and changing a forensic collection process to adapt to different collection methods might not be possible. To avoid the risk of needing to change tools, maximize compatibility with any third-party tools that support uploading to S3 buckets. The method introduced in this post to generate time-limited, scoped down credentials can be used with most third-party forensic tools that support uploading to S3 buckets.
  4. Credential vending – Use time-limited tokens, which can be vended on demand through an automated process, eliminating the need for forensic investigators to use the AWS Management Console, understand least privilege, or have any access to the AWS control plane. Forensic investigators can focus on the process of collecting and analyzing evidence.
  5. Process automation – Deploy the process as infrastructure as code (IaC) and automate it through AWS services, reducing the burden on security teams to manually perform runbook steps during an active security incident.

This post starts with an overview of the digital forensic process, provides best practices for using Amazon S3 to store forensic artifacts, details how you can create time-limited, least privilege tokens to provide secure access to upload forensic artifacts to S3 buckets, and introduces a sample architecture that automates the end-to-end process.

The digital forensic process

Organizations need to have practices and resources in place to support a digital forensic investigation environment before an incident occurs. AWS has published several resources, including Forensic investigation environment strategies in the AWS Cloud and AWS prescriptive guidance: Security Reference Architecture, Cyber forensics, to provide best practices for organizing your AWS accounts using AWS Organizations to support forensic clean-room environments. Creating segregated AWS accounts and resources for your security teams is critical to provide your incident responders a location to store and analyze any digital forensic evidence collected during an investigation.

After you’ve established a landing zone for performing digital forensics, you’re ready to collect and process digital forensic evidence. AWS supports the collection of digital forensics through extensive logging of control plane events in AWS CloudTrail, and metrics and application logs that can be stored in Amazon CloudWatch. In addition, AWS core compute services, such as Amazon Elastic Compute Cloud (Amazon EC2), support forensics operations through snapshots of the underlying Amazon Elastic Block Storage (Amazon EBS) volume. An example architecture to demonstrate how to automate the collection of EBS volume snapshots for forensic investigations can be found in How to automate forensic disk collection in AWS.

You might want to use the same AWS infrastructure to collect, examine, analyze, and report on forensic incidents that occur on other resources, such as corporate laptops. You can use existing forensic tooling to perform live response, collecting specific artifacts such as Windows NT File System (NTFS) Master File Table (MFT), logs from Linux machines, volatile memory images, or other artifacts that are specified as part of your organization’s incident response plan. These tools can be provided by third parties or built in-house, and many support uploading to S3 buckets using AWS security credentials.

Using Amazon S3 for forensic artifact collection

Amazon S3 provides the foundational requirements for collecting and storing forensic artifacts. Digital forensics requires highly available, durable, and secure storage of artifacts collected from potentially compromised systems. Amazon S3 is designed for 11 nines of durability and can be configured to provide protection against modification, deletion, and unauthorized access to sensitive forensic artifacts. You can also use S3 to store forensic artifacts of almost any size—from one byte to 5 TB—in an S3 object.

S3 buckets used to store forensic artifacts require custom configuration to provide additional security. You should configure the S3 bucket that you use to store forensic artifacts to enable the following security and governance features:

  1. Encryption in transit. You can require the use of encryption in transit and specify acceptable TLS versions using the aws:SecureTransport and s3:TlsVersion condition keys on the S3 bucket policy.
  2. Encryption at rest using a customer managed key. You can automatically encrypt all objects uploaded to the bucket using a specified customer managed key by specifying a default server-side encryption key in the bucket’s configuration. For this post, we encourage you to use a customer managed key rather than relying upon an AWS managed key, so you can control the associated key policy.
    1. Encryption at rest provides an additional layer of protection, because only entities that have both the permission to read from the bucket and permission to use the AWS Key Management Service (AWS KMS) key for decryption can download the forensic artifact from the S3 bucket.
    2. You need to adjust the example KMS policies in this post if the evidence collection S3 bucket uses the S3 Bucket Key feature.
  3. Audit logs of all S3 data event activity. You can turn on CloudTrail data events for any S3 buckets that contain forensic artifacts to provide a comprehensive audit trail of S3 object-level API activity. This helps provide a chain of custody of any artifacts stored in your forensic buckets.
  4. Fine-grained access control using IAM permissions. You can define the set of entities (both human and machine) that have access to the artifacts in the S3 bucket. This post includes how to create time-limited, least privilege access using IAM permissions for uploading files into an S3 bucket. The permissions are fine-grained enough to scope down access to specific object names or object prefixes in an S3 bucket. Additionally, access to read the artifacts can be controlled through IAM permissions and access to the encryption-at-rest KMS key.
  5. Protections against data modification and deletion. S3 provides features, such as S3 object versioning, to provide assurances that data hasn’t been modified or removed after it’s been collected. This is an additional layer of protection beyond the fine-grained access permissions, so even if an authorized entity attempts to overwrite or delete an object in the S3 bucket, the previous version of the object is still available.
  6. There are additional options that you can configure on the S3 bucket to protect your data against modification and deletion, including S3 Object Lock and multi-factor authentication (MFA) delete.

In addition to the preceding configuration, consider how to organize forensic artifacts in the S3 bucket. This post introduces a folder structure using S3 object prefixes to segregate each forensic artifact collection task into its own S3 object namespace. An example S3 namespace structure for an S3 bucket is shown in Figure 1.

Figure 1 – S3 namespace structure for an S3 forensics artifact bucket using object prefixes

Figure 1: S3 namespace structure for an S3 forensics artifact bucket using object prefixes

By separating each forensic collection task by its own prefix, you can use fine-grained IAM permissions to permit object uploads only into the active collection task. For example, scoped down credentials can be generated to only allow uploads into buckets with the CASE-0001 prefix using an IAM permission as shown in the following code example. Temporary security credentials can be generated using these limited permissions and the key is then used by the forensic acquisition tool to upload the artifacts into the S3 bucket.

{
	"Sid": "UploadToCase0001",
	"Effect": "Allow",
	"Action": [
		"s3:PutObject",
		"s3:AbortMultipartUpload"
	],
"	Resource": "arn:aws:s3:::mycompany-forensics-collection/CASE-0001/*"
}

Manually creating temporary IAM credentials for each forensic collection activity can be error-prone and time-consuming. Therefore, this post demonstrates how to use AWS tooling to automate the process of generating time-limited, scoped-down credentials.

Adapt existing forensic tools for AWS best security practices

Existing forensic tools typically use IAM access keys to perform S3 operations. Using a static IAM user secret access key isn’t a best practice. Even if the static key is associated with an IAM user that has been scoped down to only have access to the forensic collection S3 bucket as described previously, that means anyone with access to that key can potentially upload objects into that bucket. Therefore, the best practice is to create a time-limited temporary security credential unique to each collection activity, scoped down to only allow uploading files to a specific prefix in the target S3 bucket.

The examples in this post use the following resource names. Because these names will change based on your deployment, substitute your resource names in place of the names in the example code.

  1. The evidence S3 bucket is named mycompany-forensics-collection
  2. The forensics AWS account number is 112233445566. For the purposes of this example, all resources will live within this account.
  3. The customer managed key used to encrypt the forensic artifacts at rest is ForensicsEvidenceKey
  4. The IAM role that incident responders will assume when signing in to their AWS account is ForensicsUserRole
  5. The IAM role that incident responders will use for generating S3 file upload temporary credentials is ForensicsUploadRole
  6. The example uses the us-east-1 AWS Region

The following steps show you how to configure the IAM policies associated with the customer managed key ForensicsEvidenceKey and the IAM role ForensicsUploadRole.
Before you begin, create the evidence S3 bucket configured as described in Using S3 for artifact collection and a customer managed key to encrypt the forensic artifacts at rest. Configure the evidence S3 bucket to use the KMS key by opening the S3 bucket’s properties tab in the Amazon S3 console and setting the new KMS key as the default encryption key for the bucket.

Next, create an IAM role that incident responders will assume through the AWS STS AssumeRole API to generate the temporary credentials. This role will define the maximum set of permissions allowed to upload artifacts to your evidence S3 bucket. This role, ForensicsUploadRole, created using the following example code, defines the maximum allowable permissions: the ability to upload objects into the evidence S3 bucket and to use the KMS key to encrypt those uploads. The effective permissions available to the forensic tool will be scoped down even further to the specific object prefix when the AWS STS temporary security credential is generated.

Note that the policy allows the forensics upload role Decrypt permission in addition to Encrypt; this is required when uploading files larger than 5 GB using the multi-part S3 file upload feature.

{
	"Version": "2012-10-17",
	"Statement": [
			{
				"Sid": "BasePermissionsForS3Upload",
				"Effect": "Allow",
				"Action": [
					"s3:PutObject",
					"s3:AbortMultipartUpload"
				],
				"Resource": "arn:aws:s3:::mycompany-forensics-collection/*"
		},
		{
			"Sid": "KeyAccessToS3Upload",
			"Effect": "Allow",
			"Action": [
				"kms:GenerateDataKey",
				"kms:Encrypt",
				"kms:Decrypt"
			],
			"Resource": "arn:aws:kms:us-east-1:112233445566:alias/ForensicsEvidenceKey",
			"Condition": {
				"StringLike": {
					"kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::mycompany-forensics-collection/*"
				}
			}
		}
	]
}

Next, you need to provide an ability to assume this role and generate AWS STS tokens using the role’s permissions. This is accomplished by creating a trust relationship associated with the IAM role you just created. The trust relationship shown in the following code sample describes which AWS principals are allowed to assume the role—in this case, you will allow any user who has federated into the ForensicsUserRole IAM role to be able to generate AWS STS tokens for forensic artifact collection.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Statement1",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::112233445566:role/ForensicsUserRole"
			},
			"Action": "sts:AssumeRole"
		}
	]
}

After the role is established and access to the encryption key is granted, you can use the AWS STS AssumeRole API to create temporary credentials using this role. You can call this API using the AWS Command Line Interface (AWS CLI) or programmatically from a script. To scope down the token’s access to only provide permission to upload to the specific evidence object prefix, you must include a session policy as part of your AssumeRole API request to AWS STS. The following is an example session policy to restrict access to only upload objects into the CASE-0001 prefix.

[
	{
		"Effect": "Allow",
		"Action": [
			"s3:PutObject", 
			"s3:AbortMultipartUpload"
		],
		"Resource": "arn:aws:s3:::mycompany-forensics-collection/CASE-0001/*"
	},
	{
		"Effect": "Allow",
		"Action": [
			"kms:GenerateDataKey", 
			"kms:Encrypt", 
			"kms:Decrypt"
		],
		"Resource": "*",
		"Condition": {
			"StringLike": {
				"kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::mycompany-forensics-collection/CASE-0001/*"
			}
		}
	}
]

The effective permissions available to the session role will be the intersection of permissions available in the role policy (ForensicsUploadRole), the resource policy (in this case, mandating TLS-encrypted connections to the bucket), and the session policy that’s created on demand for every forensic collection (only allowing access to upload objects into the CASE-0001 prefix, as shown in the preceding example). Pictorially, this looks like the Venn diagram shown in Figure 2.

Figure 2 – Intersection of IAM policies determine the effective permissions for the restricted forensic session role.

Figure 2: Intersection of IAM policies determine the effective permissions for the restricted forensic session role.

Test the temporary credentials

Now that the bucket has been created and the AWS KMS key and roles configured, you can use AWS STS to create a temporary security credential for a collection on CASE-0001. You can use the AWS CLI to do this manually or you can write a script to automate this process using the AWS API. The IAM access key, secret access key, and session token returned by this call can then be used by any tool that can use AWS access keys to upload files into the specified S3 bucket.

The following example shows an AWS CLI call to AssumeRole using the example ForensicsUploadRole and a case named CASE-0001. The --duration-seconds parameter defines the period, in seconds, that the temporary credentials are valid; the default of 3600 seconds will provide temporary credentials that are valid for one hour.

$ aws sts assume-role \
	--role-arn arn:aws:iam::112233445566:role/ForensicsUploadRole \
	--role-session-name CASE-0001 \
	--duration-seconds 3600 \
	--policy '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["s3:PutObject", "s3:AbortMultipartUpload"], "Resource": "arn:aws:s3:::mycompany-forensics-collection/CASE-0001/*"}, {"Sid": "BasePermissionsForS3Upload", "Effect": "Allow", "Action": ["kms:GenerateDataKey", "kms:Encrypt", "kms:Decrypt"], "Resource": "*"}]}'

{
	"Credentials": {
		"AccessKeyId": "ASIAXXXX",
		"SecretAccessKey": "XXXX",
		"SessionToken": "XXXX",
		"Expiration": "2025-04-10T17:16:13+00:00"
	},
	"AssumedRoleUser": {
		"AssumedRoleId": "AROXXXX:CASE-0001",
		"Arn": "arn:aws:sts::112233445566:assumed-role/ForensicsUploadRole/CASE-0001"
	},
	"PackedPolicySize": 39
}

Now that you have obtained temporary credentials from AWS STS, you can use those credentials to upload a file into Amazon S3:

$ AWS_ACCESS_KEY_ID=ASIAXXXX \
	AWS_SECRET_ACCESS_KEY=XXXX \
	AWS_SESSION_TOKEN=XXXX \
	aws s3 cp evidence.zip s3://mycompany-forensics-collection/CASE-0001/evidence.zip

upload: evidence.zip to s3://mycompany-forensics-collection/CASE-0001/evidence.zip

You can also verify that you can’t use those credentials to upload a file into any other object prefixes or S3 buckets. For example, if you change CASE-0001 to CASE-0004 in the Amazon S3 upload command, you will receive an AccessDenied error because you’re trying to upload an object outside of the allowed key prefix.

$ AWS_ACCESS_KEY_ID=ASIAXXXX \
	AWS_SECRET_ACCESS_KEY=XXXX \
	AWS_SESSION_TOKEN=XXXX \
	aws s3 cp evidence.zip s3://mycompany-forensics-collection/CASE-0004/evidence.zip

upload failed: evidence.zip to s3://mycompany-forensics-collection/cases/CASE-0004/evidence.zip
An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:sts::112233445566:assumed-role/ForensicsUploadRole/CASE-0001 is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::mycompany-forensics-collection/CASE-0004/evidence.zip" because no session policy allows the s3:PutObject action

Additionally, if you wait more than the lifetime of the token (1 hour in this case), attempting to upload a file into the bucket will fail, because the token will no longer be valid:

$ AWS_ACCESS_KEY_ID=ASIAXXXX \
	AWS_SECRET_ACCESS_KEY=XXXX \
	AWS_SESSION_TOKEN=XXXX \
	aws s3 cp evidence.zip s3://mycompany-forensics-collection/CASE-0001/evidence.zip

upload failed: evidence.zip to s3://mycompany-forensics-collection/CASE-0001/evidence.zip

An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired.

Create an automated process to vend temporary credentials on demand

After you’ve verified the security benefits of creating temporary credentials for S3 uploads and validated that the credentials work with your forensic software of choice, you can now use them as part of an automated process.

A sample automated architecture is shown in Figure 3.

Figure 3: Architecture to automate S3 credential vending and forensic artifact collection.

Figure 3: Architecture to automate S3 credential vending and forensic artifact collection.

The workflow depicted in Figure 3 includes the following steps:

  1. The workflow is triggered by an alert from a detection source or a manual trigger from an incident responder.
  2. The workflow input is added to an Amazon Simple Queue Service (Amazon SQS) queue.
  3. The Amazon SQS queue invokes an AWS Lambda function which in turn executes a Step Functions state machine to orchestrate the workflow.
  4. First, the Step Functions workflow determines whether the target system is managed by AWS Systems Manager.
    1. If the target system isn’t managed by Systems Manager, an error is noted, and the execution is abandoned.
    2. If the target system is managed by Systems Manager, the Step Functions workflow determines the operating system (OS) of the target system and proceeds with the flow of execution.
  5. The workflow then continues by executing the Systems Manager documents that implement the forensic collection process:
    1. Downloads tooling:
      1. Generates dynamically scoped IAM temporary credentials that provide access to download the OS-specific tooling to be executed on the target system from the tooling S3 bucket. These credentials are tightly scoped to only allow downloads from the S3 prefix that corresponds to the tooling for the target system’s OS.
      2. Executes a Systems Manager command on the target system that uses the credentials generated from the previous step to download the OS tooling on the target system.
    2. Runs forensic tools:
      • Executes a Systems Manager command on the target system to execute the OS tooling on the target system.
  6. The Systems Manager commands run on the target system, which in this case is an EC2 instance.
  7. Results are uploaded to the evidence S3 bucket:
    1. Generates dynamically scoped IAM temporary credentials (as described previously) that provide access to upload the output of the previously executed tooling to the evidence S3 bucket. These credentials are tightly scoped to only allow uploads to a particular S3 prefix corresponding to the alert prefix.
    2. Executes a Systems Manager command on the target system to upload the output of the previously executed tooling to the evidence S3 bucket. After the upload is complete, it cleans up both the output and the evidence tooling from the target system.
    3. The evidence S3 bucket is tightly locked down to a subset of identities within the AWS security account. Access attempts from identities that aren’t allow listed trigger an Amazon EventBridge rule to alert the security team through an Amazon Simple Notification Service (Amazon SNS) topic.
  8. When the workflow is complete, related details and metrics are recorded in an Amazon DynamoDB table.
  9. The forensic analysis can be performed on a separate EC2 instance that has access to read from the evidence S3 bucket.

Deploying the example solution

You can use the AWS Cloud Development Kit (AWS CDK) repository to implement the architecture shown in Figure 3.

The AWS CDK solution is split into three stacks:

  1. SecurityStack: This stack contains the basic forensic artifact workflow orchestration infrastructure described in this post, including the Step Functions workflow, Lambda functions, AWS SQS queues, IAM roles, and S3 buckets.
  2. AlertStack: This stack contains the EventBridge workflow to notify administrators of anomalous activity in the evidence S3 bucket.
  3. CustomerStack: This stack contains the SSM documents that are executed for the forensic artifact workflow and an IAM role assumed by the SecurityStack when the workflow is invoked. It’s deployed into each child AWS account containing EC2 instances from which the security account is authorized to collect forensic artifacts.

Configuration

Before deploying the solution, there are several variables in the config.ts file that must be modified for the environment:

  1. SECURITY_ACCOUNT: Security Tooling AWS account ID.
  2. CUSTOMER_ACCOUNTS: Target AWS account IDs (the Child AWS account in the architecture diagram).
  3. ALERT_EMAIL_RECIPIENTS: List of email addresses that receive alerts when there is unexpected access to the evidence S3 bucket.
  4. ALLOW_LISTED_ROLE_NAMES: Roles allowed to access the evidence S3 bucket. Any other identities accessing the evidence S3 bucket will result in an alarm.

Deployment

After you’ve updated the config.ts file to reflect the account numbers, email recipients, and role names, the stacks can be deployed into your AWS infrastructure.

  1. Set Up AWS credentials using the AWS CLI:
    aws configure
  2. Install dependencies and configure constants:
    1. Clone the repository.
    2. Navigate to the project directory.
    3. Install project dependencies:
      npm install
    4. Configure constants in constants/config.ts with the required information:
      export const SECURITY_ACCOUNT = "123456789012"; // Your security tooling account ID 
      export const CUSTOMER_ACCOUNTS = ["234567890123", "345678901234"]; // Target account IDs 
      export const ALLOW_LISTED_ROLE_NAMES = ["SecurityAnalystRole"];// Roles allowed to access evidence S3 bucket 
      export const ALERT_EMAIL_RECIPIENTS = ["[email protected]"];// Email addresses for alerts

  3. Bootstrap AWS CDK in your accounts (if it hasn’t been done already):
    1. Example: cdk bootstrap aws://456789012345/us-east-1 (example security AWS account).
    2. Then bootstrap if necessary in any target AWS accounts.
  4. Deploy the AWS CDK Stacks:
    1. Synthesize the CloudFormation template:
      cdk synth
    2. Deploy the security and alert stacks in your security account:
      cdk deploy SecurityStack AlertStack
    3. Deploy the customer stacks in your workload accounts:
      cdk deploy CustomerStack-ACCOUNT_ID
  5. Set up your email alerts:
    1. After the AlertStack is deployed, it will email all addresses listed in ALERT_EMAIL_RECIPIENTS. Choose the embedded link to accept the AWS SNS topic in each of those accounts.

Testing

With deployment complete, it’s time to test the solution.

  1. Trigger an analysis
    1. Make sure you have a Linux EC2 instance running in one of your customer accounts and in the AWS Region where you deployed the preceding customer stack.
    2. Because this example uses Systems Manager to orchestrate the collection script, make sure that the EC2 instance is visible in Systems Manager either by checking the Systems Manager console, or by using the AWS CLI:
      1. Console: In the AWS Systems Manager console, choose Managed instances in the left navigation pane and verify your instance appears in the list. For more information, see Managed Instances in the AWS Systems Manager User Guide.
      2. AWS CLI: Run the following command to verify the instance is managed:
        aws ssm describe-instance-information --filters “Key=InstanceIds,Values=<instance-id>
        If the command returns instance information with PingStatus: Online, the instance is properly connected to Systems Manager.
    3. Post a message in your security account to the Amazon SQS queue to start the Step Functions workflow. Note that the values in angle brackets (for example <accountID>) are placeholders that you must update with relevant AWS account ID, tracking ticket ID, AWS Region, and EC2 instance ID values:
      aws sqs send-message --queue-url --message-body ‘{ “account”: “”, “ticket_id”: “”, “region”: “>”, “instance_id”: “” }’
  2. Go to the Step Functions console to view the successful execution of the workflow:
    Figure 4 – Workflow as shown in the Step Functions console

    Figure 4: Workflow as shown in the Step Functions console

  3. View the DynamoDB table to see the metadata for the results.
  4. Check the evidence S3 bucket to see the uploaded files from the forensic collection.

Conclusion

Collecting forensic artifacts securely is a critical component of any digital forensics investigation. This post demonstrated how to implement least privilege access controls and time-limited credentials for forensic evidence collection workflows that use Amazon S3 for artifact storage. By combining IAM session policies with AWS STS temporary credentials, you can provide forensic tools with secure, scoped-down access to upload artifacts without exposing long-lived credentials or granting overly permissive access.

The architecture presented in this post automates the process of generating temporary credentials, collecting forensic artifacts from both AWS and non-AWS resources, and securely storing them in S3 buckets with appropriate encryption, access controls, and audit logging. With this approach, your security teams can focus on analyzing evidence instead of managing credentials and permissions during active security incidents.To get started with this solution, deploy the example AWS CDK stacks provided in the collect forensic artifacts repository and customize them for your organization’s forensic investigation requirements. For more information about related AWS forensic investigation architectures, review the Automated Forensics Orchestrator for EC2 and How to build forensic kernel modules for Linux EC2 instances resources.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Jason Garman

Jason Garman

Jason is a principal security specialist solutions architect at AWS. He has 30 years of cybersecurity experience including incident response, reverse engineering, identity, and data protection. At AWS, he helps large organizations adopt the latest cloud and AI technologies while maintaining a high bar for data governance, security, and safety.

Vaishnav Murthy

Vaishnav Murthy

Vaishnav is a Senior Security Engineer with AWS CloudResponse. He has an extensive background in incident response and security automation and enjoys building automated solutions that help AWS customers investigate and respond to security incidents at scale.

Launching S3 Files, making S3 buckets accessible as file systems

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/

I’m excited to announce Amazon S3 Files, a new file system that seamlessly connects any AWS compute resource with Amazon Simple Storage Service (Amazon S3).

More than a decade ago, as an AWS trainer, I spent countless hours explaining the fundamental differences between object storage and file systems. My favorite analogy was comparing S3 objects to books in a library (you can’t edit a page, you need to replace the whole book) versus files on your computer that you can modify page by page. I drew diagrams, created metaphors, and helped customers understand why they needed different storage types for different workloads. Well, today that distinction becomes a bit more flexible.

With S3 Files, Amazon S3 is the first and only cloud object store that offers fully-featured, high-performance file system access to your data. It makes your buckets accessible as file systems. This means changes to data on the file system are automatically reflected in the S3 bucket and you have fine-grained control over synchronization. S3 Files can be attached to multiple compute resources enabling data sharing across clusters without duplication.

Until now, you had to choose between Amazon S3 cost, durability, and the services that can natively consume data from it or a file system’s interactive capabilities. S3 Files eliminates that tradeoff. S3 becomes the central hub for all your organization’s data. It’s accessible directly from any AWS compute instance, container, or function, whether you’re running production applications, training ML models, or building agentic AI systems.

You can access any general purpose bucket as a native file system on your Amazon Elastic Compute Cloud (Amazon EC2) instances, containers running on Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS), or AWS Lambda functions. The file system presents S3 objects as files and directories, supporting all Network File System (NFS) v4.1+ operations like creating, reading, updating, and deleting files.

As you work with specific files and directories through the file system, associated file metadata and contents are placed onto the file system’s high-performance storage. By default, files that benefit from low-latency access are stored and served from the high performance storage. For files not stored on high performance storage such as those needing large sequential reads, S3 Files automatically serves those files directly from Amazon S3 to maximize throughput. For byte-range reads, only the requested bytes are transferred, minimizing data movement and costs.

The system also supports intelligent pre-fetching to anticipate your data access needs. You also have fine-grained control over what gets stored on the file system’s high performance storage. You can decide whether to load full file data or metadata only, which means you can optimize for your specific access patterns.

Under the hood, S3 Files uses Amazon Elastic File System (Amazon EFS) and delivers ~1ms latencies for active data. The file system supports concurrent access from multiple compute resources with NFS close-to-open consistency, making it ideal for interactive, shared workloads that mutate data, from agentic AI agents collaborating through file-based tools to ML training pipelines processing datasets.

Let me show you how to get started.
Creating my first Amazon S3 file system, mounting, and using it from an EC2 instance is straightforward.

I have an EC2 instance and a general purpose bucket. In this demo, I configure an S3 file system and access the bucket from an EC2 instance, using regular file system commands.

For this demo, I use the AWS Management Console. You can also use the AWS Command Line Interface (AWS CLI) or infrastructure as code (IaC).

Here is the architecture diagram for this demo.

S3 Files demo architectureStep 1: Create an S3 file system.

On the Amazon S3 section of the console, I choose File systems and then Create file system.

S3 Files create file system

I enter the name of the bucket I want to expose as a file system and choose Create file system.

S3 Files create file system, part 2

Step 2: Discover the mount target.

A mount target is a network endpoint that will live in my virtual private cloud (VPC). It allows my EC2 instance to access the S3 file system.

The console creates the mount targets automatically. I take notes of the Mount target IDs on the Mount targets tab.

When using the CLI, two separate commands are necessary to create the file system and its mount targets. First, I create the S3 file system with create-file-system. Then, I create the mount target with create-mount target.

Step 3: Mount the file system on my EC2 instance.

After it’s connected to an EC2 instance, I type:

sudo mkdir /home/ec2-user/s3files sudo mount -t s3files fs-0aa860d05df9afdfe:/ /home/ec2-user/s3files

I can now work with my S3 data directly through the mounted file system in ~/s3files, using standard file operations.

When I make updates to my files in the file system, S3 automatically manages and exports all updates as a new object or a new version on an existing object back in my S3 bucket within minutes.

Changes made to objects on the S3 bucket are visible in the file system within a few seconds but can sometimes take a minute or longer.

# Create a file on the EC2 file system 
echo "Hello S3 Files" > s3files/hello.txt 

# and verify it's here 
ls -al s3files/hello.txt
 -rw-r--r--. 1 ec2-user ec2-user 15 Oct 22 13:03 s3files/hello.txt 

# See? the file is also on S3 
aws s3 ls s3://s3files-aws-news-blog/hello.txt 
2025-10-22 13:04:04 15 hello.txt 

# And the content is identical! 
aws s3 cp s3://s3files-aws-news-blog/hello.txt . && cat hello.txt
Hello S3 Files

Things to know
Let me share some important technical details that I think you’ll find useful.

Another question I frequently hear in customer conversations is about choosing the right file service for your workloads. Yes, I know what you’re thinking: AWS and its seemingly overlapping services, keeping cloud architects entertained during their architecture review meetings. Let me help demystify this one.

S3 Files works best when you need interactive, shared access to data that lives in Amazon S3 through a high performance file system interface. It’s ideal for workloads where multiple compute resources—whether production applications, agentic AI agents using Python libraries and CLI tools, or machine learning (ML) training pipelines—need to read, write, and mutate data collaboratively. You get shared access across compute clusters without data duplication, sub-millisecond latency, and automatic synchronization with your S3 bucket.

For workloads migrating from on-premises NAS environments, Amazon FSx provides the familiar features and compatibility you need. Amazon FSx is also ideal for high-performance computing (HPC) and GPU cluster storage with Amazon FSx for Lustre. It’s particularly valuable when your applications require specific file system capabilities from Amazon FSx for NetApp ONTAP, Amazon FSx for OpenZFS, or Amazon FSx for Windows File Server.

Pricing and availability
S3 Files is available today in all commercial AWS Regions.

You pay for the portion of data stored in your S3 file system, for small file read and all write operations to the file system, and for S3 requests during data synchronization between the file system and the S3 bucket. The Amazon S3 pricing page has all the details.

From discussions with customers, I believe S3 Files helps simplify cloud architectures by eliminating data silos, synchronization complexity, and manual data movement between objects and files. Whether you’re running production tools that already work with file systems, building agentic AI systems that rely on file-based Python libraries and shell scripts, or preparing datasets for ML training, S3 Files lets these interactive, shared, hierarchical workloads access S3 data directly without choosing between the durability of Amazon S3 and cost benefits and a file system’s interactive capabilities. You can now use Amazon S3 as the place for all your organizations’ data, knowing the data is accessible directly from any AWS compute instance, container, and function.

To learn more and get started, visit the S3 Files documentation.

I’d love to hear how you use this new capability. Feel free to share your feedback in the comments below.

— seb

How Aigen transformed agricultural robotics for sustainable farming with Amazon SageMaker AI

Post Syndicated from Purna Sanyal original https://aws.amazon.com/blogs/architecture/how-aigen-transformed-agricultural-robotics-for-sustainable-farming-with-amazon-sagemaker-ai/

This post is cowritten with Yuri Brigance, and Usman M. Khan from Aigen.

Aigen builds autonomous robots designed to help farmers remove herbicide-resistant weeds and improve crop yield through AI-driven technology. These robots operate without chemicals, using renewable energy, and provide real-time, field-level data to enhance decision-making. Using advanced computer vision AI, Aigen’s robots autonomously identify and remove weeds without harming crops, giving farmers an eco-friendly, cost-effective solution to traditional weed management and efficient farming. As its robotic fleet expanded, Aigen’s on-premises infrastructure became a bottleneck in scaling its model-building pipeline.

In this post, you will learn how Aigen modernized its machine learning (ML) pipeline with Amazon SageMaker AI to overcome industry-wide agricultural robotics challenges and scale sustainable farming. This post focuses on the strategies and architecture patterns that enabled Aigen to modernize its pipeline across hundreds of distributed edge solar robots and showcase the significant business outcomes unlocked through this transformation. By adopting automated data labeling and human-in-the-loop validation, Aigen increased image labeling throughput by 20x while reducing image labeling costs by 22.5x.

Key challenges of scaling field agricultural robots

Aigen’s initial ML pipeline was designed to build task-specific edge models for its field robots. Robot data was uploaded to Amazon Simple Storage Service (Amazon S3) for manual labeling. The annotated datasets were then used to train new task-specific edge models on Aigen’s on-premises infrastructure. However, this ML pipeline introduced several limitations:

  • Connectivity Constraints: Inconsistent internet in rural areas hampered communication between robots and cloud.
  • High Data Labeling Cost: Manual data labeling of thousands of new samples data per day proved prohibitively expensive and time-consuming.
  • Limited Computational Power: Training specialized edge models and fine-tuning foundation models (FMs) for specific tasks using on-premises hardware was a bottleneck due to limited parallelism and GPU compute power with on-premises RTX 3090 machines.
  • Scalability Issues: Model Training and data labeling batch Inference had to compete for the same RTX 3090 machines, causing delays either for the data science team for model training or the data labeling team for batch inference.

Solution

Aigen addresses these challenges by adopting an AWS AI-driven, cloud-native approach that enables scalable and automated operations:

  • Edge Computing: Robots use AWS IoT Core and cloud utilities to safely offload data to Amazon S3, even in low-connectivity regimes.
  • Automated Data Pipeline: Data collected by the robots flows through an Extract, Transform, and Load (ETL) pipeline for preprocessing. Data labeling is accelerated using an ensemble of vision foundation models (Grounding DINO, Owl-ViT, SAM2, CLIPSeg) along with custom expert vision models to automatically annotate large volumes of field imagery. Through active learning, the pipeline selects and down-samples the most informative samples, which are then reviewed and refined by human annotators before being passed downstream into the model training workflow.
  • Cloud native ML Pipeline: Aigen accelerates model training on Amazon SageMaker AI, using Distributed Data Parallel (DDP) across multi-GPU clusters to achieve faster iteration cycles and efficient hyperparameter tuning. By scaling training in the cloud, Aigen removes resource contention between model training and data labeling batch inference. This results in improved throughput, reduced wait times, and a more predictable ML workflow for data science and labeling teams.

Let’s take a closer look at how Aigen’s solution architecture is designed to meet diverse machine learning needs, from data labeling to real-time inference on autonomous field robots, starting with its model architecture.

Model architecture

Aigen’s models are classified in four hierarchical categories that form a progression from broad, general-purpose models to highly specialized models tailored for edge computing. Foundation Models (L1) are the starting point, with each subsequent category building on the previous model, adding specificity or performance enhancements.

Figure 1: Aigen Model Architecture

Figure 1: Aigen Model Architecture

  1. Foundation models use a combination of Aigen’s proprietary and open source foundation vision models to support plant detection, wheel detection, general object recognition, and segmentation. SAM2 is the primary model for generating segmentation masks, while Grounding DINO provides prompt-based annotation for objects like cars and people. Aigen employs a leading image generation model with ControlNet + Depth to create synthetic data with an option to fine-tune LoRA adapters to produce samples like field data. Aigen’s large vision models, trained on extensive field datasets, serve as robust foundations for crop identification and as high-quality starting points for building specialized pre-labeling models.
  2. Expert models are distilled from FMs and trained on annotated field images to perform precise, task-specific vision workloads. They generate high-quality pre-labels, bounding boxes, segmentation masks, and keypoint detections, which are then validated and refined by human annotators. Segmentation combined with key points allows the system to identify fine-grained plant anatomy, such as stems and other structural features. These models use both Vision Transformer and CNN-based architectures, and contain 10s of millions of parameters.
  3. Student models are compact, full-precision (FP32) models designed for ultra-low latency and minimal memory usage and are continuously fine-tuned on the latest data. Distilled from expert models, they remain extremely small, typically under 1.5M parameters, and are further improved through quantization-aware training (QAT), pruning, and other compression techniques. These optimizations enable efficient edge deployment, requiring as little as 2 Tera Operations Per Second (TOPS) while achieving real-time, double-digit frames per second (FPS) within the robot’s perception stack. Each student model is task-specific, tailored to individual crops (for example, tomato, cotton, sugar beets, soybeans) and various view angles such as top-down or intra-row.
  4. Edge models are built by further improving the full-precision student models for inference on the robot’s Neural Processing Unit (NPU). It undergoes QAT, followed by conversion to TFLite and INT8 quantization to reduce model size, lower power consumption, and increase inference throughput on the robot’s NPU. Purpose built for ultra-efficient edge inference, these models run on a 2.3-TOPS NPU using roughly 1.5W of power while sustaining real-time, double-digit FPS performance. These models contain 1M–1.2M parameters and occupy about 2 MB of memory.

This hybrid multi model ecosystem approach works well to balance model accuracy with edge computing constraints.

Modernized cloud native architecture for continuous model improvement

The modernized architecture forms a closed loop of nearly continuous model improvement, connecting field data collection from the robot to iterative training and rapid redeployment of updated models back onto the robot. This end-to-end cycle enables faster refinement, higher accuracy, and ongoing adaptation to real-world conditions.

Figure 2: Aigen modernized architecture

Figure 2: Aigen modernized architecture

The following sections describe the end-to-end process illustrated in the architecture diagram, from field data ingestion into AWS to continuous model delivery back to the robotic fleet. The workflow is organized into three key stages:

  1. Data Collection and Data Ingestion: Field Robots connect to AWS services using AWS IoT Core. Raw data, including navigation and crop-camera video (RGB + Depth), robot telemetry (odometry, frame timestamps), camera intrinsics/extrinsic, and job metadata, is continuously transmitted from the robots to Amazon S3 buckets. These data provide centralized storage for field, crop, and task specific downstream processing.
  2. Data Processing and Data Labeling: Aigen ETL unpacks the raw data, catalogs it, and stores it in Amazon S3. SageMaker AI processing jobs perform batch inference on this data and label the images using an ensemble of expert models running on the G5/G6 family of GPU instances. Aigen’s active learning process down-selects pre-labeled images and sends them for human review, where annotators validate and correct identified errors. Active learning analyzes images, embeddings, predictions, and other signals to identify the most informative samples for training. This approach removes the need to annotate every data point, often millions per field per season, by prioritizing images where the model struggles or those that add diversity. With multiple selection criteria, active learning helps keep dataset size manageable, control labeling effort, and verify only the most relevant samples are used to improve model performance.
  3. Model Training: The final annotated data is stored back in Amazon S3. SageMaker AI Training jobs pull this data from Amazon S3 and use multi-GPU instances to train expert, student and edge models. Edge-optimized models are deployed to the robots, while the newly finetuned expert models are used for the next cycle of data labeling.

Built on a cloud-native architecture, the workflow uses AWS services to deliver reliability, and robust performance, while effectively addressing the key challenges of scaling Aigen’s robotic fleet. The automated process collects data from field robot and use that in model training in the cloud, minimizing manual intervention while maintaining efficiency. Human-in-the-loop validation ensures high-quality training data by having annotators review and correct AI-generated pre-labels. Finally, active learning creates a positive feedback loop that continuously improves models by prioritizing the most relevant training data, enhancing robotic performance in real-world conditions.

Business benefits

This AI-powered solution delivered the following benefits:

  • Cost Efficiency: Reduced labeling costs from ~$2.00 to $0.089 per image, achieving a 22.5× cost reduction
  • Faster Annotation Pipeline: Reduced average annotation time from 14 minutes 57 seconds with manual labeling to just 41 seconds with SageMaker batch inference. This acceleration shortens model delivery for new crops from months to weeks, enabling quicker deployment and unlocking new business opportunities.
  • Rapid Scaling Gains: Experiment capacity increased from five per week on on-premises infrastructure to hundreds per week using Amazon SageMaker AI, achieving a 20× increase in throughput over previous hardware.
  • Innovation
    • The powerful GPU instances of Amazon SageMaker AI enabled the training and fine-tuning of advanced Vision Transformers models, which were not feasible on limited on-premises hardware. This access to state-of-the-art (SOTA) GPUs accelerates model innovation.
    • Scalable training infrastructure removes GPU bottlenecks by enabling parallel experimentation. This allows faster testing of new architectures and hyperparameters tuning, significantly speeding up model innovation compared to the slow, sequential workflow imposed by limited on-premises GPU capacity.

Key learnings

Amazon SageMaker AI has been instrumental in Aigen’s robotics system transformation, delivering significant benefits across the machine learning pipeline:

  • Self-Managed AI Infrastructure: SageMaker AI removes the need for Aigen to build and maintain auto scaling GPU compute infrastructure. This reduction in development costs allows Aigen to focus more on model development rather than infrastructure management, accelerating the production of deployment-ready models.
  • Streamlined ML Workflow: SageMaker AI streamlines the entire ML lifecycle, from data preparation to model deployment. Its flexibility supports the use of various built-in features and custom processes such as pre-labeling that cut down the time required to produce high-quality training data.
  • Efficient Resource Utilization: The managed infrastructure of SageMaker AI lowers operational overhead, supports continuous model updates, such as daily fine-tuning as plants grow, without resource bottlenecks. For example, when moving to a new customer’s cotton field with different soil, lighting, or crop varieties, the base cotton model may underperform. With SageMaker AI, Aigen can rapidly ingest new data and fine-tune models on this new condition to improve performance. Over multiple seasons and fields, this process builds a diverse, high-quality dataset that steadily strengthens the model family.

To achieve similar results in your organization, start by evaluating your current data labeling costs and consider implementing active learning techniques to reduce manual annotation overhead.

Conclusion

By using AWS services, particularly SageMaker AI, Aigen moved beyond the limitations of its on-premises infrastructure and established a foundation for continued growth and innovation. The new architecture delivers the scalability, efficiency, and intelligence needed to expand its fleet of eco-friendly agricultural robots, bringing sustainable farming practices to more fields worldwide. Aigen’s journey illustrates how generative AI can modernize machine learning pipelines for robotics, enabling more productive and environmentally sustainable agriculture. You can implement a similar architecture pattern to improve the machine learning pipeline.

Get started with model training and model inference by visiting Amazon SageMaker AI Studio. Creating your first Serverless ML flow pipeline is also supported in SageMaker AI Studio for additional workflow flexibility.


About the Authors

AWS Weekly Roundup: Amazon S3 turns 20, Amazon Route 53 Global Resolver general availability, and more (March 16, 2026)

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-s3-turns-20-amazon-route-53-global-resolver-general-availability-and-more-march-16-2026/

Twenty years ago this past week, Amazon S3 launched publicly on March 14, 2006. While Amazon Simple Storage Service is often considered the foundational storage service that defined cloud infrastructure, what began as a simple object storage service has grown into something far larger in scope and scale.

As of March 2026, S3 stores more than 500 trillion objects, serves more than 200 million requests per second globally across hundreds of exabytes of data, and the price has dropped to just over 2 cents per gigabyte — an approximately 85% reduction since launch. My colleague Sébastien Stormacq wrote a detailed look at the engineering and the road ahead in Twenty years of Amazon S3 and building what’s next, and if you want to read about those earliest customers and how they shaped what AWS became, I recommend How three startups helped Amazon invent cloud computing and paved the way for AI. Twenty years is worth pausing to celebrate.

Alongside the 20th anniversary of S3, Channy Yun also wrote about a new S3 feature this week: Account regional namespaces for Amazon S3 general purpose buckets. With this feature, you can create general purpose buckets in your own account regional namespace by appending your account’s unique suffix to your requested bucket name, ensuring your desired names are always reserved exclusively for your account. You can enforce adoption across your organization using AWS IAM policies and AWS Organizations service control policies with the new s3:x-amz-bucket-namespace condition key. Read Channy’s post to learn more about account regional namespaces for Amazon S3 general purpose buckets.

This week’s featured launch is one I have a personal connection to: the general availability of Amazon Route 53 Global Resolver. I wrote about the preview of this capability back in December at re:Invent 2025, and I had a great time putting that post together, so I am happy to hear that it’s generally available now.

Amazon Route 53 Global Resolver is an internet-reachable anycast DNS resolver that provides DNS resolution for authorized clients from any location. It is now generally available across 30 AWS Regions, with support for both IPv4 and IPv6 DNS query traffic. Route 53 Global Resolver gives authorized clients in your organization anycast DNS resolution of public internet domains and private domains associated with Route 53 private hosted zones — from any location, not just from within a specific VPC or Region. It also provides DNS query filtering to block potentially malicious domains, domains that are not safe for work, and domains associated with advanced DNS threats such as DNS tunneling and Domain Generation Algorithms (DGA). Centralized query logging is included as well. With general availability, Global Resolver adds protection against Dictionary DGA threats.

Last week’s launches
Here are some of the other announcements from last week:

  • Amazon Bedrock AgentCore Runtime now supports stateful MCP server features — Amazon Bedrock AgentCore Runtime now supports stateful Model Context Protocol (MCP) server features, enabling developers to build MCP servers that use elicitation, sampling, and progress notifications alongside existing support for resources, prompts, and tools. With stateful MCP sessions, each user session runs in a dedicated microVM with isolated resources, and the server maintains session context across multiple interactions using an Mcp-Session-Id header. Elicitation enables server-initiated, multi-turn conversations to gather structured input from users during tool execution. Sampling allows servers to request LLM-generated content from the client for tasks such as personalized recommendations. Progress notifications keep clients informed during long-running operations. To learn more, see the Amazon Bedrock AgentCore documentation.
  • Amazon WorkSpaces now supports Microsoft Windows Server 2025 — New bundles powered by Microsoft Windows Server 2025 are now available for Amazon WorkSpaces Personal and Amazon WorkSpaces Core. These bundles include security capabilities such as Trusted Platform Module 2.0 (TPM 2.0), Unified Extensible Firmware Interface (UEFI) Secure Boot, Secured-core server, Credential Guard, Hypervisor-protected Code Integrity (HVCI), and DNS-over-HTTPS. Existing Windows Server 2016, 2019, and 2022 bundles remain available. You can use the managed Windows Server 2025 bundles or create a custom bundle and image. This support is available in all AWS Regions where Amazon WorkSpaces is available. For more information, visit the Amazon WorkSpaces FAQs.
  • AWS Builder ID now supports Sign in with GitHub and Amazon — AWS Builder ID now supports two additional social login options: GitHub and Amazon. These options join the existing Google and Apple sign-in capabilities. With this update, developers can access their AWS Builder ID profile — and services including AWS Builder Center, AWS Training and Certification, and Kiro — using their existing GitHub or Amazon account credentials, without managing a separate set of credentials. To learn more and get started, visit the AWS Builder ID documentation.
  • Amazon Redshift introduces reusable templates for COPY operations — Amazon Redshift now supports templates for the COPY command, allowing you to store and reuse frequently used COPY parameters. Templates help maintain consistency across data ingestion operations, reduce the effort required to execute COPY commands, and simplify maintenance by applying template updates automatically to all future uses. Support for COPY templates is available in all AWS Regions where Amazon Redshift is available, including the AWS GovCloud (US) Regions. To get started, see the documentation or read the Standardize Amazon Redshift operations using Templates blog.

For a full list of AWS announcements, be sure to keep an eye on our News Blog channel the What’s New with AWS page.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

AWS Summits – Join AWS Summits in 2026, free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), and Bengaluru (April 23–24).

AWS Community Days – Community-led conferences where content is planned, sourced, and delivered by community leaders, featuring technical discussions, workshops, and hands-on labs. Upcoming events include Pune (March 21), San Francisco (April 10), and Romania (April 23-24).

AWS at NVIDIA GTC 2026 — Join us at our AWS sessions, booths, demos, and ancillary events in NVIDIA GTC 2026 on March 16 – 19, 2026 in San Jose. You can receive 20% off event passes through AWS and request a 1:1 meeting at GTC.

AWS Community GameDay Europe — Taking place on March 17, 2026, AWS Community GameDay Europe is a team-based, hands-on AWS challenge event running simultaneously across 50+ cities in Europe. Your team is dropped into a broken AWS environment — misconfigured services, failing architectures, and security gaps — and has two hours to fix as much as possible. Find your nearest city and sign up at awsgameday.eu.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Esra

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!